Method and apparatus for optimizing out-of-distribution (OOD) identification

Abstract

Methods and apparatus are presented herein for optimizing out-of-distribution (OOD) identification using a K-nearest neighbor (KNN) method in an integrated model having a plurality of artificial neural networks. The method for optimization comprises the steps of:-providing (S1) an adapted KNN detector for each of the plurality of artificial neural networks; calculating (S2), for each of the plurality of artificial neural networks, a feature vector based on the test or inference data; -providing (S3) the respective calculated feature vectors to the respective adapted KNN detectors; -determining (S4) a respective OOD score by the respective adapted KNN detectors on the basis of the respective provided feature vectors; and-averaging the determined OOD scores using a KNN method for the optimized OOD identification.

Description

Technical Field

[0001] The present invention relates to two alternative methods and two alternative devices for optimizing out-of-distribution (OOD) recognition using the K-nearest neighbor (KNN) method in an ensemble model having multiple artificial neural networks. The present invention also relates to a control device for performing such a method. The present invention also relates to a computer program and a computer-readable data carrier. Background Art

[0002] Artificial intelligence and especially machine learning are concerned with the ability of a model to correctly classify data and make predictions. In this context, the question is raised: how well or accurately can a model identify data that is significantly different from the data on which the training is based. This question is known as "out-of-distribution" (OOD) recognition, which has become increasingly important in recent years.

[0003] OOD recognition herein involves the task of identifying samples or data points that lie outside the training data distribution. These samples are abnormal or outside the distribution, which means they may be significantly different from the (training) data based on which the model was trained. Identifying OOD is crucial for the robustness and reliability of machine learning models, as it helps to identify situations in which model predictions may be unreliable or incorrect.

[0004] To effectively accomplish OOD recognition, different methods have been developed, which can be classified into three main categories: post-hoc-inference methods, training methods using OOD data, and training methods without OOD data. Post-hoc-inference methods become effective only at the inference stage and use a pre-trained network to generate an OOD score. In contrast, training methods using OOD data require re-training of the model by using OOD data, while training methods without OOD data perform model re-training without specific OOD data.

[0005] Recently, a comprehensive overview and benchmark for OOD recognition methods have been presented, which show how these different methods perform. It has been demonstrated here that post-hoc-inference methods are particularly effective in most cases and even perform better than competitors that rely on new training and OOD data. The K-nearest neighbor (KNN) method is an apparently simple but powerful method that uses the non-parametric distance between the nearest neighbors and the feature vectors of the training data to identify OOD data.

[0006] Although KNN is a powerful technique that provides high performance, it is difficult to use in an ensemble model because the feature vectors used in the KNN method may have different sizes and generally cannot be meaningfully or effectively combined.

[0007] Therefore, there is potential for improvement in the field of ensemble models, such as those used in the manufacturing field, to achieve better performance and generalization.

[0008] Therefore, the object on which the present invention is based is to describe at least one method and / or at least one device for using the K-nearest neighbor (KNN) method in an ensemble model with multiple artificial neural networks to optimize out-of-distribution (OOD) recognition.

[0009] This task is solved by a method according to the features of claim 1. This task is solved by an alternative method according to the features of claim 5. This task is solved by a device according to the features of claim 11. This task is solved by an alternative device according to the features of claim 12. This task is also solved by a control device for performing such a method. Furthermore, this task is solved by a computer program and a computer-readable data carrier. Summary of the Invention

[0010] According to a first aspect, a method for using the K-nearest neighbor (KNN) method in an ensemble model with multiple artificial neural networks to optimize out-of-distribution (OOD) recognition is proposed. The method for optimization comprises the following steps:

[0011] - Providing an adapted KNN detector for each of the multiple artificial neural networks;

[0012] - Calculating feature vectors for each of the multiple artificial neural networks based on test or inference data;

[0013] - Providing, in particular inputting, the separately calculated feature vectors into the separately adapted KNN detectors;

[0014] - Determining corresponding OOD scores by the separately adapted KNN detectors based on the separately provided feature vectors; and

[0015] - Averaging the determined OOD scores using the KNN method for optimized OOD recognition.

[0016] "K-Nearest Neighbor" (k-NN or KNN) describes a machine learning algorithm. This algorithm is based on the idea that similar data points in the feature space often have the same class or the same value. The k-NN algorithm works with a dataset consisting of data points. Each data point is described by a series of features. The "k" in k-NN represents a positive integer and is an important hyperparameter. It must be set: how many neighbors to consider. To find the k nearest neighbors for a given data point, the distance between this data point and all other data points in the dataset is calculated. Here, the Euclidean distance is used, but there are also other distance metrics that can be used depending on the application. After calculating the distances, the k data points with the shortest distances to the given data point are selected. These are the "k nearest neighbors". For classification, the majority class among the k nearest neighbors is determined, and the given data point is assigned to this class. For the regression, the average or weighted average of the k nearest neighbor values is calculated to obtain an estimated value for the given data point. After the algorithm has been trained, new data points can be predicted by repeating the above steps.

[0017] "Ensemble model" describes a machine learning model with at least two or more than two models, such as artificial neural networks. Such an ensemble model is used to solve more complex tasks. The individual models or networks can be linked to each other on the input or output side. The individual models or networks can be trained based on the same or possibly different training data.

[0018] "KNN detector" explanation: The k-NN algorithm is used for anomaly detection or classification of data. For example, the KNN detector can be used to identify abnormal patterns or outliers in the data. This means that the k-NN algorithm is applied to evaluate the similarity of data points in the feature space and identify deviations from the expected pattern.

[0019] "Feature vector" (also known as Featurevektor) describes the representation of a data point in the feature space. The feature vector consists of an ordered list of numerical values representing different features or attributes of the data point. These features are important information that can be used to characterize or describe the data point.

[0020] The "OOD score", or "ODD value", represents the "out-of-distribution score / value" and is a measure of how well a machine learning model can identify or label inputs or data points that lie outside the range or distribution of the training data. In many machine learning or classification tasks, the model is trained to assign data points to specific classes or categories based on the features or attributes of those data points. When presented with data points that lie within the distribution of the model's training data, the model may perform well, but when presented with "out-of-distribution" data points, i.e., data points that are very different from the training data, the model may have difficulties. The OOD score is a measure of the likelihood that the model is uncertain or confident when predicting data points outside of its training data. A low OOD score indicates that the model tends to classify data points as "in-distribution", even if they are actually "out-of-distribution", which may pose a risk of misclassification. In contrast, a high OOD score indicates that the model recognizes that it is encountering data that does not match the training data and is uncertain or reserved in its predictions (zurückhaltend).

[0021] "Averaging the determined OOD scores using the KNN method for optimized OOD recognition" is understood to mean: forming the average of the determined OOD scores for optimized OOD recognition.

[0022] The test or inference data can be, for example, a test pattern or an inference pattern for which feature vectors are calculated for each artificial neural network and, if necessary, normalized (normalisieren).

[0023] Here, the KNN algorithm is adapted. Here, the KNN algorithm will be refined so that it is preferably better applicable to the field of use of the ensemble model. This adaptation is advantageous because although KNN is very effective for OOD, it cannot be directly applied in the case of using an ensemble model. Two different possible adaptation schemes are proposed here. These adaptation schemes are reflected (widerspiegeln) in the methods according to the first and second aspects and the related embodiments. In both cases, preferably an adapted KNN detector is provided for each network in the ensemble model. This adaptation is preferably carried out during the training phase of the ensemble model. Adapting the KNN detector preferably means collecting feature vectors from each model or artificial neural network.

[0024] According to the first aspect, preferably an average value of the OOD scores of the respective KNN detectors is formed. This is more robust than a single OOD score, since the KNN (and other post hoc inference methods) are based on pre-trained networks located respectively at the bottom layer. In the method according to the first aspect, preferably each artificial neural network that is part of the ensemble model is trained such that the feature vectors can be retrieved (abrufen) in a simple manner and method.

[0025] The proposed method is more robust because it combines the point values of the features or feature vectors from multiple models or artificial neural networks. This also reduces the risk of overfitting and improves the performance of the ensemble model (also with respect to generalization), such that the proposed method is better suited to handle different data sets.

[0026] In one embodiment, providing an adapted KNN detector for each of the multiple artificial networks includes:

[0027] During the training phase of the ensemble model based on the training data,

[0028] - Extracting and / or collecting the feature vectors of the training data from each of the multiple artificial neural networks; and

[0029] - Adapting a KNN detector for each of the multiple artificial neural networks based on the extracted feature vectors.

[0030] Here, adapting the KNN detector thus only means collecting the training feature vectors during the training phase of the ensemble model.

[0031] In one embodiment, calculating the feature vectors for each of the multiple artificial neural networks based on test or inference data includes:

[0032] - Normalizing the feature vectors calculated for each of the multiple artificial neural networks.

[0033] "Normalizing the feature vectors computed for each of multiple artificial neural networks" involves a step in data preprocessing that serves to: make the feature vectors computed by different neural networks into a unified or comparable form. A feature vector is a representation of a data point in a feature space. It consists of numerical values representing different features or attributes of the data point. An artificial neural network is a machine learning model composed of a hierarchy of artificial neurons and used for pattern recognition, classification, and regression. Each neural network can compute a feature vector for the input data. Normalization of the feature vector typically involves: adapting the values in the vector to ensure they have a specific scale or distribution. This helps make the feature vectors comparable and enables the model to be better trained or compared.

[0034] Normalization ensures that feature vectors from different neural networks exist in a consistent form. This means that they may have the same mean, the same standard deviation, or the same value range, allowing them to be combined or compared with each other. Normalized feature vectors can help the model work more robustly and efficiently, especially when the feature vectors come from different sources and have different scales or distributions. This is also useful in the integration of neural networks where the outputs of multiple models are combined to produce better predictions. The specific method of normalization can vary according to the specific requirements and attributes of the data.

[0035] In one embodiment, the determined OOD scores are weighted based on the performance of the corresponding artificial neural network and / or based on the performance of the corresponding adapted KNN detector before averaging.

[0036] For example, if an artificial neural network in a plurality of networks of an ensemble model has higher performance or better performance compared to one or more other networks, a higher weight is assigned to that network compared to the one or more other networks. For example, if an adapted KNN detector in the plurality of KNN detectors has higher performance or better performance compared to one or more other KNN detectors, a higher weight is assigned to that KNN detector compared to the one or more other KNN detectors.

[0037] According to a second aspect, a method for optimizing out-of-distribution (OOD) recognition using the k-nearest neighbor (KNN) method in an ensemble model having multiple artificial neural networks is described.

[0038] The method for optimization includes the following steps:

[0039] - Providing an adapted KNN detector for the ensemble model;

[0040] - Calculate a feature vector for each artificial neural network among multiple artificial neural networks based on test or inference data;

[0041] - Perform PCA on each of the calculated feature vectors so as to obtain one-dimensional feature vectors respectively;

[0042] - Average the calculated one-dimensional feature vectors respectively so as to obtain an averaged feature vector; and

[0043] - Calculate an OOD score based on the averaged feature vector by means of an adapted KNN detector.

[0044] The terms defined above apply accordingly to the method according to the second aspect, and vice versa.

[0045] Principal component analysis (PCA) is a statistical method for detecting and visualizing the structure of complex data. It is used to reduce the dimensionality of data without losing important information. PCA is used to transform data into a smaller set of new variables, which are called principal components, and these variables may have a larger variance than the original variables. Therefore, the number of features to be analyzed used by PCA is smaller than the number of features to be initially examined, so as to obtain residuals capable of making statements about anomalies in the semiconductor components to be examined. Inverse PCA can be used to reconstruct data. Then it is determined whether the reconstruction error in the original data is particularly high and where it is particularly high, so as to draw the conclusion that there are anomalies therefrom.

[0046] A "unidimensional feature vector" is understood to mean a feature vector having the same dimension in a multi-dimensional vector space.

[0047] The test or inference data can be, for example, a test pattern or an inference pattern, for which a relevant feature vector is calculated for each artificial neural network and, if necessary, normalized.

[0048] According to the second aspect, PCA with a predefined number of components is applied to each feature vector. In addition, the average value of the feature vectors calculated by each artificial neural network according to PCA is determined. Then, preferably, an adapted KNN detector is applied by using this average value. In the method according to the second aspect, each artificial neural network forming part of the ensemble model is preferably trained such that the feature vectors can be invoked in a simple manner and method.

[0049] The method proposed according to the second aspect is more robust because it combines the point values of features or feature vectors from multiple models or artificial neural networks. This also reduces the risk of overfitting and improves the performance of the ensemble model (also in terms of generalization), making the proposed method better suited to handle different data sets.

[0050] In one embodiment, the adapted KNN detector for the ensemble model comprises:

[0051] During the training phase of the ensemble model based on training data,

[0052] - Extracting and / or collecting the feature vectors of the training data from each of the multiple artificial neural networks; and

[0053] - Performing PCA for each of the multiple artificial neural networks based on the extracted feature vectors so as to obtain respectively one-dimensional feature vectors in this way;

[0054] - Averaging the respectively one-dimensional feature vectors; and

[0055] - Adapting the standard KNN detector of the ensemble model based on the averaged feature vectors.

[0056] "Standard KNN detector" describes an unadapted KNN detector as the KNN detector that can be used for the networks included in the ensemble model. Here, only one KNN detector or multiple KNN detectors can be used, for example, one KNN detector for each network.

[0057] In one embodiment, calculating the feature vectors for each of the multiple artificial neural networks based on test or inference data comprises:

[0058] - Normalizing the feature vectors calculated for each of the multiple artificial neural networks.

[0059] In one embodiment, performing the corresponding PCA with a predefined number of components.

[0060] PCA (Principal Component Analysis) aims to reduce the dimension of the data set while retaining as much variance of the data set as possible. This embodiment describes a specific implementation of PCA, where the number of principal components extracted from the original data is preset in advance. These are the components that explain most of the variance in the data. The predefined number is preferably a parameter set before performing PCA, for example, a parameter set based on criteria such as variance explanation, computational complexity, and / or other specific requirements of the application scenario.

[0061] In one embodiment, a predefined number of components are selected based on hyperparameters of multiple artificial neural networks.

[0062] In the present context, "hyperparameters" are preferably understood to mean settings or configurations that are not learned from the data itself but must be set before model training. They affect the way the model or network is trained and can significantly influence the performance and behavior of the model.

[0063] In one embodiment, the number of components is less than the minimum eigenvector of each of the multiple artificial neural networks.

[0064] This is advantageous because the reduction (Verschlankung) of the feature space achieved by PCA thus also takes effect on the eigenvector with the fewest dimensions.

[0065] In one embodiment, the method has the following steps: optimally performing OOD recognition using the KNN method based on the averaged OOD score, and identifying samples in the training and / or inference data of the automated functions and / or driving functions of motor vehicles and / or drones and / or robots, which are in particular significantly deviated from the training and / or inference distribution in the context.

[0066] In one embodiment, the method further includes the following steps: performing optimized OOD recognition using the KNN method based on the averaged OOD score, and identifying scenarios in the training and / or inference data that lead to incorrect predictions by the ensemble model to avoid the occurrence of such scenarios.

[0067] For example, once one or more images (depending on the application) are identified as OOD in the training and / or inference data and are in particular confirmed as OOD by an expert, different strategies can be applied, such as retraining the images by including the newly identified image classes, or checking whether the outliers come from problems in the production line.

[0068] In one embodiment, the method further includes the following steps: performing optimized OOD recognition using the KNN method based on the averaged OOD score, and filtering the training and / or inference data identified as OOD to reduce the number of warnings for OOD use cases.

[0069] For example, such outliers may come from an ensemble model that is not aware of the new outliers. This makes it easier for an expert to identify where the problems leading to the outliers may come from.

[0070] It should be understood that the above steps and other optional steps do not necessarily have to be performed in the order shown, but may also be performed in a different order. In addition, further intermediate steps may be provided. Each step may also include one or more sub-steps without departing from the scope of the method according to the present invention.

[0071] According to a third aspect, a device for optimizing out-of-distribution (OOD) recognition using the K-nearest neighbor (KNN) method in an ensemble model having multiple artificial neural networks is described. The device for optimization has an evaluation and calculation device which is set up to perform the following steps:

[0072] - Provide an adapted KNN detector for each of the multiple artificial neural networks;

[0073] - Calculate feature vectors for each of the multiple artificial neural networks based on test or inference data;

[0074] - Provide, in particular input, the respectively calculated feature vectors to the respectively adapted KNN detectors;

[0075] - Based on the respectively provided feature vectors, determine corresponding OOD scores by the respectively adapted KNN detectors; and

[0076] - Use the KNN method to average the determined OOD scores for optimized OOD recognition.

[0077] According to a fourth aspect, a device for optimizing out-of-distribution (OOD) recognition using the K-nearest neighbor (KNN) method in an ensemble model having multiple artificial neural networks is described. The device for optimization has an evaluation and calculation device which is set up to perform the following steps:

[0078] In the training phase of the ensemble model based on training data,

[0079] - Extract feature vectors of the training data from each of the multiple artificial neural networks; and

[0080] - Perform PCA for each of the multiple artificial neural networks based on the extracted feature vectors so as to obtain respectively one-dimensional feature vectors in this way;

[0081] - Average the respectively one-dimensional feature vectors; and

[0082] - Adapt the standard KNN detector of the ensemble model based on the averaged feature vectors.

[0083] The statements made with respect to the method apply mutatis mutandis to the apparatus, and vice versa. It should be understood here that linguistic modifications of features expressed in method terms can be reformulated for the system according to common linguistic practice without having to list such formulations explicitly here.

[0084] Here, a control device is also claimed, which is included for partially automated or automated driving functions in motor vehicles and / or drones and / or robotic systems and / or for industrial machines and / or for optical inspection, and on which the method can be executed in one of its embodiments.

[0085] The methods described herein are preferably used in various fields and scenarios where the ability to identify samples whose identities deviate significantly from the training distribution is important. Exemplarily, this includes at least partially autonomous vehicles and / or drones and / or robotic systems. In addition, the proposed method has proven promising in the field of automated optical inspection (AOI) for avoiding and / or identifying unfamiliar scenarios that may lead to incorrect predictions. Once one or more images are optimally identified as OOD according to the method (depending on the application) and confirmed by domain experts, different strategies can be applied, such as: retraining by including the newly identified image class, or checking whether the outliers are due to problems in the production line.

[0086] In addition, the currently proposed solution can be used as a filter in its two alternatives to reduce the number of warnings from ML models that cannot accurately identify the outliers that occur, making it easier for those skilled in the art to identify where the relevant problems come from.

[0087] Implementing an embodiment of the method in an AOI automated modeling toolbox can also be advantageous. Here, the performance of the AOI automated modeling toolbox can be improved.

[0088] A computer program having program code is also claimed here, so that when the computer program is executed on a computer, at least part of the method is executed in one of its embodiments. In other words, the computer program (product) includes instructions which, when the program is executed by a computer, cause the computer to execute the method / the steps of the method in one of its embodiments.

[0089] A computer-readable data carrier having program code of a computer program is also proposed here, so that when the computer program is executed on a computer, at least part of the method is executed in one of its embodiments. In other words, the invention relates to a computer-readable (storage) medium including instructions which, when executed by a computer, cause the computer to execute the method / the steps of the method in one of its embodiments.

[0090] The described design solutions and extension solutions can be combined with each other arbitrarily.

[0091] Other possible design solutions, extension solutions and implementation solutions of the present invention also include combinations of features of the present invention described in the previous or following embodiments that are not explicitly mentioned. Description of the Drawings

[0092] The drawings are intended to provide a further understanding of the embodiments of the present invention. They show a plurality of embodiments and are used in conjunction with the description to explain the principles and solutions of the present invention.

[0093] Other embodiments and many of the mentioned advantages are derived in terms of the drawings. The elements shown in the drawings are not necessarily shown to scale with each other.

[0094] Wherein:

[0095] Figure 1 It shows a schematic flowchart of a method according to the first aspect;

[0096] Figure 2 It shows a schematic flowchart of a method according to the second aspect;

[0097] Figure 3 It shows a schematic block diagram of the training phase of an embodiment of a method according to the first aspect;

[0098] Figure 4 It shows a schematic block diagram of the test or inference phase of an embodiment of a method according to the first aspect.

[0099] Figure 5 It shows a schematic block diagram of the training phase of an embodiment of a method according to the second aspect; and

[0100] Figure 6 It shows a schematic block diagram of the test or inference phase of an embodiment of a method according to the second aspect.

[0101] In these figures in the drawings, unless otherwise specified, the same reference numerals all represent the same or functionally identical elements, components or assemblies. Detailed Embodiments

[0102] Figure 1 It shows a schematic flowchart of a method according to the first aspect for optimizing out-of-distribution (OOD) recognition using the K-nearest neighbor (KNN) method in an integrated model with multiple artificial neural networks.

[0103] Figure 2 It shows a schematic flowchart of a method according to the second aspect for optimizing out-of-distribution (OOD) recognition using the K-nearest neighbor (KNN) method in an integrated model with multiple artificial neural networks.

[0104] The method according to the first or second aspect can be at least partially performed by the device 100 in any implementation. For this purpose, it may include components not shown in detail, such as one or more providing devices and / or at least one evaluation and calculation device. It should be understood that the providing device may be co-designed with the evaluation and calculation device or may be different therefrom. In addition, the system may include a storage device and / or an output device and / or a display device and / or an input device.

[0105] Figure 1 The computer-implemented method according to the first aspect schematically shown in at least includes the following steps:

[0106] In step S1, an adapted KNN detector is provided for each of the multiple artificial neural networks.

[0107] In step S2, for each of the multiple artificial neural networks, a feature vector is calculated based on test or inference data.

[0108] In step S3, the separately calculated feature vectors are provided, in particular input, to the separately adapted KNN detectors.

[0109] In step S4, based on the separately provided feature vectors, the corresponding OOD scores are determined by the separately adapted KNN detectors.

[0110] In step S5, the KNN method is used to average the determined OOD scores for optimized OOD recognition.

[0111] Figure 2 The computer-implemented method according to the second aspect schematically shown in at least includes the following steps:

[0112] In step S10, an adapted KNN detector for the integrated model is provided.

[0113] In step S11, for each of the multiple artificial neural networks, a feature vector is calculated based on test or inference data.

[0114] In step S12, PCA is performed on each of the calculated feature vectors so as to obtain separately one-dimensional feature vectors in this way.

[0115] In step S13, the separately calculated one-dimensional feature vectors are averaged so as to obtain an averaged feature vector.

[0116] In step S14, an OOD score is calculated based on the averaged feature vector by the adapted KNN detector.

[0117] The method according to the first or second aspect can also be executed on the control device 1000, as Figure 1 and Figure 2 schematically shown.

[0118] In Figures 3 - 6 the block diagram shown, by way of example only, it is assumed that the ensemble model has three models or artificial neural networks. Of course, ensemble models with more or fewer than three models or artificial neural networks may also occur in other embodiments.

[0119] Figure 3 A fragment of an embodiment of the method according to the first aspect is shown as a block diagram. Here, it is shown in more detail how to provide an S1-adapted KNN detector for each of the multiple artificial neural networks. This is done during the training phase of the ensemble model 3000 based on the training data 312: the adaptation or fitting of the KNN detectors 300, 302, 304 of the multiple artificial neural networks 306, 308, 310. Here, the following is done: extracting the feature vectors 314 of the training data 312 from each of the multiple artificial neural networks 306, 308, 310, and adapting the corresponding KNN detectors 300, 302, 304 for each of the multiple artificial neural networks 306, 308, 310 based on the extracted feature vectors 314.

[0120] Figure 4 A fragment of an embodiment of the method according to the first aspect is shown as a block diagram. Here, it is shown how to calculate the feature vectors 402, 404, 406 for each of the multiple artificial neural networks 306, 308, 310 based on the test or inference data 400. The separately calculated feature vectors 402, 404, 406 are provided to the separately adapted KNN detectors 300, 302, 304, and based on the separately provided feature vectors 402, 404, 406, the corresponding OOD scores are determined by the separately adapted KNN detectors 300, 302, 304. The calculated or determined OOD scores are averaged to an averaged OOD score 408 in order to enable optimized OOD recognition using the KNN method.

[0121] Figure 5A fragment of an embodiment of the present method according to the second aspect is shown as a block diagram. Here, it is shown in more detail how to provide an S10-adapted KNN detector for each of the multiple artificial neural networks. During the training phase of the ensemble model 5000 based on the training data 508, the adaptation or fitting of the KNN detectors 500 of the multiple artificial neural networks 502, 504, 506 is performed. Here, the following is done: extracting the feature vectors 510 of the training data 412 from each of the multiple artificial neural networks 506, 508, 510, and performing PCA 512 on each of the multiple artificial neural networks 502, 504, 506 based on the extracted feature vectors 510, so as to obtain the respectively one-dimensional feature vectors 514 in this way. The respectively one-dimensional feature vectors are averaged into an averaged one-way feature vector 516. Then, the standard KNN detector 518 of the ensemble model 5000v is adapted based on the averaged feature vector 516.

[0122] Figure 6 A fragment of an embodiment of the present method according to the second aspect is shown as a block diagram. Here, it is shown how to calculate the feature vectors 602, 604, 606 for each of the multiple artificial neural networks 506, 508, 510 based on the test or inference data 400. PCA 512 is performed on each of the calculated feature vectors 602, 604, 606, so as to obtain the respectively one-dimensional feature vectors 608 in this way. The respectively one-dimensional feature vectors 608 that are calculated are averaged to obtain an averaged feature vector 610. In addition, the (common) OOD score 612 is determined based on the averaged feature vector 610 by the adapted KNN detector 500.

Claims

1. A method for optimizing out-of-distribution (OOD) recognition using the K-Nearest Neighbor (KNN) method in an ensemble model having multiple artificial neural networks, The method for optimization includes the following steps: - Providing (S1) an adapted KNN detector for each of the multiple artificial neural networks; - Calculating (S2) feature vectors for each of the multiple artificial neural networks based on test or inference data; - Providing (S3) the separately calculated feature vectors to the separately adapted KNN detectors; - Determining (S4) corresponding OOD scores through the separately adapted KNN detectors based on the separately provided feature vectors; and - Averaging (S5) the determined OOD scores using the KNN method for optimized OOD recognition.

2. The method according to claim 1, wherein providing (S1) an adapted KNN detector for each of the multiple artificial neural networks includes: During the training phase of the ensemble model based on training data, - Extracting feature vectors of the training data from each of the multiple artificial neural networks; and - Adapting KNN detectors for each of the multiple artificial neural networks based on the extracted feature vectors.

3. The method according to claim 1 or 2, wherein calculating (S2) the feature vectors for each of the multiple artificial neural networks based on test or inference data includes: - Normalizing the feature vectors calculated for each of the multiple artificial neural networks.

4. The method according to any one of the preceding claims, wherein the determined OOD scores are weighted based on the performance of the corresponding artificial neural network and / or based on the performance of the corresponding adapted KNN detector before averaging (S5).

5. A method for optimizing out-of-distribution (OOD) recognition using the K-Nearest Neighbor (KNN) method in an ensemble model having multiple artificial neural networks, The method for optimization includes the following steps: - Providing (S10) an adapted KNN detector for the ensemble model; - Calculating (S11) feature vectors for each of the multiple artificial neural networks based on test or inference data; - Performing (S12) PCA on each of the calculated feature vectors so as to obtain one-dimensional feature vectors respectively; - Averaging (S13) the calculated one-dimensional feature vectors respectively so as to obtain an averaged feature vector; and - Calculating (S14) an OOD score through the adapted KNN detector based on the averaged feature vector.

6. The method according to claim 5, wherein providing (S10) the adapted KNN detector for the ensemble model includes: During the training phase of the ensemble model based on training data, - Extracting feature vectors of the training data from each of the multiple artificial neural networks; - Perform PCA for each artificial neural network in the multiple artificial neural networks based on the extracted feature vectors, so as to obtain one-dimensional feature vectors respectively in this way; - Take the average of the one-dimensional feature vectors respectively; and - Adapt the standard KNN detector of the ensemble model based on the averaged feature vectors.

7. The method according to claim 5 or 6, wherein calculating (S11) the feature vectors for each artificial neural network in the multiple artificial neural networks based on test or inference data includes: - Normalize the feature vectors calculated for each artificial neural network in the multiple artificial neural networks.

8. The method according to any one of claims 5 to 7, wherein perform the corresponding PCA with a predefined number of components, wherein preferably, the predefined number of components is selected based on the hyperparameters of the multiple artificial neural networks, and wherein preferably, the number of components is less than the minimum feature vector of each artificial neural network in the multiple artificial neural networks.

9. The method according to any one of the preceding claims, the method further comprising the steps of: Perform OOD recognition optimally using the KNN method based on the averaged OOD scores, and identify samples in the training and / or inference data of the automated functions and / or driving functions of motor vehicles and / or drones and / or robots, which are particularly significantly deviated from the training and / or inference distribution in the context.

10. The method according to any one of claims 1 to 8, the method further comprising the steps of: Perform optimized OOD recognition using the KNN method based on the averaged OOD scores, and identify scenarios in the training and / or inference data that lead to incorrect predictions by the ensemble model, so as to avoid the occurrence of such scenarios.

11. A device (100) for using the K-nearest neighbor (KNN) method to optimize out-of-distribution (OOD) recognition in an ensemble model having multiple artificial neural networks, The device (100) for optimization has an evaluation and calculation device, which is configured to perform the following steps: - Provide an adapted KNN detector for each artificial neural network in the multiple artificial neural networks; - Calculate feature vectors for each artificial neural network in the multiple artificial neural networks based on test or inference data; - Provide, in particular input, the respectively calculated feature vectors to the respectively adapted KNN detectors; - Determine the corresponding OOD scores through the respectively adapted KNN detectors based on the respectively provided feature vectors; and - Use the KNN method to average the determined OOD scores for optimized OOD recognition.

12. A device (100) for using the K-nearest neighbor (KNN) method to optimize out-of-distribution (OOD) recognition in an ensemble model having multiple artificial neural networks, The device (100) for optimization has an evaluation and calculation device, which is configured to perform the following steps: - Provide an adapted KNN detector of the ensemble model; - Calculate feature vectors for each artificial neural network in the multiple artificial neural networks based on test or inference data; - Perform PCA on each of the calculated feature vectors so as to obtain one-dimensional feature vectors respectively in this way; - Average the calculated one-dimensional feature vectors respectively to obtain an averaged feature vector; and - Calculate the OOD score based on the averaged feature vector by means of an adapted KNN detector.

13. A control device (1000) for an automated driving function of a motor vehicle, a drone, an automated function of a robot and / or for an automated optical inspection of components and / or samples, wherein the control device is designed to carry out the method according to any one of claims 1 to 10.

14. A computer program having program code for carrying out at least part of the method according to any one of claims 1 to 10 when the computer program is executed on a computer.

15. A computer-readable data carrier having program code of a computer program for carrying out at least part of the method according to any one of claims 1 to 10 when the computer program is executed on a computer.

Images