METHOD AND DEVICE FOR CLASSIFYING SENSOR DATA

DE502020009677D1Active Publication Date: 2024-11-28MICROVISION INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
DE502020009677
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-10-14
Publication Date
2024-11-28
Estimated Expiration
2040-10-14

AI Technical Summary

Technical Problem

Current methods for classifying sensor data in driverless navigation systems, particularly using neural networks, fail to reliably detect unknown scenarios that are not represented in the training data, leading to potential incorrect decisions and lack of coverage for all possible environmental conditions.

Method used

A method that calculates averaged feature vectors for each class, uses cosine similarity and Mahalanobis distance to identify anomalies, and assigns confidence values, allowing the system to recognize scenarios not covered by the training data, and automatically supplements the training data with new scenarios detected.

Benefits of technology

Enables reliable detection of unknown scenarios, prevents errors in driverless navigation by recognizing deviations from training data, and automatically updates the training data to cover new scenarios, improving the system's robustness and accuracy.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The present invention relates to a method and a device for classifying sensor data according to the independent claims. State of the art

[0002] Particularly in driverless vehicle navigation, a classification of objects detected by sensors, such as road users, is essential in order to be able to navigate based on them.

[0003] State-of-the-art technology requires that a detected object be assigned at least one predefined class. However, it is equally crucial that an object be detected if it does not fall into one of the predefined classes. Only then can it be guaranteed that a decision regarding vehicle control is not based on incorrect assumptions.

[0004] This issue is even more important when neural networks are used for classification. It is known that neural networks only perform well in evaluating data that was presented to the neural network in at least a similar way during training—that is, when this data follows the distribution of the training data.

[0005] The paper KELVIN WONG ET AL: "Identifying Unknown Instances for Autonomous Driving," ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, ITHACA, NY 14853, October 24, 2019 (2019-10-24), discloses a method for detecting unknown object categories for application in autonomous vehicles. The method uses neural networks to process lidar data points.

[0006] "In the Wild: From ML Models to Pragmatic ML Systems" by Matthew Wallingford et al describes an approach to developing pragmatic ML systems.

[0007] “A Boundary Based Out-of-Distribution Classifier for Generalized Zero-Shot Learning” by Xingyu Chen et al concerns an out-of-distribution classifier. Description of the invention: task, solution, advantages

[0008] The present invention is therefore based on the object of improving a method for classifying sensor data in such a way that it can be reliably assessed whether an unknown scenario exists that is foreign to a neural network used for classification.

[0009] The aforementioned problem is solved by a method for classifying sensor data, which comprises providing a respective averaged feature vector for a plurality of classes. Thus, a separate averaged feature vector is provided for each class.

[0010] The method uses a classifier, which is a neural network trained on training data. The plurality of classes is, in particular, a closed and predefined number of classes on which the neural network was previously trained. In other words, the averaged feature vector is an average vector determined for each previously defined class.

[0011] The method comprises determining a feature vector based on sensor data using the classifier.

[0012] Furthermore, the method comprises, using the classifier, determining a cosine similarity between the previously determined feature vector and a respective averaged feature vector for the plurality of classes, comparing the respective cosine similarity with a threshold value previously determined for each class, and detecting a scenario not represented by the plurality of classes if the threshold values ​​for all classes are exceeded. The method is thus designed to detect anomalies in the classification of sensor data.

[0013] Cosine similarity is primarily a numerical value that expresses how similar vectors are, ranging from +1 to -1. For example, if the cosine similarity is +1 for a class, the corresponding feature vector exactly corresponds to the averaged feature vector of the corresponding class.

[0014] The neural network was previously trained based on training data. This training data comprises a multitude of classes. In particular, each class represents an object category, e.g., a pedestrian or a bicycle. In particular, the training data for each class includes a multitude of training samples, with the training samples representing the object represented by the class in as many different scenarios as possible. Each training sample thus comprises an object in a specific environmental condition. The training samples are preferably image data or lidar data, especially point clouds. Of course, not all possible scenarios can be covered.

[0015] A scenario is preferably defined by the object it encompasses and the environmental conditions, e.g., the weather condition. A scenario can primarily represent an object in combination with a weather condition, such as snow, rain, fog, dust, or hail, particularly depending on the severity of the weather condition, e.g., the density of the fog. A pedestrian in normal visibility (e.g., under overcast skies) represents a different scenario than a pedestrian in a hailstorm. However, both fall under the "pedestrian" class, although the "pedestrian in a hailstorm" scenario may not have been included in the training data, as such a scenario occurs extremely rarely.An object recognition method should be able to recognize an object in as many environmental conditions as possible, but should also be aware of its limitations, in the sense that the method recognizes when it cannot recognize an object with sufficient confidence based on the available scenarios because it was not represented in the training data.

[0016] A scenario not represented in the training data, in other words, an unknown or foreign scenario, can thus represent a previously unrecorded, i.e., new, object category and / or a previously unrecorded, i.e., new, environmental condition, especially weather condition, of an object category contained in the training data. If the threshold value of at least one class is exceeded, the method can include assigning the corresponding class and, preferably, a confidence level. In particular, the cosine similarity can be assigned as the confidence level.

[0017] The method is capable of detecting that a scenario is present that is not covered by the multitude of classes if the cosine similarity thresholds for all classes are exceeded. The neural network is thus able to detect a deviation from the expected results based on its training. In other words, it enables a decision to be made as to whether the classification was valid or not, which can have a significant impact on the further processing of the information for driverless vehicle navigation.

[0018] The method can thus be designed to recognize a scenario of a specific class in the sensor data, but if all threshold values ​​are undershot, to recognize that the scenario is not sufficiently represented in the training data. Furthermore, special cases may also exist, for example a pothole or an accident site with vehicle parts on the road, whereby the method recognizes these as not being represented by the training data, since they represent a truly new class, in other words an object category. Also, for example, a scenario with a new object (e.g. a new road user) that was not included in the training data is a scenario not represented in the training data, since the entire class is not represented. An example would be a cargo bike, which resembles a bicycle but nevertheless represents a separate class.In all of the aforementioned cases, it is determined that the detected scenario is not represented in the training data. A scenario not represented by the multitude of classes is therefore an out-of-distribution sample or an out-of-distribution detection.

[0019] The method thus differs from prior art methods, which assign a classification to each scenario based on similarity to the classes. The present method can therefore also be used in safety-critical applications, such as driverless vehicle navigation, since the classifier is designed to identify unknown scenarios as safety-critical. This not only enables error prevention during application but also the detection of novel scenarios, allowing them to be increasingly included in the training data, thus ensuring coverage of all occurring scenarios.

[0020] Furthermore, the method can include determining a Mahalanobis distance between a covariance matrix, the averaged feature vector, and the determined feature vector based on the sensor data for each class. This is particularly useful for better detecting scenarios not considered by the multitude of classes, since the cosine similarity does not consider the distribution of the features, but only their averaged values. Therefore, the Mahalanobis distance can also be calculated.

[0021] Since the Mahalanobis distance is unbounded, which is necessary to estimate it in terms of confidence, a reference value, specifically the function value of the cumulative density function, is determined based on a χ 2< distribution. This works because it is known that the Mahalanobis distance follows the χ 2< distribution when the number of degrees of freedom corresponds to the dimension of the features. The reference value for each class is determined by determining the function value of the χ 2< distribution at the point of the respective Mahalanobis distance. In other words, the Mahalanobis distance of the class is the X value that is inserted into the χ 2< distribution. The corresponding Y value is subtracted from one to obtain the reference value for the class. The respective reference values ​​can again be compared with a threshold previously defined for each class.If the threshold values ​​for all classes are exceeded, a scenario that is not represented by the multitude of classes is detected. This double check, using cosine similarity and the Mahalanobis distance, serves to ensure that scenarios not considered in the multitude of classes are also detected.

[0022] In particular, if the threshold values ​​for all classes are exceeded, the method can label the recognized scenario as unknown compared to the training data, in other words, mark it as unknown.

[0023] Furthermore, a previously defined minimum confidence can be assigned to the detected scenario. In particular, a minimum confidence can be assigned to each class. In particular, the previously defined minimum confidence is significantly lower than the corresponding cosine similarity for each class. This minimum confidence can, for example, be zero.

[0024] The process can include saving the sensor data in which the unrepresentative scenario was detected as a supplement to the training data. This allows the training data used to train the neural network to be supplemented. Sensor data for new scenarios can be saved automatically. This eliminates the laborious manual review of scenarios when "running in" training data, which is not only time-consuming and therefore expensive, but also error-prone. When recording training data, hours of sensor data must be recorded and manually evaluated or labeled to ensure the reliability of a classification process. Naturally, "regular" scenarios are recorded more frequently than, for example, scenarios with very rare weather phenomena or with rare road users.The present method allows for the storage of sensor data from scenarios not included in the training data and, preferably, for use in a subsequent step, for example, in a subsequent update of the classifier, to train it for the new scenarios as well. Prior to this, the stored sensor data can be manually labeled to assign a new class to the scenario unknown based on the training data.

[0025] According to the invention, the sensor data is lidar data. In other words, the sensor data is, in particular, a point cloud. The method can, in particular, comprise recording the sensor data using a corresponding sensor, in particular a lidar sensor, preferably a solid-state lidar sensor. The method can include preprocessing the sensor data. In particular, a preprocessing unit can detect points in a point cloud that appear to be reflections from a common object. These related points, in other words, clusters, can be passed on to the classifier as preprocessed sensor data.

[0026] The classifier's neural network can be divided, in particular, functionally into a first unit and a second unit. The first unit and the second unit comprise different layers of the neural network, with the distinction between the first unit and the second unit being made based on their function. The first unit is designed as a feature extractor for extracting features from the sensor data. The second unit is designed as a classification unit. The first unit serves to prepare the sensor data for the second unit.

[0027] In particular, the feature extractor comprises a multi-layer perceptron (MLP) network that includes at least one multi-layer perceptron as a layer (MLP layer). Each layer contains a plurality of artificial neurons. In particular, the MLP network comprises a plurality of MLP layers. The MLP layers are preferably each activated by a non-linear function, in particular a so-called ReLU function. A ReLU function is a function that assigns zero as a Y value to a negative x-value, while otherwise representing a linear function with a slope of 1. The MLP preferably has at least 3, in particular at least 5, layers, which may include multiple sublayers.

[0028] Furthermore, the MLP network can comprise at least one fully connected layer, which can also be activated by a non-linear function, in particular a so-called ReLU function.

[0029] The MLP network is capable of processing input in the form of point clouds. The network's output is thus features, specifically in the form of vectors. The last layer, in particular, has no activation. Finally, a softmax normalization can be performed. The result—before softmax normalization—is preferably a bottleneck vector. The bottleneck vector is thus the result of the last layer of the neural network before any softmax normalization and represents a feature vector.

[0030] Alternatively, the feature extractor can also be a CNN (convolutional neural network). However, this requires that the point cloud, which is available as sensor data, be first transformed into an image representation. The CNN comprises a multitude of layers. In particular, the CNN alternates between convolutional and pooling layers. The CNN is designed to extract lines, L-features, and / or circle features. A feature can also be referred to as a feature.

[0031] The second unit of the classifier comprises at least one, preferably two or three, fully connected layers, which directly adjoin the last layer of the first unit. Preferably, the second unit consists of at least one fully connected layer. Each layer comprises a plurality of artificial neurons. The second unit serves to process the feature vectors of the first unit.

[0032] According to the invention, the method comprises training the classifier, specifically the neural network of the classifier, using the training data, wherein the training data comprises first training samples and second training samples. The second training samples are generated using a Generative Adversarial Network (GAN) comprising a generator and a discriminator, while the first training samples are based on the recording of scenarios using a sensor. The first training data thus represents the typical training data acquired using corresponding sensors, i.e., recorded in actual situations. The first training samples thus comprise a label, in particular one assigned manually.

[0033] Specifically, the classifier is based on a cross-entropy loss based on the first training samples combined with a Kullback-Leibler (KL) divergence based on the second training samples. The KL divergence calculates the difference between two distributions, and by adding the divergence to a loss, the distributions are adjusted accordingly. The KL divergence is added to the loss. The goal is for the classifier to detect objects from the second training samples with a uniformly distributed probability across all classes, so that, in the application case, the cosine similarity of these objects with all averaged feature vectors of the first training samples is below the corresponding thresholds.Training is based on the manual labels of the first training samples and the labels generated by the classifier. These labels are compared using cross entropy loss, the additive KL divergence, and an adjustment of the weights of the classifier's neural network. Since the averaged feature vectors are not available during training, training is based on uncertainties generated by softmax normalization. The goal of training is to distribute the uncertainties equally across all classes in the second training samples, so that in the application case—i.e., after training is complete—for a scenario not represented in the training data, the cosine similarities are below the thresholds and are thus identified as such.

[0034] Both the generator and the discriminator of the GAN can be artificial neural networks. The generator generates so-called candidates, in this case second training samples, which the discriminator evaluates. The goal of the generator is to learn to produce results according to a specific distribution. The discriminator, on the other hand, is trained to distinguish the generator's results from the data from the true, predefined distribution. The generator's objective is to produce results that the discriminator cannot distinguish. In this way, second training samples are generated that lie on the decision boundaries of the classifier. The classifier evaluates the results with a uniformly distributed probability. Therefore, the KL divergence is also added additively to the generator loss.

[0035] The method specifically involves training the GAN, especially the generator and discriminator, with the classifier and GAN being trained jointly. Furthermore, the feature extractor can also be trained jointly.

[0036] The generator is trained using a binary cross entropy loss on the second training samples, plus a KL divergence based on the second training samples. The second training samples are evaluated using the discriminator. The discriminator initially evaluates the first training samples, for example, with a one, while second training samples are evaluated differently, for example, with a zero. The binary cross entropy loss refers to the evaluation of the second training samples by the discriminator. The KL divergence is based on the classifier's classification of the second training samples compared to a uniformly distributed probability. The generator's weights are adjusted during training so that it produces second training samples that the discriminator evaluates the same as the first training samples, but the classifier assigns a uniformly distributed probability to all classes.

[0037] The discriminator is also trained using a cross-entropy loss, specifically binary cross-entropy loss, based on the first training samples and the second training samples generated by the generator. The joint training of the classifier and the GAN ensures that the generator not only attempts to deceive the discriminator but also generates unknown scenarios for the classifier, which the classifier evaluates with a uniformly distributed probability. This significantly increases the reliability of recognizing unknown scenarios. Here, too, the weights of the neural networks are adjusted during training.

[0038] The discriminator can, in particular, have at least one, preferably two, convolutional layers, and be activated with a ReLU function, especially a leaky ReLU function. A leaky ReLU function is a function that assigns an x-value as a y-value a small value greater than zero (e.g., 0.01x), while otherwise representing a linear function with a slope of 1. Finally, a fully connected layer with a single output neuron is preferably used. In particular, the discriminator is designed as follows:

[0039] The generator can consist of a fully connected layer with a batch normalization and activation using a leaky ReLU. This can be followed by three transposed convolutional blocks, each with a 5x5 kernel, a batch normalization, and a leaky ReLU activation. The final layer can consist of a transposed convolutional layer, preferably again with a 5x5 kernel size.

[0040] In particular, the generator is designed as follows:

[0041] Training can be performed primarily as follows: First, the discriminator's loss, namely the cross-entropy loss, is generated to distinguish between the first training data and the second training data, which originate from the generator. Thus, the discriminator loss is defined as the sum of the binary cross-entropy loss of the first training samples and the generated second training samples. D in stands for the first training sample. "noise" is a 100-dimensional random vector generated from a standard Gaussian distribution (mean=0, stddev=1). This random vector is the input for the generator. Loss d = BCE disc D in , 1 _ + BCE disc gen noise , 0 _

[0042] The generator is trained not only to deceive the discriminator but also to generate second training samples at the classifier's decision boundaries. Thus, the loss of the generator is defined as: Loss g = BCE disc gen noise , 1 _ + KL cls gen noise ‖ 1 c , where KL stands for the KL divergence, |c| is the number of classes, and 1 / |c| represents the uniformly distributed probability across the classes. In the manner described above, the corresponding weights of the neural networks are adjusted to deceive the discriminator into believing that the second training samples follow the distribution of the first training sample and to achieve an equal distribution of the classifier's probabilities across the second training samples. This is important because it ensures that an unknown scenario falls below all thresholds of the plurality of classes.

[0043] The loss of the classifier is defined as: Loss cls = CE cls D in , labels ) + KL cls gen noise ‖ 1 c

[0044] All weights are initialized with the Xavier Uniform Initializer.

[0045] Advantageously, an averaged feature vector is determined for each class, wherein the training data comprises a number of first training samples per class, and wherein the averaged feature vector of a class is determined by the following steps: First, a feature vector is determined for each first training sample of a class, in particular using the classifier (before softmax normalization). This is in particular the bottleneck vector. These feature vectors are added to obtain an added feature vector. This is divided by the number of first training samples of the corresponding class. The result is the averaged feature vector of a class. The averaged feature vector is in particular the averaged bottleneck vector of the corresponding class. The averaged feature vectors are determined in particular after training the classifier.

[0046] According to the invention, the threshold value is determined for each class, whereby the threshold value of a class is determined through the following steps: First, the cosine similarity of the feature vectors of each first training sample with the averaged feature vector of the class is determined. The determined cosine similarities are then either averaged, thus calculating an averaged cosine similarity, or the smallest cosine similarity is determined from the set of cosine similarities of the first training data of the class. The threshold value is set either as the averaged cosine similarity or the smallest cosine similarity.

[0047] For example, the threshold may have been set to 0.85. If a cosine similarity of 0.92 is determined between a detected feature vector and the averaged feature vector, the corresponding class can be output with high confidence, thus confirming a valid detection.

[0048] In particular, the present method for classifying sensor data is part of a method for driverless navigation of a vehicle.

[0049] In a further aspect, the invention comprises a device for classifying sensor data, wherein the device comprises a classifier and is configured to carry out a method described above. In particular, the classifier is trained as described above. Furthermore, the device can comprise a GAN described above, which is trained in particular as described above. The classifier can in particular comprise a decision module that assigns a class and / or a confidence and / or a label as unknown based on the threshold values ​​and the determined cosine similarities and preferably Mahalanobis distances. In particular, the device serves for the navigation of a driverless vehicle.

[0050] Furthermore, the invention relates to a computer-readable storage medium on which a program is stored which, after being loaded into the memory of the computer, enables a computer to carry out a method for classifying sensor data as described above in conjunction with a device as described above.

[0051] Furthermore, the invention comprises a computer program product comprising a computer-readable storage medium as described above. Short description of the characters

[0052] They show in purely schematic representation Figure 1: a process diagram of a method according to the invention, Figure 2: a training of a classifier and a GAN, and Figure 3: a determination of an averaged feature vector. Preferred embodiments of the invention

[0053] Figure 1shows a process diagram of a method 100 according to the invention for classifying objects. The method 100 comprises providing 110 a respective averaged feature vector 30 for a plurality of classes. A classifier 10 determines 120 a feature vector based on sensor data. A cosine similarity between the determined feature vector and the respective averaged feature vectors for all classes is determined 130, and the cosine similarities are compared 131 with a previously determined threshold value. If the threshold values ​​for all classes are undershot, a scenario not represented by the plurality of classes is detected 133.

[0054] The method 100 may, in particular, comprise the training 101 of the classifier 10 and the training 102 of the GAN 20. This is particularly carried out jointly.

[0055] Furthermore, the respective averaged feature vectors 30 for the plurality of classes can be determined 111. For this purpose, in particular, a feature vector is determined 111a for each first training sample of a respective class, the corresponding feature vectors of all first training samples for the respective class are added 111b and divided by the number of first training samples of the class 111c.

[0056] Furthermore, the respective threshold value for the plurality of classes can be determined 112. This is done in particular by determining 112a the cosine similarities of the feature vector of each first training sample with the averaged feature vector 30 of the class. The averaged cosine similarity or the smallest cosine similarity is determined 112b. The averaged cosine similarity is calculated in particular by adding the cosine similarities of all first training samples and dividing by the number of first training samples. The smallest cosine similarity is determined simply by determining the smallest value of the cosine similarities. The averaged cosine similarity or the smallest cosine similarity is set as the threshold value 112c.

[0057] As a further parameter for determining whether the sensor data includes a scenario represented in the training data or not, a Mahalanobis distance can be determined between a covariance matrix of the averaged feature vector and the determined feature vector 132. Based on the Mahalanobis distance and the χ 2< distribution, a reference value can be determined for each class, which is compared with the respective threshold value of the class. The covariance matrix for each class can be determined beforehand 113.

[0058] If the threshold values ​​for all classes are exceeded, the detected scenario can be labelled as unknown compared to the training data 150, a minimum confidence can be assigned to the scenario 151 and / or the corresponding sensor data can be saved as a supplement to the training data 152.

[0059] Figure 2shows the training of a classifier 10 and a GAN 20, wherein the GAN 20 comprises a generator 21 and a discriminator 22. First training samples 51 are provided as input for the discriminator 22, and second training samples 52 are provided by the generator 21. The generator 21 generates second training samples 52 at the decision boundaries of the classifier 10, thus attempting to deceive the discriminator 22 and have the classifier 10 assign a uniformly distributed probability to the second training samples 52.

[0060] Both the classifier 10 and the generator 21 and the discriminator 22 are neural networks 11. The classifier 10 is illustrated as an example of neurons 12 arranged in layers. The neural network 11 of the classifier 10 can be functionally divided into two units: a first unit 11a and a second unit 11b. The first unit 11a is configured as a feature extractor, while the second unit 11b serves as a classification unit.

[0061] To the right of classifier 10, the uncertainties 60 determined by softmax normalization 31 are presented in tabular form, divided among the numerous classes. The upper section shows the uncertainties 60 of softmax normalization 31 for an initial training sample, with the first class by far the highest, so that at least this class is assigned with a high degree of confidence. Since no averaged feature vector is available during training, the uncertainties obtained after softmax normalization are used.

[0062] The situation is different in the example below, which shows a second training sample in which the uncertainties 60 of the softmax normalization 31 are evenly distributed for all classes.

[0063] Figure 3shows how, after training, the classifier 10 determines an averaged feature vector 30, specifically before a softmax normalization 31. This occurs in the last layer of the second unit, i.e., the classifier, before the softmax normalization 31, based on all the first training samples 51, correspondingly for each class. In the application case, the cosine similarity is then calculated based on sensor data and the averaged feature vectors 30 for the classes. Figure 1 certainly. Examples of neural networks

[0064] The feature extractor can be structured as follows: Execution as MLP: Input: Batchsize × 2048 × 3 Layer 0: ∘ Layer 0.1: Multi Layer Perceptrons (MLP) ▪ Number of functions: 64 ▪ Kernel 1x3 ▪ Batch normalization ▪ Activation function: ReLU ∘ Layer 0.2: MLPs ▪ Number of functions: 128 ▪ Kernel 1x1 ▪ Batch normalization ▪ Activation function: ReLU ∘ Layer 0.3: MLPs ▪ Number of functions: 1024 ▪ Kernel1x1 ▪ Batch normalization ▪ Activation function: ReLU ∘ Layer 0.4: Fully Connected ▪ Output: 512 ▪ Batch normalization ▪ Activation function: ReLU o Layer 0.5: Fully Connected ▪ Output: 256 ▪ Batch normalization ▪ Activation function: ReLU ∘ Layer 0.5: Fully Connected ▪ Output: 9 ▪ No activation function ▪ Initialization of the variables with 0 ∘ Layer 0.6: Addition with identity matrix [1 0 0 0 1 0 0 0 1] ∘ Output: Batch size × 3 × 3 Layer 1: Input * Output Layer 0 Layer 2: MLPs ∘ Number of functions: 64 ∘ Kernel 1x3 ∘ Batch normalization ∘ Activation function: ReLU Layer 3: MLPs ∘ Number of functions: 64 ∘ Kernel 1×1 ∘ Batch normalization ∘Activation function: ReLU Layer 4: Learning matrix for multiplication ∘ Layer 4.1: MLP ▪ Number of functions: 64 ▪ Kernel1x1 ▪ Batch normalization ▪ Activation function: ReLU ∘ Layer 4.2: MLPs ▪ Number of functions: 128 ▪ Kernel 1x1 ▪ Batch normalization ▪ Activation function: ReLU ∘ Layer 4.3: MLPs ▪ Number of functions: 1024 ▪ Kernel1x1 ▪ Batch normalization ▪ Activation function: ReLU ∘ Layer 4.4: Fully Connected ▪ Output: 512 ▪ Batch normalization ▪ Activation function: ReLU ∘ Layer 4.5: Fully Connected ▪ Output: 256 ▪ Batch normalization ▪ Activation function: ReLU ∘ Layer 4.5: Fully Connected ▪ Output: 4096 ▪ No activation function ▪ Initialization of the variables with 0 ∘ Layer 4.6: Addition with identity matrix (size 64) ∘ Output: Batchsize × 64 × 64 Layer 5: Layer 3 * Output Layer 4 Layer 6: MLPs ∘ Number of functions: 64 ∘ Kernel 1×1 ∘ Batch normalization ∘ Activation function: ReLU Layer 7: MLPs ∘ Number of functions: 128 ∘ Kernel 1×1 ∘ Batch normalization ∘Activation function: ReLU Layer 8: MLPs ∘ Number of functions: 512 ∘ Kernel 1×1 ∘ Batch normalization ∘ Activation function: ReLU Layer 9: Classification unit ∘ Layer 9.1 Fully Connected Layer: ▪ Number of output neurons: 512 ▪ Activation function: ReLU ∘ Layer 9.2: Dropout ▪ Dropout rate: 0.5 ∘ Layer 9.3 Fully Connected Layer: ▪ Number of output neurons: 512 ▪ Activation function: ReLU ∘ Layer 9.4: Dropout ▪ Dropout rate: 0.5 ∘ Layer 9.5: Fully Connected Layer (= bottle neck vector): ▪ Number of output neurons: Number of classes to be classified GGF Softmax normalization layer Execution as CNN Input: Batchsize × 80 × 128 × 2 (2 channels for solid state: intensity and depth) Layer 1: Convolution ∘ Kernelsize: 3x3 ∘ Number of output feature maps: 64 ∘ Activation: ReLU Layer 2: Convolution ∘ Kernelsize: 3x3 ∘ Number of output feature maps: 64 ∘ Activation: ReLU Layer 3: Max Pooling ∘ Kernelsize: 2x2 Layer 4: Convolution ∘ Kernelsize: 3x3 ∘ Number of output feature maps: 128 ∘ Activation:ReLU Layer 5: Convolution ∘ Kernel size: 3x3 ∘ Number of output feature maps: 128 ∘ Activation: ReLU Layer 6: Max Pooling ∘ Kernel size: 2x2 Layer 7: Convolution ∘ Kernel size: 3x3 ∘ Number of output feature maps: 256 ∘ Activation: ReLU Layer 8: Convolution ∘ Kernel size: 3x3 ∘ Number of output feature maps: 256 ∘ Activation: ReLU Layer 9: Max Pooling ∘ Kernel size: 2x2 Layer 10: Convolution ∘ Kernel size: 3x3 ∘ Number of output feature maps: 512 ∘ Activation: ReLU Layer 11: Convolution ∘ Kernel size: 3x3 ∘ Number of output feature maps: 512 ∘ Activation: ReLU Layer 12: Max Pooling ∘ Kernel size: 2x2 Layer 13: Convolution ∘ Kernel size: 3x3 ∘ Number of output feature maps: 512 ∘ Activation: ReLU Layer 14: Convolution ∘ Kernel size: 3x3 ∘ Number of output feature maps: 512 ∘ Activation: ReLU Layer 15: Max Pooling ∘ Kernel size: 2x2 Layer 16: Flattening of features (preparation, in other words adaptation of the dimension, as input for the classifier) ​​Layer 17: Classification unit ∘ Layer 17.1 Fully ConnectedLayer: ▪ Number of output neurons: 512 ▪ Activation function: ReLU ∘ Layer 17.2: Dropout ▪ Dropout rate: 0.5 ∘ Layer 17.3 Fully Connected Layer: ▪ Number of output neurons: 512 ▪ Activation function: ReLU ∘ Layer 17.4: Dropout ▪ Dropout rate: 0.5 ∘ Layer 17.5: Fully Connected Layer (= bottle neck vector): ▪ Number of output neurons: Number of classes to be classified GGF Normalization with Softmax Normalization

[0065] The training of the classifier and the GAN can be determined by the following parameters: Optimizer: AdamOptimizer (each with its own optimizer for discriminator, generator and classifier) ​​LearningRate: 0.0001 ∘ Exponential decay: Staircase ▪ The learning rate decreases like a step function. After 200,000 steps, it decreases by half until it reaches a minimum of 0.00001, after which it remains constant Batchsize: 32 Initialization of all variables (except initialization with 0): Xavier Initialization Total epochs: 100 List of reference symbols

[0066] 10Classifier 11Neural network 11aFirst unit 11bSecond unit 12Neuron 20GAN 21Generator 22Discriminator 30averaged feature vector 31Softmax normalization 50Training data 51First training samples 52Second training samples 60Uncertainties 100 procedures 101Training the Classifier 102Training the GAN 110Providing a respective averaged feature vector for a plurality of classes 111Determining a respective averaged feature vector for the plurality of classes 111aDetermining a feature vector for each first training sample of a class 111bAdding the feature vectors of all first training samples of a class to obtain an added feature vector 111cDividing the added feature vector by the number of first training samples of the class 112Determining a respective threshold value for the plurality of classes 112aDetermining the cosine similarity of the feature vector of each first training sample with the averaged feature vector of the class 112bDetermining the averaged cosine similarity or the smallest cosine similarity of the first training data of the class 112cDefining the averaged cosine similarity or the smallest cosine similarity as the threshold value of the class 113Determining a covariance matrix 120Determining a feature vector based on sensor data 130Determination of a cosine similarity between the feature vector and a respective averaged feature vector for the plurality of classes 131Comparison of the respective cosine similarity with a threshold value previously determined for each class 132Determination of a Mahalanobis distance between a covariance matrix, the averaged feature vector and the determined feature vector 133Detection of a scenario not represented by the plurality of classes when the threshold values ​​for all classes are exceeded 150If the threshold values ​​for all classes are exceeded, label the detected scenario as unknown compared to the training data 151If the threshold values ​​for all classes are exceeded, assign a previously defined minimum confidence to the detected scenario 152Save the sensor data as a supplement to the training data

Claims

1. A method (100) for classifying sensor data which comprises providing (110) a respective averaged feature vector (30) for a plurality of classes, wherein the method (100) comprises the following steps carried out by means of a classifier (10), wherein the classifier (10) comprises at least one neural network (11) trained on the basis of training data (50): • determining (120) a feature vector on the basis of sensor data, wherein the sensor data is lidar data, • respective determining (130) a cosine similarity between the feature vector and the respective averaged feature vector (30) for the plurality of classes, • comparing (131) the respective cosine similarity with a threshold value established for each class beforehand, and • detecting (133) a scenario not represented by the plurality of classes if the threshold values for all classes are not reached, wherein the method further comprises training (101) the classifier (10) according to the training data (50), wherein the training data (50) comprises first training samples (51) and second training samples (52), wherein the second training samples (52) are generated by means of a Generative Adversarial Network (GAN; 20), wherein the GAN comprises a generator (21) and a discriminator (22), wherein the first training samples (51) are based on recording scenarios by means of a sensor, wherein the threshold value is determined (112) for each class, wherein the threshold value of a class is determined by the following steps: • determining (112a) the cosine similarity of the feature vector of each first training sample with the averaged feature vector of the class, • determining (112b) the averaged cosine similarity or the smallest cosine similarity of the first training data of the class; • establishing (112c) the averaged cosine similarity or the smallest cosine similarity as the threshold value of the class.

2. The method (100) according to claim 1, characterized in that the classifier (10) further determines (132) a Mahalanobis distance between a covariance matrix and the averaged feature vector (30) and the determined feature vector for each class.

3. The method (100) according to claim 2, characterized in that a reference value is determined for each class on the basis of the Mahalanobis distance and the χ2 distribution, wherein the reference value is compared with a threshold value established for each class, wherein the method (100) comprises detecting (133) a scenario not represented by the plurality of classes if the threshold values for all classes are not reached.

4. The method (100) according to one of the preceding claims, characterized in that if the threshold values for all classes are not reached, the detected scenario is labelled (150) as unknown compared to the training data (50).

5. The method (100) according to one of the preceding claims, characterized in that if the threshold values for all classes are not reached, a previously defined minimum confidence level is assigned to the detected scenario (151).

6. The method (100) according to one of the preceding claims, characterized in that the method comprises storing (152) the sensor data as a supplement to the training data (50).

7. The method (100) according to claim 1, characterized in that the training (101) of the classifier (10) is carried out on the basis of a cross entropy loss on the basis of the first training samples (51) added with a Kullback-Leibler divergence on the basis of the second training samples (52).

8. The method (100) according to one of claims 1 or 7, characterized in that the method comprises training (102) the GAN (20), namely the generator (21) and the discriminator (22), wherein the training (101) of the classifier (10) and the training (102) of the GAN (20) are carried out together.

9. The method (100) according to claim 8, characterized in that the training of the generator (21) is performed on the basis of a binary cross entropy loss on the basis of the second training samples (52) added with a Kullback-Leibler divergence on the basis of the second training samples (52).

10. The method (100) according to one of claims 1 or 7 to 9, characterized in that the averaged feature vector (30) is determined for each class (111), wherein the training data (50) comprises a number of first training samples (51) per class, wherein the averaged feature vector (30) of a class is determined by the following steps: • determining (111a) a feature vector for each first training sample (51) of a class, • adding (111b) the feature vectors of all first training samples (51) of a class to obtain an added feature vector • dividing (111c) the added feature vector by the number of the first training samples of the class.

11. Device for classifying sensor data, characterized in that the device comprises a classifier (10) and is configured to carry out a method (100) according to one of claims 1 to 10.

12. A computer-readable storage medium on which is stored a program which, after being loaded into the memory of the computer, enables a computer to carry out a method according to one of claims 1 to 10 in cooperation with a device according to claim 11.

13. A computer program product comprising a computer-readable storage medium according to claim 12.