Computer-implemented method for deriving a data set for training a classifier and / or checking a classifier performance

Abstract

The invention relates to a computer-implemented method for training a classifier and / or checking a performance of a classifier, a computer program product, a machine-readable storage medium and a device, the method comprising the steps of: providing a first data set of image data; training an encoder and a decoder such that data points of the first data set are mapped into a low-dimensional space by means of the encoder and inverse transformed again by means of the decoder; computing a first data representation, wherein the first data representation is computed by applying the encoder to data points of the first data set; computing an additional data representation within a convex hull of the computed first data representation; determining an inverse-transformed data set by applying the decoder to the additional data representation; computing a second data set based on the inverse-transformed data set; training a classifier by means of the second data set and / or checking a performance of a classifier.

Description

Technical Field

[0001] The present invention relates to a computer-implemented method for obtaining a dataset to train a classifier and / or check the performance of a classifier, an apparatus for implementing the method, a computer program for implementing the method, and a machine-readable storage medium on which the computer program is stored. Background Technology

[0002] Computer-implemented machine learning methods, especially neural networks, are frequently used as part of the environmental recognition of partially automated, highly automated, or fully automated robots, particularly automated vehicles. However, the algorithms used for this purpose are not perfect and can produce (more or less serious) erroneous predictions, which in automated vehicles can lead to misinterpretations and system malfunctions, for example.

[0003] There is a significant need to determine the following priors: under what conditions will a machine learning method output incorrect predictions? To ensure that the machine learning method does not commit any serious errors during operation, randomized validation and / or test datasets are currently used. The basic assumption is that the behavior of the machine learning method can be generalized from the validation and / or test datasets to subsequent runs. Summary of the Invention

[0004] The problem with the above approach is that it's undisclosed which validation and / or test data must be used to allow the behavior of the machine learning method to be generalized to future runs. Currently, data is selected based on semantic aspects, such as in the case of automated vehicles, based on weather phenomena, time of day, or traffic coverage. However, machine learning methods do not make decisions based on semantic aspects but rather on data-specific characteristics, such as the pixel values ​​of images from camera sensors. Therefore, to ensure the generalization of results obtained on validation and / or test datasets, these data-specific characteristics must be considered.

[0005] This invention describes a computer-implemented method for training classifiers and / or regressors, and / or for checking the performance of classifiers and / or regressors, particularly for enabling classifiers and / or regressors to control partially automated, highly automated, or fully automated robots. The method includes the following steps:

[0006] The first dataset providing image data;

[0007] By training the encoder and decoder in this way, the encoder can map the data points of the first dataset to a low-dimensional space, and the decoder can map them back to the first dataset.

[0008] The first data representation is obtained by applying the encoder to the data points of the first dataset.

[0009] Obtain the additional data representation within the convex hull of the first data representation;

[0010] The inversely transformed dataset is determined by applying the decoder to the additional data representation;

[0011] The second dataset is obtained based on the dataset after inverse transformation;

[0012] Use the second dataset to train the classifier and / or regressor, and / or check the performance of the classifier and / or regressor.

[0013] Robots can be understood, for example, as industrial robots, automated work machines, or automated vehicles. In particular, robots can be understood as partially automated, highly automated, or fully automated vehicles that can perform driving operations, especially the matching of longitudinal and / or lateral movements, at least temporarily without human intervention.

[0014] A classifier can be understood as a computer-implemented machine learning method, particularly a neural network, constructed such that it can classify a given image data item, or parts thereof, into one or more categories in a set of categories. For example, a classifier can determine whether a vehicle can be seen in a given image. However, multiple classifications can also be performed. For instance, in the case of an object classifier, it can determine whether a region contains a vehicle for different areas of an image data item.

[0015] Generally, classifiers may output incorrect predictions. For example, a classifier may predict the wrong category.

[0016] The inverse transform is specifically understood as an inverse transform to the original space, from which the data points of the first dataset originate. Typically, this original space is a higher-dimensional space.

[0017] To determine the extent to which a classifier is functioning correctly, annotated data samples can be used. These data samples can be analyzed to process different metrics, allowing for empirical evaluation of the classifier's performance. For a classifier, classification accuracy, for example, can be used. Classification accuracy is understood as the ratio of correct classifications to the sum of all correct and incorrect classifications.

[0018] The first dataset required for this method can be provided in different ways. For example, it can be extracted from a database of image data, wherein the image data in the database has semantic attributes that reflect the content of the corresponding image data items. In the case of highly automated or fully automated vehicles, these semantic attributes can, for example, describe the different weather conditions, road materials or road conditions, time of day, or types of other traffic participants mapped in the corresponding image data items.

[0019] However, suitable datasets can also be obtained from other sources. For example, for many applications, especially for highly automated and fully automated vehicles, there are image datasets available from the Internet. These image datasets can also be used in this method.

[0020] If a suitable dataset is not available, the corresponding image data can be recorded directly.

[0021] For the first dataset, different types of image data can be considered, especially video data, LiDAR data, radar data, or ultrasound data, as well as thermal imager data. Furthermore, it can be suggested that combinations of the aforementioned sensor data can be used as image data.

[0022] To process image data, a combination of encoders and decoders is used to map the image data into a low-dimensional space. Autoencoders, especially variational autoencoders, are suitable for this purpose. An autoencoder can be understood as a neural network that takes image data as input, produces a compressed representation of the input data, and then inversely projects this compressed representation back onto the input data, making the compressed representation as similar as possible to the input data.

[0023] For training, the parameters of the autoencoder are tuned such that, for a given input data item in the dataset, the inverse projection is as similar as possible to that input data item. For example, in the case of camera-based image data, the pixel values ​​of the inverse projection should correspond as accurately as possible to the pixel values ​​of the input data item. Here, the encoder and decoder can have non-linear and linear operations to produce representations and / or inverse projections.

[0024] As an alternative to autoencoders, it can be proposed that encoders and decoders can also be given by a flow-based model. In this case, the model can be understood as an encoder and a decoder. The forward computation of the flow performed through the model can be understood as the encoding steps, while the backward computation of the flow can be understood as the decoding steps.

[0025] After the encoder and decoder have been trained accordingly, the encoder is used in the next step of the method to project the image data of the first dataset into a low-dimensional space. The resulting first data representation can be understood as a compression of the image data. Therefore, this first data representation contains the necessary information from the perspective of the encoder and decoder for generating the reconstruction of the corresponding input data items. However, gaps may exist between the first data representations, i.e., the corresponding image data that generates representations at these locations is missing from the first dataset. This additional image data can generally contain very valuable information. This information can, for example, be used to improve the predictive accuracy of the classifier. In another example, the additional data points can be used to evaluate the predictive accuracy of the classifier. This evaluation can be used, in particular, to determine whether the predictive accuracy is high enough to enable the classifier to operate as part of a safety-critical product, such as a highly automated or fully automated vehicle.

[0026] Therefore, in the proposed method, it is first determined which additional data representations should be generated. To do this, the convex hull of the first data representation is first calculated. Then, all additional data representations are calculated such that they lie within the convex hull of the first data representation. The advantage of this method is that the additional data representations thus contain meaningful information about the dataset, because they represent a linear combination of the first data representations. In the case of image data for highly automated or fully automated vehicles, for example, the following situation may occur: the first data representation includes red and yellow vehicles, but not orange vehicles. In this case, a linear combination of the corresponding first data representations can produce additional data representations mapping orange vehicles.

[0027] The additional data representation may differ significantly from the image data in the first dataset in terms of its type and appearance (e.g., dimensionality). Therefore, this additional data representation generally cannot be used directly by the classifier. To preprocess the information contained in the data representation so that it can be used by the classifier, the inversely transformed dataset is obtained in the next step of the method using a decoder. This dataset may contain information from the additional data representation, but is shown in the domain of the image data. In the example above, the color orange can be encoded as vector values ​​in the data representation and can be shown as orange pixels in the image data after the decoder is applied.

[0028] The advantage of this method lies in its ability to obtain additional image data, which can be used to determine and / or better estimate the classifier's performance. This is particularly advantageous for the use of the classifier in safety-critical applications. Although datasets are typically derived based on semantic attributes determined by humans, this method allows for the generation of a second dataset that can describe information contained in the image data that was not considered in the previously used semantic attributes.

[0029] In another embodiment of the method, the second dataset corresponds to the dataset after inverse transformation.

[0030] The advantage of this implementation is that the decoder output can be directly used as a dataset, and no other computationally intensive methods are required.

[0031] In another embodiment of the method, the second dataset consists of the selection of data points from the third dataset, wherein the data points are selected such that each data point has a minimum distance to the corresponding inversely transformed data point.

[0032] Therefore, the selected data points in the third dataset can be understood as being similar in appearance to the data points in the dataset after the inverse transformation.

[0033] For this embodiment, a third dataset is required, which can be provided in the same way as the first dataset. Next, data points in the third dataset are determined for each data point in the inversely transformed dataset, and these data points are similar to the corresponding data points in the second dataset. For this purpose, a spacing index can be used, for example. In the case of camera-based image data, for example, the absolute difference between the pixel values ​​of image data items in the supplementary dataset and the pixel values ​​of image data items in the third dataset can be used as an index to determine which image data in the third dataset are most similar to the corresponding image data in the second dataset.

[0034] Optionally, in this example, it can also be specified that the absolute pixel difference must be below a predefined threshold. If this is not the case, additional image data items can be added to the third dataset, which have a sufficiently small absolute pixel difference to the corresponding image data items in the additional dataset.

[0035] The advantage of this implementation is that the second dataset consists of real image data rather than synthetically generated image data. This can be advantageous for training classifiers and / or regressors, as real data can, in some cases, compensate for smaller errors in the decoder's projection.

[0036] In another embodiment of the method, an additional data representation is generated by a weighted sum of selections of two or more initial data representations, such that the additional data representation exceeds a predefined spacing threshold to all initial data representations and / or other additional data representations.

[0037] The advantage of this implementation is that it can generate data representations in a structured manner, which can compensate for gaps in the space of the data representation.

[0038] In another embodiment, it may be proposed to iteratively generate additional data representations randomly within the convex hull of the first data representation, wherein, in each iteration step, the additional data representations generated exceed a predefined spacing threshold up to all first data representations and the additional data representations generated so far.

[0039] The advantage of this method is that it requires less computation time compared to generating additional data representations in a structured manner. This means that more data representations can be generated over a constant amount of time, thus making the second dataset larger. Therefore, a classifier trained on this dataset can extract more information from the training data because there is more data. This allows for higher classification accuracy.

[0040] In addition, a computer program configured to implement one of the above methods is claimed, and a machine-readable storage medium on which the computer program is stored is also claimed.

[0041] In addition, protection is claimed for a device configured to implement one of the methods based on the present invention.

[0042] The classifiers or regressors trained and / or tested in this application are particularly capable of being used for driving or control functions of robots, especially at least partially automated vehicles. Based on the results of the classifiers and / or regressors, robot manipulation is particularly possible. For example, interventions can be made to control the robot's longitudinal and / or lateral movements based on the results. Attached Figure Description

[0043] Figure 1 A schematic flowchart of the proposed method is shown;

[0044] Figure 2 The training of the encoder and decoder is demonstrated exemplarily;

[0045] Figure 3 The method for generating additional datasets is shown exemplarily. Detailed Implementation

[0046] In the first embodiment, a computer-implemented method is used to generate a dataset of image data for training a classifier. This method... Figure 1 It is shown schematically in the diagram.

[0047] In step 101, a first dataset of camera images is provided. For this embodiment, a highly automated driving application scenario is selected. Furthermore, in this embodiment, the dataset required for the method is extracted from a database of camera images with semantic attributes for highly automated vehicles. Here, semantic attributes represent the characteristics of the environment in which the vehicle operates. The dataset is selected such that it covers all environmental characteristics appearing in the database.

[0048] In step 102, a variational autoencoder (VAE) is trained using the dataset from step 101. To do this, the data from the first dataset is projected into the latent space of the VAE using the encoder portion of the VAE. The resulting data representation is then inversely transformed using the decoder. The encoder and decoder parameters are then matched to minimize the error between the data from the first dataset and the inversely projected data. This process of projection and inverse transformation in the latent space is continued until the difference in error between successive iterations is no longer significant.

[0049] In step 103, the encoder of the VAE trained in step 102 is used to generate a data representation of the first dataset. In this embodiment, the data representation is the latent space representation of the image data of the first dataset.

[0050] In step 104, additional data representations are obtained in addition to the data representations obtained in step 103. For this purpose, paired elements of the data representations obtained in step 103 can be used. For example, two data representations can be randomly selected separately. Alternatively, it can be proposed that all pairs of data representations be used.

[0051] Then, data representations can be obtained for this pair by linearly interpolating between the corresponding data representations of the first dataset. The portions of the data representation that can be used as additional data representations can then be obtained at intervals that are optionally equal along the interpolation direction.

[0052] Based on the additional data representation generated in step 104, the inversely transformed dataset can be obtained through the decoder in step 105. For this purpose, the decoder can be applied to the additional data representation. The output obtained through this application then forms the inversely transformed dataset.

[0053] In step 106, a second dataset is obtained based on the inversely transformed dataset generated in step 105. For this purpose, image data recorded by a camera from an image database can be used, for example. For example, it is conceivable that for at least a subset of data points in the inversely transformed dataset, images most similar to the corresponding data points are determined in the image database. For example, it is conceivable that for data points in the inversely transformed dataset, images with the minimum pixel deviation from the data points are obtained from the image database.

[0054] Alternatively, a second dataset can be derived using a GAN (Generative Adversarial Network) based on the inversely transformed dataset. Preferably, a recurrent GAN can be trained such that it can transform the inversely transformed images from the decoder into images similar to those in the image database.

[0055] Then, in the final step 107, the classifier can be trained using the second dataset. Alternatively, it can be proposed to use the dataset to check the classification accuracy of the classifier.

[0056] Figure 2 The structure for training an autoencoder is schematically illustrated, comprising an encoder 301 and a decoder 303. In another embodiment, a dataset 201 is provided for training. The encoder 301 obtains a set 302 of low-dimensional representations of the dataset 201, which is then transformed again by the decoder 303 into an inversely transformed representation 304. For training, the weights of the autoencoder are thus adapted such that the differences between images in the dataset gradually decrease in the inversely transformed representation.

[0057] Figure 3 The acquisition 106 of the second dataset in another embodiment is illustrated schematically. An encoder 301 trained according to the previous example is used to acquire a set of low-dimensional representations 401 for the first dataset 201. Then, based on the low-dimensional representations 401, an additional representation 402 is acquired within the convex hull of the low-dimensional representations 401. In this embodiment, the additional representation 402 is acquired through interpolation corresponding to the first embodiment.

[0058] Then, the additional representation 402 can be transformed using the decoder 303 to obtain the inversely transformed dataset 403. As described in the first embodiment, a second dataset can be obtained based on this inversely transformed dataset.

[0059] In another embodiment, a classifier is used to identify objects in driving functions. Based on the classification results, the vehicle, which is at least partially automated, is controlled.

Claims

1. A computer-implemented method for training a classifier and / or for checking the performance of a classifier, the computer-implemented method comprising the following steps: The first dataset (201) provides image data; The encoder (301) and decoder (303) are trained in such a way that the data points of the first dataset (201) are mapped to a low-dimensional space (302) by means of the encoder (301), and then inversely transformed by means of the decoder (303); The first data representation (401) is obtained by applying the encoder (301) to the data points of the first dataset (201). Obtain the additional data representation (402) within the convex hull of the first data representation (401). The inversely transformed dataset (403) is determined by applying the decoder (303) to the additional data representation (402); The second dataset is obtained based on the dataset after the inverse transformation; The second dataset is used to train a classifier and / or check the classifier's performance.

2. The method according to claim 1, wherein, The second dataset corresponds to the dataset (403) after the inverse transformation.

3. The method according to claim 1, wherein, The second dataset consists of the selection of data points from the third dataset, wherein the selection of the data points is performed such that the data points are selected from the third dataset such that each data point has a minimum distance to the corresponding inversely transformed data point.

4. The method according to claim 1, wherein, The second dataset is generated by applying a generative adversarial network (GAN) to the inversely transformed dataset.

5. The method according to any one of claims 1 to 4, wherein, The encoder (301) and the decoder (303) are part of an auto encoder.

6. The method according to claim 5, wherein, The autoencoder is a variational autoencoder.

7. The method according to any one of claims 1 to 4, wherein, The encoder (301) and the decoder (303) are part of a stream-based model.

8. The method according to any one of claims 1 to 4, wherein, Thus, an additional data representation (402) is generated by a weighted sum of the selection of two or more initial data representations (401), such that the additional data representation (402) exceeds a predefined spacing threshold for all initial data representations and / or other additional data representations (402).

9. The method according to any one of claims 1 to 4, wherein, The additional data representation (402) is generated iteratively within the convex hull of the first data representation (401), wherein, in each iteration step, the additional data representation generated exceeds a predefined spacing threshold for all first data representations and the additional data representations (402) generated so far.

10. A computer program product comprising instructions that, when implemented by a computer, cause the computer to perform the method according to any one of claims 1 to 9.

11. A machine-readable storage medium having a computer program product according to claim 10 stored thereon.

12. An apparatus comprising a processor and a storage medium having instructions stored thereon, wherein, when the instructions are executed by the processor, the apparatus is capable of performing the method according to any one of claims 1 to 9.

Images