Classification of image data with granular degree adaptation

Abstract

A device (1) for classifying image data (2) includes: • a trainable preprocessing unit (11, 11a-11d) configured to retrieve (111) at least one predefined (3) from trained associations regarding the degree to which the granularity of the image data (2) should be reduced, and to reduce (112) the granularity of the image data (2) according to the predefined (3); and • a trainable classifier (12) configured to map the detail-reduced image data (4) onto an assignment relationship (5) to one or more predefined categories. The method (100) for training the device (1) wherein the parameter (11*) characterizing the properties of the trainable preprocessing units (11, 11a-11d) is optimized (130) to the objective that the device (1) maps the learning image data (2a) to the learning assignment relation (5a), and at the same time the granularity of the learning image data (2a) performed by the preprocessing units (11, 11a-11d) is reduced by an average corresponding to a predefined (3a).

Description

Technical Field

[0001] The present invention relates to a device for classifying image data, which is particularly useful for at least partially automating driving vehicles in traffic. Background Technology

[0002] Continuous monitoring of the vehicle's surroundings is absolutely essential for safe driving in road traffic. Corresponding sensing technologies provide images in various modalities. To obtain machine-processable information about traffic conditions, trainable classifiers are employed. Such classifiers process image data into one or more predefined categories. For example, these categories could represent traffic signs or other objects to be identified in the image.

[0003] The recorded images often cannot be used directly for classification and must first be preprocessed. From DE 10 2017211 331 A1, a system with a preprocessing module is known, which can be trained in a targeted manner to highlight information relevant to the actual image being processed. Summary of the Invention

[0004] Within the scope of this invention, an apparatus for classifying image data has been developed. The apparatus includes a trainable preprocessing unit. The preprocessing unit is configured to, based on the image data, retrieve at least one predefined value from a trained association regarding the desired level of granularity reduction of the image data, and correspondingly reduce the granularity of the image data. Furthermore, the apparatus includes a classifier configured to map the detail-reduced image data onto one or more predefined class assignments.

[0005] In this context, the term "classification" also includes semantic segmentation of an image, pixels of an image, or other sub-regions.

[0006] Image data can be, for example, camera images, video images, radar images, ultrasonic images, or lidar (LIDAR) images. These are imaging modalities that are most frequently used, individually or in combination, to monitor the vehicle's surroundings.

[0007] The assignment relationship to one or more categories can be, for example, a "one-hot" vector, which has a non-zero component for only one of the available categories. However, the assignment relationship can also be, for example, a Softmax vector, which assigns non-zero confidence and / or probability to multiple categories, the total confidence and / or probability of which sum to 1.

[0008] Trainable preprocessing units or trainable classifiers are specifically considered as modules that, for generalization, strongly embody functions parameterized using adaptable parameters. When training such modules, these parameters are adapted such that the corresponding learning inputs of the respective modules or the device as a whole are mapped as well as possible to their pre-known learning outputs. Preprocessing units or classifiers may, in particular, comprise artificial neural networks (KNNs), and / or may be KNNs.

[0009] It has been recognized that the maximum detail richness of image data itself does not guarantee that image data will be correctly assigned to categories by a classifier. More precisely, given a training level, particularly characterized by the type, composition, and quality of the training data and the number of training epochs, a trainable classifier can only reasonably handle images with a defined degree of granularity. If, for example, a camera sensor is replaced with a new camera sensor with a higher pixel resolution, images with higher granularity are fed in, no further improvement in classification accuracy is achieved. More precisely, if the increase in resolution is not accompanied by an improvement in the quantity and / or quality of the training, accuracy may even be compromised: higher pixel resolution may prove to be an “intrusion gate” for image noise, but it may also prove to be an “intrusion gate” for targeted attacks with “adversarial examples.” The latter are manipulations specifically introduced into the image data that may go completely unnoticed by the human observer of the image, but may prompt the classifier to assign the image to one or more other categories.

[0010] Therefore, a predetermined training level may be sufficient to reliably classify images with the typical granularity of Full HD (1920 × 1080 pixels). However, to reliably classify images with the typical granularity of 4K (4096 × 2160 pixels), longer training, training with greater variability in the training data, and / or training with lower label noise in the training data may be required. In specific applications where the training level of the classifier is predetermined, post-hoc reduction of the granularity can prevent the susceptibility of image noise or "adversarial examples" due to over-granularity from taking effect.

[0011] Correspondingly, starting from the given sensory mechanisms to be used for image recording and the application of the classification task, the cost of obtaining training data can be saved. If, for example, a camera with too much granularity for a specific task is chosen, the subsequent reduction of the granularity may result in training with less training data than might be needed in other cases for the selected resolution of the camera, beyond the pre-specified accuracy to be achieved on the classification task.

[0012] As a vivid example, pedestrian identification is worth mentioning. Since pedestrians are the most vulnerable participants in traffic, identifying each and every one of them is absolutely crucial. Conversely, what matters less is how the pedestrian is dressed, whether their clothing includes printed themes, shoulder pads, or other accessories, or what their physique is. Therefore, the granularity of image data can be significantly reduced without diminishing the accuracy in identifying pedestrians in traffic conditions.

[0013] The lower granularity also allows for a more economical determination of the classifier hardware size. If less detail needs to be processed in total, the classifier of KNN can be implemented, for example, on a smaller computing unit (such as a GPU), which is less expensive and consumes less energy.

[0014] In particular, reducing the degree of granularity may, for example, cause simple variations (and in this invention, especially variations within individual image pixels) to be flattened. Subsequently, the classifier no longer reacts to such variations.

[0015] To reduce granularity, any transfer function can be used in principle, characterized by one or more optimizable parameters. For example, specific image regions that are not important for classification can be masked. In particular, this example shows that the optimal granularity reduction for the corresponding application is specific to each image. Therefore, it is important to retrieve, for each image, a predefined degree to which the granularity should be reduced from the trained associations.

[0016] In a particularly advantageous construction scheme, the preprocessing unit and the classifier are constructed as a common artificial neural network (KNN). This makes it easy to "hand in hand" train the two modules together, as the required level of granularity may, for example, depend on the type of object to be identified. As mentioned earlier, the specific shape of a pedestrian is often irrelevant to pedestrian identification. What is important, for example, is to distinguish based on the shape of the vehicle, whether it is a passenger car or a commercial vehicle, because these vehicle types differ fundamentally in their driving dynamics.

[0017] Constructing a common KNN does not require that the preprocessing unit and the classifier be implemented in a common component or otherwise spatially adjacent in hardware. Rather, what is crucial is that the parts of the common KNN that form the preprocessing unit and the parts that form the classifier are data-technically connected to each other, so that the parameters of these two parts of the KNN can be trained together.

[0018] Granularity reduction can be particularly useful in monitoring the vehicle's surroundings to save transmission bandwidth within the vehicle. Granularity reduction can be achieved through preprocessing units near the relevant sensors. Subsequently, the reduced-detail image data has a significantly smaller file size, and / or this image data can be compressed to a significantly smaller file size due to its lower information content for transmission to the classifier within the vehicle.

[0019] Therefore, in another particularly advantageous configuration, the preprocessing unit is connected to at least one image source carried by the vehicle via a dedicated broadband connection. The preprocessing unit is connected to the classifier via the vehicle's bus system, which is shared by other onboard systems of the vehicle.

[0020] Most vehicles today are equipped with a CAN bus or other bus system, to which many other vehicle systems connect. This bus system allows all connected participants to communicate with each other. This represents a significant reduction in wiring overhead compared to the earlier tree-like structure of dedicated cables between each pair of communicating participants. However, the trade-off is that connected participants must share the bus system's bandwidth. Currently, typically only one user can transmit at a time. If multiple sensors (such as high-resolution cameras) are now used to monitor the entire vehicle's surroundings, a large amount of data is generated, which may not even need to be transmitted entirely via the bus system. Even the "high-speed" CAN bus only has a maximum bandwidth of 1 Mbit / s, which is too small for full HD video data streams. However, this bandwidth is now sufficient for transmitting data from multiple cameras, compressed in the same way, by significantly reducing granularity and thus lossily compressing the data before transmission via the bus system. For this purpose, a preprocessing unit, in particular, could be part of the sensor module used to record image data from the vehicle's surroundings.

[0021] In a particularly advantageous configuration, the preprocessing unit is configured to transform the image data into a representation in the workspace and reduce the granularity of that representation. If, in the workspace used, details important to the corresponding application can be distinguished particularly well from less important details, then in this way, more important details can be preserved and more unimportant details can be omitted. This representation can then be transformed back, for example, into the space to which the original image data also belongs. This has the advantage that the classifier itself does not need to be modified, but can continue to operate as if the original image data might have been fed to the classifier. However, the representation can also be processed directly by the classifier, for example. This is particularly advantageous when transforming back into the space of the original image data is more difficult or it is not even clear whether it is possible.

[0022] In a particularly advantageous configuration, the preprocessing unit is configured to determine the representation in the workspace as a linear combination of the basis functions of that workspace represented by a set of coefficients. The preprocessing unit is further configured to, based on the image data, retrieve at least one predefined rule from trained associations regarding which coefficients to be reduced or eliminated according to numerical values, and reduce or eliminate the coefficients of the representation according to that predefined rule.

[0023] In a particularly advantageous configuration, the preprocessing unit is structured to determine the representation in the workspace as a linear combination of wavelet coefficients characterized by a set of wavelet coefficients. A wavelet is a function that is non-zero only in a finite region of its (spatial) domain. In particular, the wavelet (“mother wavelet”) can generate an entire system of functions, for example, according to different size scales, which can then be used for multiscale analysis of the image data. Furthermore, wavelets are characterized in that the useful content of the image is typically concentrated in only a few wavelet coefficients with large numerical values, while noise, for example, is removed via many wavelet coefficients with small numerical values.

[0024] To reduce granularity, wavelet coefficients that involve relatively fine details may be preferred, especially those involving large spatial image regions. Such preferred image proportions are particularly robust to noise in the original image and also particularly robust to manipulation of the original image using "adversarial examples." That is, noise (or manipulation using "adversarial examples") is flattened by the preprocessing unit and no longer reaches the classifier. For example, discrete cosine transform or Fourier transform can be used instead of wavelet transform.

[0025] In a particularly advantageous construction scheme, the preprocessing unit is configured to retrieve coefficients according to a predetermined proportion, which are to be reduced or eliminated based on their numerical values. Thus, the learned associations can, for example, suggest considering a specific image in all its details, but removing 50% or more of those details from other images.

[0026] For example, the preprocessing unit can learn to consider a very accurate and sharp image in all its details, while significantly reducing blurry or otherwise unclear images in terms of their granularity. This prevents, for example, noise in such unclear images from implying the presence of important details that are not actually there.

[0027] In particular, for example, coefficients can be determined according to the proportion of the number of images, taking into account the entropy of the image data: coefficients that should be reduced or eliminated according to their numerical values.

[0028] For example, a fixed allocation (Quote) can be specified for coefficients that should not be changed, and then, for example, this fixed allocation can be specified according to the wavelet size scale. Thus, for example, the number of coefficients that should not be changed can be specified on the hyperparameter a for the largest size scale (order 0), and then on a for smaller size scales (orders i = 1, 2, 3, ...). iThe integer part is rounded up to the next larger integer.

[0029] Alternatively, the preprocessing unit can be configured to retrieve the dimensions of the latent space serving as the workspace for the autoencoder from trained associations based on the image data, and transform the image data into this workspace using the autoencoder. In this respect, the autoencoder is a trainable module that transforms the image data into a representation with significantly reduced dimensions. This representation can then be translated back by the associated decoder into image data that is very similar to the original image data. Typically, the encoder and decoder are trained together with the objective that the translated image data is as similar as possible to the original image data.

[0030] In this construction scheme, the dimension of the latent space is adjusted by a screw (Stellschraube) based on the granularity of the image data. The representations in the latent space, which serves as the working space, can be directly processed by the classifier. However, the translated image data can also be processed by the classifier.

[0031] For example, the device-provided allocation relationships of image data to one or more categories can be evaluated in any way within a vehicle by a driver assistance system and / or a system for at least partially automated driving, and can form the basis for planning driving exercises. In particular, the device can be used as a data source for any such system.

[0032] The present invention also relates to a method for specifically training an apparatus for a construction scheme in which a preprocessing unit transforms image data into a representation in a working space (e.g., wavelet coefficient space or latent space).

[0033] Within the scope of this method, training image data and their associated learning assignments are provided, and the device is to nominally map the training image data to the associated learning assignments. Furthermore, a predefined hyperparameter is defined to aim for a reduction in granularity during training. Parameters characterizing the features of the device's trainable preprocessing units are now optimized to the following objectives: • The device maps the learned image data to the learned assignment relationship, and • At the same time, the granularity of the learning image data reduced by the preprocessing unit corresponds on average to the predefined level.

[0034] As mentioned earlier, hyperparameters specify the proportion of detail from the raw records that is averaged and fed to the classifier. During training of the preprocessing unit's parameters, the predefined "ration" information is then learned to be distributed across different training images, allowing the classifier to ultimately provide optimal accuracy. For example, it might be predefined that the granularity should be reduced by an average of 50%. Then, at the end of training on, for example, one million images, a few images can be selected as key images, considered in all detail, while 95% of the details in these key images are not considered in the majority of other images.

[0035] It is expected that, in practical applications, the hyperparameters will exhibit the following optimal values: these optimal values ​​may be related to, for example, the architecture and size of the KNN, the type of training data, label quality and variability, the learning rate, and the number of pixels in the processed image. These optimal values ​​can be obtained through arbitrary parameter optimization methods.

[0036] In a particularly advantageous configuration, additional parameters characterizing the classifier of the device are optimized to the objective that the device maps learned image data to learned assignment relationships. As mentioned earlier, the preprocessing unit and the classifier can then be trained "hand in hand," for example, simultaneously, alternately, or in any other way, alternating between the preprocessing unit and the classifier. For example, the classifier can learn, within the scope of its training progress, which details in the image are particularly relevant to the decision between categories; while the preprocessing unit can learn to highlight these details in the image data.

[0037] In another particularly advantageous construction scheme, the optimization of the parameters characterizing the classifier's properties is also aimed at minimizing the granularity of the image data used by the classifier. This enhances the classifier's tendency to make decisions about class assignments more relevant to the slowly varying proportions of the image. This further improves robustness against noise and adversarial examples.

[0038] Not only the devices but also the methods described herein can be implemented entirely or partially using a computer. Therefore, the invention also relates to a computer program having machine-readable instructions that, when executed on one or more computers, cause the computers to upgrade to the aforementioned devices and / or cause the computers to implement the aforementioned methods. In this sense, control devices for vehicles and embedded systems for technical equipment should also be considered as computers, and these control devices for vehicles and embedded systems for technical equipment are also capable of executing machine-readable instructions.

[0039] Similarly, the present invention also relates to a machine-readable data carrier having a computer program, and / or to a downloadable product having a computer program. A downloadable product is a digital product that can be transmitted via a data network, that is, a digital product that can be downloaded by a user of a data network, such as a digital product that can be sold in an online store for immediate download.

[0040] In addition, a computer may be equipped with computer programs, machine-readable data carriers, or downloadable products. Attached Figure Description

[0041] Below, together with the description of preferred embodiments of the invention, other measures to improve the invention are shown in more detail with reference to the accompanying drawings.

[0042] Figure 1 An embodiment of device 1 is shown; Figure 2 Other embodiments of device 1 in vehicle 6 are shown; Figure 3 shows a schematic example of granularity reduction for pedestrian 4; Figure 4 An embodiment of method 100 is shown. Detailed Implementation

[0043] Figure 1 An embodiment of a device 1 for classifying image data 2 is shown. Device 1 includes a trainable preprocessing unit 11 and a classifier 12.

[0044] Preprocessing unit 11 receives image data 2 from any source and, in block 111, determines at least one pre-defined requirement 3 regarding the degree to which the granularity of the image data 2 should be reduced. In block 112, the granularity of the image data 2 is reduced according to the pre-defined requirement 3, resulting in reduced-detail image data 4. This reduced-detail image data 4 is mapped to one or more pre-defined category assignment relationships 5.

[0045] Figure 2Another embodiment of device 1 built into vehicle 6 is shown. Vehicle 6 has four cameras 61a-61d that monitor areas 60a-60d of the surrounding environment of vehicle 6. Each of these cameras 61a-61d provides image data 2 to its respective assigned preprocessing units 11a-11d via dedicated broadband connections 62a-62d. All preprocessing units 11a-11d are connected to the central bus system 63 of vehicle 6, which is also shared by other onboard systems 64-66 of vehicle 6. Preprocessing units 11a-11d provide strongly compressed, reduced-detail image data 4 to a trainable classifier 12. In this way, classifier 12 can process the most important details of image data 2 without having to transmit the entire data stream of image data 2 via the bus system 63 of vehicle 6, which was not designed for this purpose. For clarity, Figure 2 The classifier is not drawn into the reaction of the actuators of vehicle 6.

[0046] Figure 3 shows an example of reduced detail for pedestrian 7. Figure 3a This is a schematic diagram of image data 2 provided by the camera. Pedestrian 7 has a face 71 with clear and distinct facial features. Pedestrian 7 is also wearing a T-shirt 72 with lettering 73 and shoulder pads 74.

[0047] All these details are irrelevant to the crucial identification of pedestrians. Some details might even distract classifier 12. Thus, for example, a specific facial shape in face 71 or the tattered state of a T-shirt 72 might prompt classifier 12 to misclassify pedestrian 7 as a scarecrow. Similarly, shoulder pads 74 might prompt classifier 12 to misclassify pedestrian 7 as a mannequin. Both situations would be detrimental to pedestrian 7 because a system designed for at least partially automated driving might assume that a collision with a scarecrow or mannequin would result in only minor material damage, and the system might prioritize that collision over a collision with another vehicle if in doubt. A similar situation could occur when misclassification is intentionally caused by manipulative "counter-attack" patterns 75 in image data 2 (e.g., caused by a translucent sticker on a camera lens).

[0048] Figure 3b Image data 4, after reduction in detail, is shown. In this invention, only the pedestrian 7 is abstractly identified. All other details are omitted, so that no additional bandwidth and computing power are required for its processing. The "counter" pattern 75 is also smoothed into uniform spots, and thus made harmless.

[0049] Figure 4An embodiment of a method 100 for training device 1 is shown. In step 110, learning image data 2a and the associated learning assignment relationship 5a are provided. In step 120, to reduce the granularity of the learning image data 2a, a predefined average averaging 3a is specified. In step 130, parameters 11* characterizing the features of the trainable preprocessing units 11, 11a-11d of device 1 are optimized to the following objective: • Device 1 maps the learned image data 2a to the learned assignment relation 5a, and • Meanwhile, the reduction in granulation of the learning image data 2a by the preprocessing units 11, 11a-11d corresponds to the predefined 3a.

[0050] Additionally, in this embodiment, in step 140, the parameters 12* characterizing the properties of the classifier 12 of device 1 are also optimized to the objective that device 1 maps the learned image data 2a to the learned assignment relation 5a. This training is interleaved with the training 130 of the parameters 11* of the preprocessing units 11, 11a-11d.

[0051] According to block 141, the optimization 140 of the parameters 12* of classifier 12 is also additionally aligned with the objective of minimizing the granularity of the image data 2 used by classifier 12.

Claims

1. A device (1) for classifying image data (2), comprising: • A trainable preprocessing unit (11, 11a-11d), the preprocessing unit (11, 11a-11d) being configured to: (i) transform the image data (2) into a representation in a working space, the representation being a linear combination of the basis functions of the working space characterized by a set of coefficients; (ii) retrieve (111) from trained associations, based on the image data (2), at least one pre-defined rule (3) regarding which coefficients to reduce or eliminate numerically, the at least one pre-defined rule (3) defining the degree to which the granularity of the image data (2) should be reduced according to the type of object to be detected; and (iii) generate reduced-detail image data (4) by reducing (112) or eliminating the coefficients of the representation according to the at least one pre-defined rule (3) to mask image regions that are not important for the classification; and • A trainable classifier (12) configured to map the reduced-detail image data (4) onto an assignment relationship (5) to one or more predefined categories.

2. The device (1) according to claim 1, wherein, The preprocessing units (11, 11a-11d) and the classifier (12) are constructed as a common artificial neural network KNN.

3. The device (1) according to claim 1, wherein, • The preprocessing unit (11, 11a-11d) is connected to at least one image source (61a-61d) carried by the vehicle (6) via a dedicated broadband connection (62a-62d), and • The preprocessing unit (11, 11a-11d) is connected to the classifier (12) via the bus system (63) of the vehicle (6), which is shared by other onboard systems (64-66) of the vehicle (6).

4. The device (1) according to claim 1, wherein, The preprocessing unit (11, 11a-11d) is configured to determine the representation in the workspace as a wavelet linear combination characterized by a set of wavelet coefficients.

5. The device (1) according to any one of claims 1 to 4, wherein, The preprocessing unit (11, 11a-11d) is configured to retrieve coefficients in proportion to the following as predefined (3): coefficients to be reduced or eliminated numerically.

6. The device (1) according to claim 5, wherein The preprocessing unit (11, 11a-11d) is configured to determine, taking into account the entropy of the image data (2), the coefficients to be reduced or eliminated in numerical proportion.

7. The apparatus of claim 1, wherein, The preprocessing units (11, 11a-11d) are configured such that, • Based on the image data (2), retrieve the dimensions of the latent space serving as the workspace for the autoencoder from the trained associations, and • The image data (2) is transformed into the workspace using the autoencoder.

8. A method (100) for classifying image data (2), comprising the steps of: - The image data (2) is represented in the workspace, which is a linear combination of the basis functions of the workspace characterized by a set of coefficients. -Based on the image data (2), retrieve (111) from the trained associations at least one pre-defined rule (3) regarding which coefficients to reduce or eliminate according to numerical values, the at least one pre-defined rule (3) defining the degree to which the granularity of the image data (2) should be reduced according to the type of object to be detected. - Image data with reduced detail (4) is generated by reducing (112) or eliminating the coefficients of the representation according to at least one predefined (3) to mask image regions that are not important for the classification, and - Map the reduced detail image data (4) to an assignment relationship (5) to one or more predefined categories.

9. A computer program product comprising machine-readable instructions that, when executed on one or more computers, cause the one or more computers to perform the method (100) according to claim 8.

10. A machine-readable data carrier and / or downloadable product having a computer program product according to claim 9.

11. A computer having a computer program product according to claim 9, and / or having a machine-readable data carrier and / or download product according to claim 10.

Images