Method for applying a machine learning algorithm
By leveraging a priori knowledge to selectively quantify relevant intermediate data in neural networks, the method improves precision in object recognition by reducing quantization errors without increasing computing power.
Patent Information
- Application Number
- DE102023204275
- Authority / Receiving Office
- DE · DE
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-09
- Publication Date
- 2024-11-14
AI Technical Summary
Existing quantization algorithms for machine learning algorithms, particularly neural networks, fail to account for a priori knowledge of data relevance, leading to unnecessarily high quantization errors and reduced precision in applications like object recognition, without increasing computing power.
Utilize a priori knowledge to selectively quantify only relevant intermediate data values based on their significance, ignoring irrelevant data, such as background classifications, to reduce quantization errors and improve precision without increasing computing power.
Reduces quantization errors and enhances precision in object recognition tasks by focusing quantization on relevant data, maintaining or reducing computing power requirements.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[0001] The present invention relates to a method for applying a machine learning algorithm, e.g. an artificial neural network, for a given application, in particular object recognition, as well as a computing unit and a computer program for carrying out the method. Background of the invention
[0002] Machine learning algorithms such as artificial neural networks can be used for various applications, such as object recognition. This allows one or more objects, such as animals, people, or specific items, to be recognized in an image. Disclosure of the invention
[0003] According to the invention, a method for applying a machine learning algorithm, as well as a computing unit and a computer program for implementing the method, are proposed, having the features of the independent patent claims. Advantageous embodiments are the subject of the dependent claims and the following description.
[0004] The invention relates to machine learning algorithms, particularly (artificial) neural networks. Applying a machine learning algorithm typically involves providing input data, e.g., an image or image data, for the machine learning algorithm, processing the input data to obtain or determine output data, and providing the output data, e.g., information about objects recognized in the image. Before application, machine learning algorithms must be trained, for which purpose training data, e.g., images with known objects, are typically provided. Based on this, the weights of the machine learning algorithm or a corresponding machine learning model are then adjusted. Both the application and the training can each take place on a single computing unit.
[0005] Both application and training involve processing different values such as probability values, position or size values. The values can be in a suitable numerical format. Training is often carried out, for example, using a 32-bit floating point representation (fp32). Since multiplications and additions in this numerical format are expensive in terms of hardware requirements and energy costs, machine learning algorithms or neural networks can be converted into more cost-effective data types or numerical formats such as 8-bit integer, 16-bit integer or brain floating point (bfloat) representations for execution (or application), i.e. the inference phase; this is also referred to as quantization. Quantization mechanisms can be used during training (also known as quantization-aware training orA distinction can be made between training (also called quantification-aware training) and training after training (also called post-training).
[0006] In the context of the present invention, we will primarily focus on post-training quantization, which has proven to be very effective for the most commonly used data type, 8-bit integer.
[0007] Various algorithms can be used for post-training quantization, as described in "SKZDZYMWMKK Amir Gholami, A Survey of Quantization Methods for Efficient Neural Network Inference, 2021." Differentiations can be made based on aspects such as uniform vs. non-uniform quantization, symmetric vs. asymmetric quantization, or static vs. dynamic quantization. All algorithms generally have in common that they attempt to minimize some form of quantization error.
[0008] Although the concept presented within the scope of the present invention is not limited to object detection, object detection is one application for a neural network that particularly benefits from this concept. Various architectures can be used for object detection, many of which are based on some form of object hypotheses or object suggestions, which are condensed into actual object detections in a second step, i.e., a post-processing step. These post-processing steps typically include non-maximum suppression to combine overlapping bounding boxes into a final bounding box for each detected object. An example of this is explained in more detail below.
[0009] It has now been shown that quantization errors can be reduced by appropriately using a priori knowledge of data when using a machine learning algorithm or a neural network.
[0010] When applying a machine learning algorithm or neural network, as mentioned above, output data is determined based on input data. This may require various processing steps, e.g., passing through various processing layers. For example, as already mentioned, it may also be planned that the input data is processed in a first step, which then, for example, yields intermediate data with (usually a large number of) values. In a second step, or post-processing step, the actual object recognition can then be performed based on this.
[0011] The intermediate data obtained during such processing, or the values of the intermediate data on the basis of which the output data is then determined (directly or indirectly), can therefore be used with different weightings for the given application case, for example. This knowledge that such different weightings exist is the aforementioned a priori knowledge. The different weightings can be of various types. One example of this is that a selected portion of the values of the intermediate data is not used for the application case. A remaining portion of the values of the intermediate data, however, is already used for the application case.
[0012] When the output data is determined based on the intermediate data, the values of the intermediate data are then quantized depending on the weighting within the framework of quantization, or are present in such a form. Preferably, the values of the intermediate data are or will be represented in a predefined numerical format, e.g., the aforementioned 8-bit integer, 16-bit integer, or brain floating-point representation. Quantization in this context particularly refers to reducing the storage space required to represent the values.
[0013] The values of the intermediate data can, in general, be, for example, (trained) parameters of the model (machine learning model or neural network), but also, for example, values or results that are obtained during processing (e.g. so-called “activation maps”).
[0014] If this a priori knowledge is not used, as has been the case so far, this leads to unnecessarily high quantization errors. Typical quantization algorithms attempt to minimize the quantization error of the entire output (i.e., the intermediate data) instead of focusing on the relevant parts of the output. Therefore, the quantization range, i.e., the range of values of data to be quantized, can be larger than necessary, leading to lower resolution and thus to a larger quantization error of the relevant parts of the output. With the proposed concept, however, as mentioned, quantization errors are reduced, which leads to more accurate results when applying the machine learning algorithm, for example, more accurate or better object recognition. The required computing power, on the other hand, is not increased; in fact, it can even be reduced, for example, if a smaller data format is used.
[0015] Machine learning algorithms or neural networks for object recognition are often based on a two-stage approach, which involves the neural network, in a first step, outputting a multitude of object hypotheses and, in a post-processing step, condensing these hypotheses into actual object recognitions. Using this example, i.e., object recognition based on an image as a use case, the invention will be explained in more detail below.
[0016] The intermediate data (output) of the machine learning algorithm or neural network can include classification data and regression data for multiple anchor points. This can be the case, for example, in object detection. The anchor points can be, for example, cells or grid cells into which an image (input data) is divided.
[0017] The classification data includes, for each of the multiple anchor points, a probability value for each of several classes, where the multiple classes include, in particular, one or more object classes and a background class. This allows each anchor point to be assigned a class with the highest probability value—i.e., a class. The classification data can also directly include a classification value for each anchor point, with the classification value then indicating the most probable class. The classification data thus indicates, for example, which class has the highest probability for a specific position in the image.
[0018] The regression data includes, for each of the multiple anchor points, one or more bounding box values that define one or more bounding boxes. The regression data thus specifies the corresponding bounding box, which typically consists of an offset in the x and y directions, as well as a width and height of the bounding box relative to an anchor point.
[0019] These anchor points (or simply anchors) can be arranged, for example, in a 2D grid. Consequently, the regression data includes or is a tensor of the form (number of anchor points, 4), i.e., four values per anchor coordinate (or anchor point) to encode the x-position and y-position as well as the size of the bounding box per anchor coordinate. Additional values may also be present per anchor coordinate. The classification data then includes or is, for example, a tensor of the form (number of anchor points, number of object classes +1), which encodes the class probabilities (probability value per class), including the background class. The number of anchor points indicates the total number of anchor points, which can also be subdivided into the x-direction or y-direction if necessary. In the case of object recognition, the object classes are those classes that the machine learning algorithm can or should recognize, e.g.the object classes dog, cat, human, car, bicycle, and the like. The background class is then understood to be a class in which everything is classified that is not or cannot be assigned to one of the object classes.
[0020] For example, if there are three object classes: dog, car, and bicycle, a probability value can be determined for each of these three object classes for an anchor point. A fourth probability value indicates how likely it is that the anchor point does not belong to any of these three object classes—i.e., to the background class.
[0021] The regression data then includes, in particular, the selected portion of the intermediate data or the values of the intermediate data. In other words, it is a portion of the bounding box values that are not used for the application, i.e., object detection.
[0022] Based on the classification data and the regression data, the initial data is then determined. The initial data then comprises one or more data sets, each containing information about a final class linked to a final bounding box. While a bounding box generally indicates a detected object, including the final bounding box, the final class determines which object was detected. Based on the probability values of the classification data, one of the classes can be selected as the one to which the object of a bounding box is to be assigned or assigned – this is the class that can also be specified using the aforementioned classification value. This could, for example, be the object class with the highest probability value.
[0023] In order to determine or calculate the final bounding boxes and classes per object, the bounding boxes encoded in the regression data (in this case, these are in particular bounding box suggestions) can be condensed, for example, using thresholding and non-maximum suppression (combining overlapping boxes).
[0024] The threshold can be used here in particular in such a way that anchor points which have a probability value for the background class that is higher than an absolute and / or relative threshold are assigned to the background class. It is conceivable, for example, that if the probability value for the background class is greater than each of the probability values for the other object classes, the anchor point is assigned to the background class (relative threshold). Likewise, for example, an anchor point can be assigned to the background class if the probability value for the background class is more than, for example, 40% (absolute threshold). In a case in which an anchor point is or will be assigned to the background class, this can also be referred to as non-detection or invalid detection, since none of the detectable objects was detected.
[0025] The bounding box values of anchor points assigned to the background class can now be included in the selected portion of the intermediate data, i.e., the values that are not considered for quantization. These values are still quantized, but they have no influence on the quantization method.
[0026] The values of the classification data of such a neural network typically lie within a predefined data or value range. The values of the regression data (i.e., the width and height of the bounding box and the bounding box offsets in the x and y directions) are typically expressed relative to the anchor, and the values of this data lie within a predefined value range, e.g., between -1 and +1, for a valid detection (i.e., when an object has been detected). This can be achieved, for example, through appropriate normalization.
[0027] However, for an invalid detection, such as an anchor point classified as background, there is no fixed data or value range in the quantization algorithms used so far. This is due to the fact that the regression data is not regularized during training if the corresponding classification is background. Therefore, the regression data contains arbitrary values and generally has no limited data or value range.
[0028] However, quantization algorithms typically consider all output values (values of the intermediate data) to determine the required range and accuracy of a particular layer of a neural network. This applies regardless of the actual quantization method, e.g., min-max quantization or error propagation. In particular, post-training quantization algorithms are not aware of the meaning of the output values. Consequently, large values of the regression data, which are ignored in later steps because they do not correspond to an actual object, can lead to significant quantization losses.
[0029] However, if a priori knowledge is used, it is possible to consider only relevant output values (i.e. values of the intermediate data) for quantization or to ignore irrelevant values.
[0030] It should be noted that this propagates back through the neural network and affects all layers, since the outputs of previous layers can also be ignored during quantization. The idea presented here is not limited to any quantization algorithm, but a very simple approach could be, for example, using layer-by-layer asymmetric min-max quantization. This involves determining the minimum and maximum values of each layer, and based on this, a scaling factor can be calculated that maps each value to the new target data range (value range) for the quantized values.
[0031] While the concept is primarily explained here for object detection in images, it is not limited to that. Any algorithm based on suggestions, i.e., whose outputs (values of intermediate data) may not be relevant, can benefit from this concept.
[0032] Instead of a binary decision about which outputs (values of intermediate data) of the neural network are considered for quantization, a weighted approach can also be used. For example, larger values can be used with lower significance.
[0033] Instead of a threshold for the background class, as already mentioned, the value of the background class can also be used in comparison to other probability values to determine whether a value from the regression data is considered for quantization. For example, a value from the regression data can be ignored if the probability value of the background class is greater than any other probability value.
[0034] The presented concept does not require labeled data for quantization, as the output (intermediate data) of the neural network (e.g., probability value for the background class) is used to determine whether an output is eligible for quantization. Nevertheless, labeled data can also be used. In this case, the ground truth can be used to determine which outputs (values of the intermediate data) are used for quantization, or the training loss function could be reused in quantization, as it also encodes the relevance of the individual network outputs.
[0035] A computing unit according to the invention, e.g. a computer or a server, e.g. in the cloud, is configured, in particular in terms of programming, to carry out a method according to the invention.
[0036] The implementation of a method according to the invention in the form of a computer program or computer program product with program code for carrying out all method steps is also advantageous, since this entails particularly low costs, in particular if an executing control unit is also used for other tasks and is therefore already present. Finally, a machine-readable storage medium is provided with a computer program stored thereon, as described above. Suitable storage media or data carriers for providing the computer program are, in particular, magnetic, optical and electrical memories, such as hard disks, flash memories, EEPROMs, DVDs, and others. Downloading a program via computer networks (Internet, intranet, etc.) is also possible. Such a download can be wired or cable-based or wireless (e.g. via a WLAN network, a 3G, 4G, 5G or 6G connection, etc.).
[0037] Further advantages and embodiments of the invention will become apparent from the description and the accompanying drawings.
[0038] The invention is illustrated schematically in the drawing using an embodiment and is described below with reference to the drawing. Short description of the drawings Fig. 1 shows schematically a sequence of a method according to the invention in a preferred embodiment. Fig. 2 shows schematic images for explaining a method according to the invention in a preferred embodiment. Fig. 3 schematically shows histograms for explaining a method according to the invention in a preferred embodiment. Embodiment(s) of the invention
[0039] In Fig. Figure 1 schematically illustrates a preferred embodiment of a method according to the invention. In step 100, input data 102 is provided for a machine learning algorithm or neural network. Based on the input data 102, intermediate data 110 is then determined in step 104. Values of the intermediate data are used with different weightings for the specified application.
[0040] In step 112, output data 114 is then determined based on the intermediate data 110. The values of the intermediate data are or will be quantized depending on the weighting, in particular, converted into a predetermined numerical format. In step 116, the output data are then provided. The process, which is only briefly presented here, will be explained below with reference to the Fig. 2 and Fig. 3 will be explained in more detail using a more concrete example.
[0041] In Fig. 2 schematically illustrates images for explaining a method according to the invention in a preferred embodiment, and in Fig. 3 schematically shows histograms to explain a method according to the invention in a preferred embodiment, which will be discussed in more detail below.
[0042] In Fig. Figure 2 shows an image 200 that can be passed as input data to a machine learning algorithm or a neural network to detect objects in the image 200. Examples of objects shown in the image are a dog 212, a bicycle 214, and a car 216. Depending on how the neural network is trained, only dogs, bicycles, and cars can be detected as objects.
[0043] The image 200 can now, for example, be divided into individual grid cells 202, so-called anchor points or anchors (2D grid); this can be done in the x-direction and the y-direction. For example, both the x-direction and the y-direction of the image 200 are each divided into seven grid cells, resulting in a total of 49 grid cells or anchor points.
[0044] As with regard to Fig. As mentioned in Figure 1, based on the input data, in this case the image 200 or the corresponding data thereto, intermediate data (or outputs of the neural network) can be obtained during processing by the neural network. This intermediate data includes classification data 220 and regression data 240. Both the classification data and the regression data are represented here as an image.
[0045] The classification data 220 includes, for each of the multiple anchor points 202, a classification value that specifies one of several classes. In this example, the multiple classes include the three object classes dogs, bicycles, and cars, as well as a background class; everything that cannot be assigned to any of the three object classes is assigned to the background class.
[0046] Here, Fig. 2 for the classification data 220, exactly one class is shown for each anchor point, i.e., one of the object classes or the background class—the one according to the classification value. In each case, the class that has the highest probability value among all classes for this anchor point is shown as an example. These different classes are object class 222 for dogs, object class 224 for bicycles, object class 226 for cars, and background class 228 for everything else. By comparing it with image 200, it can be seen that object classes 222, 224, and 226 roughly correspond to the underlying objects in image 200.
[0047] The output of the neural network typically consists of one or more classification outputs (classification data) and one or more corresponding regression outputs (regression data). The classification outputs indicate which object class has the highest probability for a given position in the image. The regression outputs specify the corresponding bounding box, which typically consists of an offset in the x and y directions, as well as a width and height of the box relative to a so-called anchor.
[0048] The regression data 240 includes, for each of the multiple anchor points, one or more bounding box values that define one or more bounding boxes. For example, a bounding box that specifies the car is designated 242. As mentioned, the bounding box values can include, for example, a value for a width and a value for a height of the bounding box, as well as a value for an x-offset and a value for a y-offset. The information can be relative to the anchor point, e.g., the center point of the grid cell.
[0049] Based on the classification data 220 and the regression data 240, i.e., based on the intermediate data, the output data can then be determined. The output data can comprise one or more data sets, each containing information about a final class linked to a final bounding box.
[0050] In Fig. In Figure 2, the initial data is labeled 260 and represented as an image. Three data sets 262, 264, and 266 are shown as examples, each represented as a rectangle, which in turn can be viewed as the corresponding final bounding box. Data set 262 is intended to link information about the final class "Dogs" with the corresponding final bounding box. Data set 264 is intended to link information about the final class "Bicycles" with the corresponding final bounding box. Data set 266 is intended to link information about the final class "Cars" to the corresponding final bounding box.
[0051] The background class is not included in the source data because, as already mentioned, it is about the detection of specific objects, while the background class only serves as a kind of catchall class for everything that cannot be assigned to any of the specific object classes.
[0052] The values of the classification data of such a neural network typically lie within a predefined data or value range. The values of the regression data 240 are typically expressed relative to the anchor, and the values of this data lie within a predefined value range, e.g., between -1 and +1, for a valid detection (i.e., when an object has been detected, or when the assigned class is an object class). This can be achieved, for example, by suitable normalization.
[0053] However, for an invalid detection (i.e., when no object was detected, or when the assigned class is the background class), there is no fixed data or value range in the quantization algorithms used so far. Therefore, the regression data contains arbitrary values and generally has no limited data or value range.
[0054] A histogram of all values (output values) of a regression layer is shown in Fig. 3, where a number N is plotted against a value w. The histogram on the right corresponds to the histogram on the left, but with a greatly restricted range in the vertical axis.
[0055] In the case of a background, i.e., if no object is detected at a certain position, the regression data is ignored as mentioned (e.g., excluded by score thresholding), but can be significantly outside the range of [-1,1]. A high classification score for the background does not necessarily lead to a high regression output, but it can be the case. Fig. 3 with 300 denotes values that are assigned to the background or background class, while 302 denotes values that are assigned to an object or object class.
[0056] This plot shows that all regression results corresponding to an actual object class are in the range [-1,1]. However, several regression results corresponding to the background are less than -1 or greater than 1.
[0057] However, quantization algorithms typically consider all output values (intermediate data values) to determine the required range and accuracy of a particular neural network layer. Consequently, large values of the regression data, which are ignored in later steps because they do not correspond to an actual object, can lead to significant quantization losses.
[0058] However, if a priori knowledge is used, in the case shown that regression data is ignored (i.e. not used for the application case) if no object is assigned to a position, it is possible to consider only relevant values for quantization or to ignore irrelevant values.
[0059] Consequently, only the values within the specified range, e.g., between -1 and +1, need to be represented using an 8-bit integer representation, but not values from, e.g., -2 to +2. This shows that quantization loss can sometimes be significantly reduced here. QUOTES CONTAINED IN THE DESCRIPTION
[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature
[0000] SKZDZYMWMKK Amir Gholami, A Survey of Quantization Methods for Efficient Neural Network Inference, 2021
[0007]
Claims
[1] Method for applying a machine learning algorithm for a given application, in particular object recognition, comprising: Providing (100) input data (102, 200) for the machine learning algorithm (108); Determining (106), based on the input data (102), intermediate data (110), wherein values of the intermediate data are used with different weightings for the given application case; Determining (112), based on the intermediate data (110), output data (114), wherein the values of the intermediate data are or will be quantized as a function of the weighting within the framework of a quantization, in particular brought into a predetermined numerical format; and Providing (116) the output data. [2] Method according to claim 1, wherein a selected part of the values of the intermediate data is not used for the application, and wherein only values of the intermediate data which do not include the selected part are or will be taken into account in the quantization [3] Method according to claim 1 or 2, wherein the intermediate data comprises classification data (220) and regression data (240) for a plurality of anchor points (202), wherein the classification data (220), for each of the plurality of anchor points, each comprise a probability value for each of a plurality of classes (222, 224, 226, 228), or a classification value indicating the one of the plurality of classes with the highest probability value, wherein the plurality of classes in particular comprise one or more object classes (222, 224, 226) and a background class (228), and wherein the regression data (240), for each of the plurality of anchor points, each comprise one or more bounding box values defining one or more bounding boxes. [4] The method of claim 2 and 3, wherein the regression data comprises the selected portion of the intermediate data. [5] Method according to claim 3 or 4, wherein the output data comprises one or more data sets (262, 264, 266) each containing information about a final class linked to a final bounding box, wherein, for the one or each of the plurality of data sets, the information on the final class associated with the final bounding box is determined based on the bounding boxes of the regression data, taking into account classes assigned according to the anchor points, wherein the selected part of the intermediate data comprises bounding box values of anchor points assigned to a predetermined one of the plurality of classes, in particular the background class, in particular having a probability value for the background class that is higher than an absolute and / or relative threshold value. [6] Method according to one of the preceding claims, with reference to claims 3 and 4, wherein for anchor points which are not assigned to a predetermined class, in particular the background class, one or more of the bounding box values each lie within a predetermined range of values. [7] Method according to one of the preceding claims, wherein the values of the intermediate data are or are brought into a predetermined numerical format depending on the weighting, wherein the predetermined numerical format is an 8-bit integer, a 16-bit integer or a brain floating point representation. [8] Computing unit which is designed to carry out all method steps of a method according to one of the preceding claims. [9] Computer program which causes a computing unit to carry out all the method steps of a method according to one of claims 1 to 7 when executed on the computing unit. [10] A machine-readable storage medium having stored thereon a computer program according to claim 9.
