Machine learning system and method and device for creating a machine learning system

Abstract

Machine learning system and method and device for creating a machine learning system. The present invention relates to a machine learning system (10) comprising at least one temporal filter (21). Input variables comprising a temporal sequence of images are processed by means of the machine learning system (10) using the filter (21). The machine learning system (10) is configured to apply the filter (21) to a sequence of pixels that are all at the same coordinates of the image or at the same coordinates of an intermediate result. The filter coefficients of the filter (21) are quantized. The present invention also relates to a method, a computer program, and a device for creating a machine learning system (10).

Description

Technical Field

[0001] The present invention relates to a machine learning system with quantized parameters. Furthermore, the present invention relates to a method and a computer program and an apparatus for creating a machine learning system. Background Art

[0002] The unpublished document DE 10 2017 218 889.8 discloses an artificial intelligence module configured to process one or more input variables into one or more output variables via an internal processing chain. The internal processing chain is defined by one or more parameters. An allocation module is provided, which is configured to determine the parameters based on at least one statistical distribution.

[0003] The non-prepublished document DE 10 2018 216 471.1 discloses a quantized neural network.

[0004] It is possible to create quantized neural networks with extremely strong quantization, as shown, for example, by Hubara et al. in “Quantized neural networks: Training neural networks with low precision weights and activations” (The Journal of Machine Learning Research, 2017, 18, No. 1, pp. 6869-6898).

[0005] Recurrent neural networks (RNNs) have the advantage that they can also process information contained in the temporal dimension of sensor data. However, a disadvantage of RNNs is that they cannot be strongly quantized.

[0006] However, it is desirable to quantize the parameters of RNNs so that they can be run on mobile computing units with more adequate performance. Summary of the Invention

[0007] According to independent claim 1, neural networks with temporal convolutions are used instead of RNNs. This has the advantage that, despite the computationally expensive convolutions, these neural networks can surprisingly be more energy-efficient than comparable RNNs in terms of the quantization of their parameters, while retaining the aforementioned advantages of RNNs.

[0008] In a first aspect, a particularly memory- and energy-efficient machine learning system for processing time series / sequences is proposed.

[0009] A machine learning system, in particular a neural network, having the features of claim 1 includes at least one filter, wherein at least the filter coefficients of the filter are quantized. It is also conceivable that other parameters of the machine learning system, in addition to the filter coefficients, are also quantized. Alternatively or additionally, at least the activations can be quantized. Quantizing the activations can be useful if the number of activations exceeds the number of parameters (e.g., weights) of the machine learning system.

[0010] The machine learning system is configured to determine an output variable using a filter based on an input variable. The input variable may comprise a time series of images. Alternatively, a plurality of temporally sequentially detected variables, such as one-dimensional sensor data, may be used. Furthermore, the machine learning system is configured to apply the filter to a sequence of pixels that are all located at the same coordinates in an image or at the same coordinates in an intermediate result of the machine learning system, particularly an intermediate processed image. When a detected variable is used instead of an image, the filter is applied to data points of the detected variable or intermediate result that are located at the same position. The intermediate result may be, for example, the activation of a layer of a neural network.

[0011] It is also conceivable that the filter coefficients and preferably other parameters of the machine learning system each have values from a predefinable quantized value range. The predefinable quantized value range can be defined by a predefinable number of quantization levels and / or by a predefinable number of bits.

[0012] Quantization can be understood as meaning that the filter coefficients can only adopt quantized values expressed in decimal, preferably in floating point, fixed point or integer digital format. Preferably, the filter coefficients are quantized with a maximum of 32 bits, in particular with a maximum of 16 bits or 8 bits and preferably with 4 bits or 1 bit. Quantization can also be understood as meaning that the filter coefficients are rounded up or down to the nearest quantization level of a plurality of predefined quantization levels. The filter coefficients then each have values of a plurality of quantization levels, wherein these values are stored in a memory. Advantageously, the quantization levels can be stored in a memory-efficient manner. The number of quantization levels is related to the number of bits used for quantization. Alternatively, the trained machine learning system can be quantized, wherein the parameters of the machine learning system are adapted at full resolution during training and then quantized.

[0013] The predefinable quantized value range can also be a list. It is conceivable that the filter coefficients have a reduced resolution, wherein the resolution is reduced by means of quantization.

[0014] The resolution can characterize how many different possible, in particular quantized, values are used to represent each parameter and / or intermediate result.

[0015] Applying a filter can be understood as mathematically convolving the filter with all data points or segments having the same coordinates.

[0016] The machine learning system according to the first aspect has the following advantages: quantization requires fewer bits to store filter coefficients, and therefore fewer memory locations. This allows for the use of smaller memories, which allows for a more compact hardware design of the machine learning system. Since the energy consumption for storing and reading parameters and activations is proportional to the number of bits, and the energy consumption for multiplication is at least quadratically proportional to the resolution (number of bits) of the parameters and activations, reducing the number of bits (particularly through quantization) is particularly efficient in terms of computer resources. These advantages also apply when the machine learning system is implemented in software, as less memory is required.

[0017] Since filters have the greatest impact on energy consumption, except for high-resolution input variables such as images, quantization of filters is advantageous. The resulting interaction, which conserves computer resources, allows machine learning systems to be used not only in stationary applications (e.g., in computing centers) but also in mobile applications (e.g., mobile data processing devices or embedded systems). Such quantized machine learning systems are particularly suitable for processing data measured near or in the sensors themselves, which can be compressed and transmitted in a more bandwidth-efficient manner.

[0018] The image may be a lidar image, a radar image, an ultrasound image, or a camera image.

[0019] The input variable is a tensor that includes a sequence of multiple detected variables or images. The detected variables or images are arranged along a first dimension of the tensor. A filter is then applied along at least the first dimension. The application of a filter along the first dimension representing a time series is therefore called temporal convolution.

[0020] The advantage of this machine learning system is that the information contained along the time axis is processed together through temporal filtering. This makes the machine learning system recursive. However, recursive connections are not required, which typically cannot be strongly quantized. However, quantized filters allow for more efficient processing of sequences of input variables than with RNNs.

[0021] Furthermore, it is proposed that the filter is multidimensional and that the machine learning system is further configured to apply the filter to a sequence of segments which are all located at respectively the same coordinates of the image or at respectively the same coordinates of the intermediate result.

[0022] A segment comprises a plurality of data points or pixels at corresponding coordinates of a detected variable, image, or intermediate result. The segment can be one-dimensional or multi-dimensional; the number of filter dimensions must be selected accordingly. It should be noted that, in addition, when filtering along the first dimension, a plurality of pixels or data points are considered, all along, and in particular parallel to, the first dimension.

[0023] For example, the segment can be a line or a rectangle.Different orientations of the segment within the image are conceivable, for example vertically or horizontally or diagonally.

[0024] Furthermore, it is proposed that the machine learning system is furthermore configured to additionally apply the filter to a sequence of further segments which are respectively located at different coordinates within the image or respectively at different coordinates within the intermediate result.

[0025] In this case, the advantage is that additional information is added to the output variable by applying the filter along each of the detected variables. This means that the filter is applied to additional dimensions of the tensor. This can be referred to as temporal and spatial convolution.

[0026] Furthermore, it is proposed that, in addition to the filter coefficients, intermediate results of the machine learning system, in particular activations, are also quantized.

[0027] Furthermore, it is conceivable that the intermediate results each have a value from the predefinable quantized value range or from another predefinable quantized value range.

[0028] Furthermore, it is proposed that the filter coefficients and / or intermediate results are binary or ternary. This is understood to mean that the corresponding filter coefficients and / or intermediate results can assume up to two or three different values. Advantageously, these different values are represented using a maximum of one or two bits. Ideally, they are represented using one bit, since this results in a binary neural network, which is particularly computationally and memory-efficient.

[0029] Furthermore, it is proposed that the filter omit individual data points during filtering, in particular along the time dimension. This has the advantage that a larger field of view of the filter is achieved with fewer filter coefficients and thus it is possible to look further into the past.

[0030] Furthermore, it is proposed that the machine learning system includes a discarding operation (pooling) arranged directly after the filter.

[0031] A discarding operation is understood to mean that individual filtered data points or filtered pixels are ignored, in particular discarded. Preferably, local discarding (local pooling) is used, wherein data points within a cluster whose values are around zero are ignored. Alternatively, average pooling (average pooling) can be used.

[0032] Furthermore, it is proposed that the images or the detected variables are arranged successively according to a predefinable sequence, in particular according to the corresponding detected times of the images or the detected variables. Preferably, the arrangement is performed in ascending or descending order according to the detected times.

[0033] It should be noted that the machine learning system can be implemented not only in hardware but also in software or in a mixture of software and hardware, whereby less memory must be provided for storing parameters, so that the machine learning system according to the first aspect of the invention can also be installed and used in the smallest technical system.

[0034] In a second aspect of the present invention, a method, in particular a computer-implemented method, for operating the machine learning system of the first aspect of the present invention is proposed. The method comprises the following steps:

[0035] A plurality of detected variables are grouped into input variables for a machine learning system. Output variables are then determined from the input variables using filters. In this case, filters of the machine learning system are used and, in particular, temporal filtering of the input variables or intermediate results is performed.

[0036] According to a second aspect of the invention, images are detected successively at immediately subsequent points in time. If an output variable is determined, the detected new image is added to the input variable and one of the included images, in particular the image containing the longest time, is removed from the input variable.

[0037] Preferably, the images of the input variable are shifted by one position along a first dimension of the tensor, wherein the detected new image is added at the position that becomes free and the oldest detected image of the input variable is removed from the tensor.

[0038] Furthermore, it is proposed that variables, in particular images, are each assigned to a time point in the sequence of time points at which the corresponding variable was detected, wherein a new variable is detected at each time point and the detected new variable is added to the input variables in addition to the variable detected at the immediately preceding time point, and the detected variable assigned to the longest past time point is removed from the input variables.

[0039] In another aspect of the present invention, a method for training a machine learning system according to the first aspect is provided. The training comprises at least the step of providing training data. The training data comprises at least a plurality of image sequences or time series of detected variables. At least one output variable is then determined for each of the image sequences or time series. A cost function is then determined based on the determined output variables and the training data. The parameters of the machine learning system are then adapted based on the cost function so that the cost function is optimized with respect to predeterminable criteria. The parameters, in particular the filter coefficients, are then quantized.

[0040] Advantageously, quantization is used from the outset and the machine learning system is trained taking this quantization into account.

[0041] In another aspect, the machine learning system of the first aspect or the machine learning system generated according to the second aspect can determine an output variable based on the detected sensor variables of the sensor, in particular an image, which is then used to determine the control variable by means of the control unit.

[0042] Control variables can be used to control actuators of a technical system. A technical system can be, for example, an at least partially autonomous machine, an at least partially autonomous vehicle, a robot, a tool, a factory machine, or an aerial vehicle, such as a drone. Input variables can be determined, for example, based on detected sensor data and provided to a machine learning system. The sensor data can be detected by sensors of the technical system, such as cameras, or alternatively received externally.

[0043] In another aspect, a computer program is provided. The computer program is configured to implement one of the aforementioned methods of the second aspect of the present invention. The computer program includes instructions that, when executed on a computer, cause the computer to perform one of the aforementioned methods including all of its steps. Furthermore, a machine-readable storage module is provided, on which the computer program is stored. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Embodiments of the above-mentioned aspects are shown in the drawings and explained in more detail in the following description.

[0045] Figure 1 A schematic diagram showing the energy consumption of two different neural networks, each for processing a time series, plotted as a function of the quantization resolution, is shown;

[0046] Figure 2 A schematic diagram showing a deep neural network including filters for temporal convolution;

[0047] Figure 3 A schematic diagram showing temporal convolution;

[0048] Figure 4 A schematic diagram showing a flow chart of a method for running a neural network including a filter for temporal convolution;

[0049] Figure 5 Schematic diagram showing the use of a neural network including filters for temporal convolution for robotics. DETAILED DESCRIPTION

[0050] Figure 1 A schematic diagram of a diagram is shown, in which the measured energy consumption E of a neural network with temporal filtering (NN-temp) and the measured energy consumption E of an RNN are plotted against the quantization resolution Q. Note that both networks achieve similar results on the same dataset. The y-axis shows the energy consumption E, which is lowest at the origin of the diagram and increases with increasing distance from the origin. The x-axis plots the quantization resolution Q, which increases toward the origin of the diagram. That is, the closer to the origin, the higher the quantization resolution, in particular the number of bits used to represent the parameters. Below, an exemplary case is considered: a neural network with temporal filtering (NN-temp) has significantly higher energy consumption at high quantization resolutions than a similar RNN at the same quantization resolution.

[0051] If the resolution of the parameters of the two neural networks is reduced, the energy consumption of both networks decreases. Comparing the minimum quantization resolution of the neural network with temporal filtering (NN-temp) with the minimum quantization resolution of the RNN reveals that, while the quality of the results of the two neural networks remains essentially unchanged, the minimum quantization resolution of the neural network with temporal filtering (NN-temp) is significantly lower than that of the RNN. This means that, due to strong quantization, the neural network with temporal filtering (NN-temp) consumes less energy than the RNN.

[0052] Figure 2 A schematic diagram of a machine learning system (10), in particular a deep neural network, is shown. The machine learning system (10) comprises a plurality of layers (11) which are connected to one another in a predeterminable sequence. The machine learning system (10) determines an output variable y as a function of input variables (12).

[0053] The machine learning system (10) may output as output variable y a classification of the input variable (12), a segmentation of one of the images or one of the most recent detected images. Alternatively, the output variable y may be a regression.

[0054] In this embodiment, the input variable (12) is a tensor. The tensor comprises a plurality of successively detected images (13). The images are arranged along a first dimension of the tensor.

[0055] The layers may be fully connected layers, wherein at least one of the layers is configured to perform temporal filtering. Temporal filtering is used as the input variables (12) are propagated through the machine learning system. Temporal filtering is performed in a subsequent Figure 3 is explained in more detail in .

[0056] Figure 3 A schematic diagram showing temporal convolution, in other words temporal filtering, is shown.

[0057] Figure 3 The tensor (20) depicted in FIG can be an input variable (12) or an intermediate variable of the machine learning system (10). The intermediate variable is, for example, an output variable of one of the layers (11).

[0058] A coordinate system (23) is schematically depicted below the tensor (20) on the left. The coordinate system (23) comprises, by way of example, three dimensions (h, t, w) along which the tensor (20) extends. It should be noted that the images (13) are arranged continuously in the tensor (20) along the "time" dimension t. The dimensions h and w are, by way of example, spatial dimensions along which the image (13) extends.

[0059] The tensor (20) is now temporally filtered using the filter (21). This occurs as follows: the filter performs filtering along the tensor (20) along the "time" dimension t. The temporal convolution of the filter (21) with the tensor (20), in particular an exemplary progression of the temporal convolution, is schematically illustrated by the dashed arrow (22).

[0060] Temporal filtering can alternatively be performed using one of the other filters (21a, 21b, 21c). The other filters differ from one another in that they have different dimensionalities. For example, the other filter (21c) is a three-dimensional filter. A possible curve of the convolution of the three-dimensional filter (21c) with the tensor (20) is depicted by way of example by the dashed arrow (22b) next to the three-dimensional filter (21c). It should be noted that if the convolution is performed according to the dashed arrow (22b), the filtering of the three-dimensional filter (21c) with the tensor (20) is not only temporal filtering but also spatial filtering.

[0061] Figure 4 A schematic flow chart of a method for operating a machine learning system (10) is shown.

[0062] In a first embodiment, the method begins with step S40 , in which input variables ( 12 ) are provided to the machine learning system ( 10 ).

[0063] In the following step S41, the provided input variables (12) are propagated through the machine learning system (10). During the propagation, the input variables (12) or intermediate variables of the machine learning system (10) are temporally filtered by means of a filter (21), see Figure 3 The filter has quantized filter coefficients. Preferably, all other parameters of the machine learning system (10) are also quantized.

[0064] In the following step S42 , the output variable of the machine learning system is output.

[0065] Then, in the subsequent step S43 , a new image is detected and added to the input variable ( 12 ) and the image contained in the input variable ( 12 ) is removed from the input variable ( 12 ).

[0066] The method is then repeatedly executed with all its steps S40 to S43 .

[0067] In another embodiment of the method for operating a machine learning system (10), steps S40 to S43 are used to train the machine learning system (10). During training, the parameters of the machine learning system are additionally optimized after step S43 according to a cost function (loss function). The cost function in this case characterizes the deviation between the determined output variable of the machine learning system (10) and the assigned training output variable of the respectively used input variable. When determining the output variable, already quantized parameters can be used. The optimization of the cost function with respect to the parameters of the machine learning system can be performed, for example, with the aid of a gradient descent method, such as "backpropagation (through-time)". In this case, a gradient descent method is used in order to minimize or maximize the cost function depending on the parameters. After the parameter optimization, the parameters, in particular the filter coefficients, are quantized. Advantageously, a parameter resolution of at least 32 bits is used for optimizing the parameters. During quantization, the resolution is then reduced, for example to 8 bits or even to 1 bit.

[0068] In a refinement of the first embodiment of the method for operating a machine learning system (10), output variables of the machine learning system (10) can optionally be used after step S43 in order to control the at least partially autonomous robot in accordance with the output variables. Figure 5 It is shown as an example in FIG.

[0069] Figure 5A schematic diagram of an at least partially autonomous robot is shown, which in a first embodiment is provided by an at least partially autonomous vehicle (100). In another embodiment, the at least partially autonomous robot can be a service robot, an assembly robot or a stationary production robot, or alternatively an autonomous flying object such as a drone.

[0070] The at least partially autonomous vehicle (100) may include a detection unit (30). The detection unit (30) may be, for example, a camera that detects the environment of the vehicle (100). The detection unit (30) may be connected to a machine learning system (10). The machine learning system (10) determines an output variable based on provided input variables, for example provided by the detection unit (30), and based on a plurality of parameters of the machine learning system (10). The output variable may be forwarded to a control unit (40).

[0071] The control unit (40) controls the actuators based on the output variables of the machine learning system (10), preferably controlling the actuators so that the vehicle (100) performs a collision-free maneuver. In a first embodiment, the actuators may be an engine or a braking system of the vehicle (100).

[0072] In another embodiment, the partially autonomous robot can be a tool, a factory machine, or a production robot. The material of the workpiece can be classified with the aid of a machine learning system (10). The actuator in this case can be, for example, an electric motor that operates a grinding head.

[0073] Furthermore, the vehicle (100), in particular a partially autonomous robot, comprises a computing unit (50) and a machine-readable storage element (60). A computer program can be stored on the storage element (60), which comprises instructions that, when executed on the computing unit (50), result in the machine learning system (10) being operated by means of the computing unit (50).

Claims

1. A machine learning system (10) comprising at least one filter (21), wherein the machine learning system (10) is configured to determine an output variable based on the input variable (12) using the filter (21), wherein the input variables (12) include a time series of images, wherein the machine learning system (10) is configured to apply the filter (21) to a sequence of pixels that are all at respectively the same coordinates of the image or at respectively the same coordinates of an intermediate result of the machine learning system (10), wherein the filter coefficients of the filter (21) are quantized with the highest 8 bits, wherein the quantization of the filter coefficients of the filter (21) is obtained when training the machine learning system taking into account the quantization, The filter (21) is configured to omit individual pixels of the image or the intermediate result when using the filter (21).

2. The machine learning system (10) of claim 1, wherein the machine learning system (10) is a deep neural network having convolutional layers.

3. The machine learning system (10) according to claim 1, wherein the filter (21) is three-dimensional, and The machine learning system (10) is configured to apply the filter (21) to a sequence of segments, which are all at respectively the same coordinates of the image or at respectively the same coordinates of the intermediate result.

4. The machine learning system (10) according to claim 3, The machine learning system (10) is further configured to additionally apply the filter (21) to a sequence of further segments, each of which is located at a different coordinate within the image or at a different coordinate within the intermediate result.

5. The machine learning system (10) according to any one of claims 1 to 4, In addition to the filter coefficients, the intermediate results are also quantized.

6. The machine learning system (10) according to claim 5, The filter coefficients and / or the intermediate results are binary or ternary.

7. The machine learning system (10) according to any one of claims 1 to 4, The machine learning system (10) includes a discarding operation arranged directly after the filter (21).

8. The machine learning system (10) according to any one of claims 1 to 4, wherein the images are arranged successively according to a predefinable order, The filter (21) is applied to a sequence of pixels or segments, the pixels or segments being respectively located at the same coordinates, and the pixels or segments of the sequence being arranged according to the order.

9. A method for operating a machine learning system (10) according to any one of the preceding claims, comprising the steps of: Grouping a sequence of images into input variables of the machine learning system (10); When using the filter (21), an output variable is determined as a function of the input variable.

10. The method according to claim 9, wherein the images are respectively detected successively at directly subsequent points in time, wherein if the output variable is determined, the detected new image is added to the input variable and one of the images contained in the input variable is removed from the input variable, Therein, a further output variable is subsequently determined as a function of the input variable using the filter.

11. The method according to claim 10, One of the images contained therein is the one containing the longest time.

12. A method for training a machine learning system (10) according to any one of claims 1 to 8, comprising the steps of: Providing training data, the training data comprising at least a plurality of image sequences and assigned training output variables; determining at least one output variable for each of the image sequences; determining a cost function based on the determined output variables and the training output variables; Adapting the parameters of the machine learning system (10) according to the cost function so that the cost function becomes optimal with respect to predeterminable criteria; and Quantifying parameters of the machine learning system (10).

13. The method according to claim 12, The parameters are filter coefficients of the machine learning system (10). 14 . A computer program product comprising a computer program, the computer program including instructions which, when executed by a computer, are configured to cause the computer to carry out the method according to claim 9 . 15 . A machine-readable storage element having a computer program stored thereon, the computer program comprising instructions configured to cause a computer to carry out the method according to claim 9 when the instructions are executed by the computer.

Images