MACHINE LEARNING SYSTEM, AS WELL AS A METHOD, A COMPUTER PROGRAM AND A DEVICE FOR CREATING THE MACHINE LEARNING SYSTEM

Description

Technical area

[0001] The invention relates to a machine learning system that has at least parameters with a lower resolution. Furthermore, the invention relates to a method and a computer program, as well as a device for creating the machine learning system. State of the art

[0002] Document DE 10 2017 218 889.8 discloses an artificial intelligence module designed to process one or more input variables through an internal processing chain to produce one or more output variables. The internal processing chain is defined by one or more parameters. A distribution module is provided, which is designed to determine the parameters depending on at least one statistical distribution.

[0003] It is possible to create quantized neural networks with extremely strong quantization, as shown, for example, by HUBARA, Itay, et al. "Quantized neural networks: Training neural networks with low precision weights and activations" The Journal of Machine Learning Research, 2017, Vol. 18, No. 1, pp. 6869-6898. C. Baskin, N. Liss, E. Zheltonozhskii, A.M. Bronstein and A. Mendelson, "Streaming Architecture for Large-Scale Quantized Neural Networks on an FPGA-Based Dataflow Platform," 2018 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), Vancouver, BC, Canada, 2018, pp. 162-169, doi: 10.1109 / IPDPSW.2018.00032 presents a streaming architecture for running QNNs on FPGAs. Disclosure of the invention

[0004] A first aspect relates to a computer-implemented machine learning system according to claim 1.

[0005] It should be noted that the parameters are advantageously stored in a memory, which requires less storage space by reducing their resolution.

[0006] A preceding layer is the layer that determines its output before the respective connected layer and provides its output to the respective connected layer as input. A subsequent layer is the layer that receives the output of the respective connected layer as input.

[0007] Connected on the input side can be understood to mean that an output variable of one of the layers, in particular of a preceding and / or a subsequent layer, is used as an input variable of the layer connected to this layer.

[0008] The machine learning system of the first aspect has the advantage that, due to the reduced resolution of said parameters, these parameters can be stored in memory particularly efficiently with minimal storage space requirements, without, surprisingly, compromising the performance of the machine learning system. Furthermore, the reduced resolution of the parameters can reduce the computational requirements of this machine learning system, since, for example, through clever exploitation of the computing architecture and the lower resolution of the parameters, the parameters can be processed in parallel. A further advantage is that, due to the resulting resource-efficient handling, the machine learning system can be used for both stationary (e.g., in data centers) and mobile applications (e.g., mobile data processing systems or embedded systems).Such compressed machine learning systems are particularly suitable for processing measured data close to the sensor or in the sensor itself, allowing measured data to be compressed and transmitted more bandwidth-efficiently.

[0009] The fact that the deliberate lower resolution, in particular stronger quantization, of those parameters of the layers that are connected to several other layers does not impair the performance, especially the accuracy of the results, of the machine learning system, is because the additional connections between further different layers provide differently processed output variables. This means that the layer that is connected to several layers receives more information and the input variables can be represented in a better or more completely described state space. Therefore, these layers are more robust against disturbances, which means that the parameters and activations of this layer can, for example, be more quantized without impairing the performance of the machine learning system. It should be noted that everything regardingthe resolutions of the parameters, can also be applied to the activations.

[0010] Using fewer bits to store the parameters requires less memory, allowing for a more compact design of the machine learning system. Since the energy consumption for storing and reading the parameters and activations is proportional to the number of bits, and for multiplications is quadratic to the resolution (in bits) of the parameters and activations, reducing the number of bits, especially through quantization, is particularly resource-efficient.

[0011] It is further proposed that the parameters of those layers which are connected to a further layer each have a size from a first list of possible, in particular discrete, sizes, and the parameters of those layers which are connected to a plurality of further layers each have a size from a second list of possible, in particular discrete, sizes.

[0012] For example, the sizes of the lists can be determined by determining cluster centers of the parameters, which can then be used to efficiently store the parameters in memory based on the, in particular discrete, cluster centers. The advantage of the lists is that the lists each contain, for example, the determined cluster centers. This means that suitable sizes can be assigned to the parameters by comparing the sizes of the parameters with the sizes of the lists, and the parameters have a lower resolution due to the smaller number of different possible sizes of the parameters. It should be noted that the lists can alternatively or additionally contain rounded values ​​of the determined parameters or the determined cluster centers. Alternatively, the lists can be determined separately for each layer.

[0013] It is further proposed that the first list contain more entries than the second list. It is conceivable that the sizes of the lists are quantized, with the values ​​in the second list being more highly quantized, i.e., in particular, having fewer quantization levels, than the values ​​in the first list.

[0014] Furthermore, it is proposed that the resolution of the parameters and / or activations depends on the number of layers directly connected on the input side.

[0015] It should be noted that the machine learning system can be implemented either exclusively in hardware or in a mixture of software and hardware, which means that less memory needs to be provided to store the parameters, which is why the machine learning system according to the first aspect of the invention can also be installed and used in the smallest technical systems.

[0016] Furthermore, it should be noted that each of the layers can be connected on the input side to at least two further layers of the plurality of layers, in particular that each layer has a skip connection. According to the above-mentioned embodiments of the first aspect of the invention, a machine learning system with these layers can have a lower resolution, in particular a stronger quantization, than an equivalent machine learning system in which these layers are connected on the input side to only one further layer.

[0017] A second aspect relates to a computer-implemented method according to claim 4 for creating the machine learning system proposed according to the first aspect.

[0018] Preferably, a compressed machine learning system is structurally identical to a provided machine learning system, which is why corresponding resolution of parameters or activations can be understood to mean that the parameters or activations can be found at the same position of the other machine learning system with the corresponding resolution.

[0019] It is further proposed that when reducing the resolution of the parameters, at least two lists, each with possible parameter sizes, be provided. The sizes of the list can be determined by means of quantization, clustering, or rounding the sizes of the changed parameters during training. The parameters of those layers whose input size is the output size of one of the further layers are each assigned a size from a first list of possible, in particular discrete, sizes, and the parameter of the layer whose input size is the majority of the output sizes of further layers is each assigned a size from a second list of possible, in particular discrete, sizes.

[0020] Furthermore, it is proposed that the training is carried out depending on a cost function, where the cost function has an additional term that penalizes the sizes of the parameters if they are not equal to a possible quantized size of the parameters.

[0021] The machine learning system, which was created according to the second aspect, determines an output variable depending on a detected sensor variable of a sensor, which output variable is then used to determine a control variable by means of a control unit.

[0022] The control variable will be used to control an actuator of a technical system. The technical system can be, for example, an at least partially autonomous machine, an at least partially autonomous vehicle, a robot, a tool, a machine tool, or a flying object such as a drone. The input variable can, for example, be determined based on acquired sensor data and provided to the machine learning system. The sensor data can be acquired by a sensor, such as a camera, of the technical system or, alternatively, received externally.

[0023] According to the invention, the machine learning system is alternatively configured to classify the sensor data.

[0024] Further aspects relate to a computer program according to claim 7, a machine-readable storage medium according to claim 8, and a device according to claim 9.

[0025] Embodiments of the above-mentioned aspects are illustrated in the accompanying drawings and explained in more detail in the following description. They show: Short description of the drawings

[0026] Fig. 1 is a schematic representation of a vehicle; Fig. 2 is a schematic representation of a deep neural network with a bridging connection and a memory; Fig. 3 is a schematic representation of an embodiment of the method for generating a machine learning system with compressed parameters; Fig. 4 is a schematic representation of an embodiment of an apparatus that can be used to train the machine learning system.

[0027] Figure 1shows a schematic representation of an at least partially autonomous robot, which in a first embodiment is provided by an at least partially autonomous vehicle (10). In a further embodiment, the at least partially autonomous robot can be a service, assembly, or stationary production robot, or alternatively an autonomous flying object, such as a drone.

[0028] The at least partially autonomous vehicle (10) can comprise a detection unit (11). The detection unit (11) can be, for example, a camera that detects the surroundings of the vehicle (10). The detection unit (11) can be connected to a machine learning system (12). The machine learning system (12) determines an output variable depending on a provided input variable, e.g., provided by the detection unit (11), and depending on a plurality of parameters of the machine learning system (12). The output variable can be forwarded to a control unit (13).

[0029] The control unit (13) controls an actuator depending on the output variable of the machine learning system (12). Preferably, the actuator is controlled in such a way that the vehicle (10) executes a collision-free maneuver. In the first embodiment, the actuator can be an engine or a braking system of the vehicle (10).

[0030] In a further embodiment, the semi-autonomous robot can be a tool, a machine tool, or a production robot. A workpiece material can be classified using the machine learning system (12). The actuator can be, for example, a motor that drives a grinding head.

[0031] Furthermore, the vehicle (10), in particular the semi-autonomous robot, comprises a computing unit (14) and a machine-readable memory element (15). A computer program comprising instructions that, when executed on the computing unit (14), result in the machine learning system (12) being operated by means of the computing unit (14).

[0032] In a further embodiment of the semi-autonomous robot, the control unit (13) comprises an authorization system. The authorization system decides whether an object, e.g., a detected robot or a detected person, has access to an area, depending on the output variable of the machine learning system (12). Preferably, the actuator, for example, a door-opening mechanism, is controlled by the control unit (13).

[0033] In an alternative embodiment, the machine learning system (12), which is connected to the detection unit (11) as in the first embodiment, can be used for building control. User behavior is detected by a sensor, for example, a camera or a motion detector, and the control unit controls, for example, a heat pump of a heating system depending on the output variable of the machine learning system (12). The machine learning system (12) can then be configured to determine which operating mode of the building control system is desired based on the detected user behavior.

[0034] In a further embodiment, the machine learning system (12) is used in a measuring system which is not shown in the figures. The measuring system differs from the vehicle (10) according to Figure 1in that the measuring system does not comprise a control unit (13). Instead of forwarding the output variable of the first machine learning system (12) to the control unit (13), the measuring system can store or display it, for example by means of visual or auditory representations.

[0035] It is also conceivable that, in a further development of the measuring system, the acquisition unit (11) acquires an image of a human or animal body or a part thereof. For example, this can be done using an optical signal, an ultrasound signal, or an MRI / CT method. In this further development, the measuring system can include the machine learning system (12), which is trained to output a classification depending on the input variable, e.g., which clinical picture may be present based on this input variable.

[0036] In a further embodiment, the machine learning system (12) can be integrated into a sensor or into the acquisition unit (11) in order to process the acquired sensor values ​​using the machine learning system (12). The sensor values ​​processed by the machine learning system (12) are preferably compressed or already evaluated sensor values, so that they can then be efficiently transmitted, for example, to the control unit (13). Preferably, at least the parameters are stored on a chip of the sensor or the machine learning system is integrated into hardware on the chip.

[0037] Figure 2shows a schematic representation of the machine learning system (12), which in this embodiment is provided by a deep neural network (20). In further embodiments, the machine learning system (12) can be provided by a convolutional neural network or a recurrent neural network.

[0038] The deep neural network (20) comprises several layers, each of which is connected to one another by means of connections (23) and each of which comprises several neurons (22). Furthermore, the deep neural network (20) has a bridging connection (24). The bridging connection (24) in Figure 2has the effect that an output variable from the first layer of the deep neural network (20) is forwarded directly to the third layer and provided there as an input variable. This bridges the second layer. It should be noted that the bridging connection (24) can also be configured in the opposite direction. For example, an output variable from the third layer can be supplied to a preceding layer as an input variable by means of a recurrent bridging connection.

[0039] The deep neural network (20) determines an output variable (25) layer by layer depending on an input variable (21) of the deep neural network (20). For this purpose, each layer determines an output variable depending on the input variable provided to it and the parameters of that layer. The output variable is then forwarded to the subsequent layers via the connections (23).

[0040] In this exemplary embodiment, the parameters of the respective layers can, for example, each have at least a first variable and / or a second variable and a third variable. The layers can each weight the input variable using the first variable and / or filter it using the second variable.

[0041] The layers can add the third value to the weighted / filtered input value. Optionally, this processed input value is further processed using an activation function (e.g., ReLu) depending on the parameter. Optionally, the layer can also include a batch normalization layer.

[0042] The special feature of the neural network (20) is that the parameters of the layer that receives the output variable of the first layer as input via the bridging connection (24) are stored in at least one memory (200) with a lower resolution than the parameters of the other layers that do not have a bridging connection. For example, the parameters of the third layer are stored in the memory (200) using 5 bits, and the parameters of the further layers are stored in the memory (200) using at least 8 bits. It should be noted that the activations can also be represented at a reduced resolution, e.g., 5 bits, due to the reduced resolution of the parameters.

[0043] Figure 3 shows a schematic representation of a method (30) for creating the machine learning system (12).

[0044] The method begins with step 31. Here, training data is provided to the machine learning system (12). The training data preferably comprises training input variables and respectively associated training output variables (labels). The training input variables can be, for example, images from a camera, and the training output variables can be, for example, a classification or segmentation of the camera images. Additionally or alternatively, the training data can comprise a plurality of labeled sound sequences, text excerpts, radar, lidar, or ultrasound signals.

[0045] In the following step 32, the machine learning system (12) is trained.

[0046] When training the machine learning system (12), the parameters of the respective layers of the machine learning system (12) are adapted in such a way that the machine learning system determines the respectively assigned output variables depending on the provided training input variables.

[0047] Preferably, the training is carried out as a function of a cost function which, for example, characterizes a difference between the determined output variables (25) and the training output variables. The cost function can be optimized with regard to the parameters using a gradient descent method. In the gradient descent method, the cost function is optimized iteratively, whereby a gradient with a resolution of 32 bits is usually used. The parameters can be adapted as a function of the gradient, whereby the cost function changes so that it is ideally minimized or maximized. Since the gradient usually has a resolution of 32 bits, the adapted parameters are stored in the memory (200) as a function of the gradient, preferably with a resolution of 32 bits.

[0048] After the training has been performed in step 32, step 33 follows. In step 33, the adapted parameters, each with a resolution of 32 bits, are quantized. In addition to the quantization of the parameters, the resolution of the activations can be reduced, preferably relative to the quantization of the respective layers.

[0049] The parameters can be quantized using linear quantization or nonlinear quantization, such as logarithmic quantization. In addition to the deterministic quantizations mentioned above, stochastic quantizations are also conceivable; see the documents mentioned above for more information.

[0050] The quantization can, for example, be carried out such that all layers have the same quantization, preferably a quantization with 8 bits. Then, those parameters of the layers that are connected to several further layers can be quantized more strongly, preferably with 5 bits. Alternatively, different quantizations can be used for the parameters of those layers that are connected to only one further layer, preferably quantizations with more than 8 bits, in particular 8 to 12 bits. It is also conceivable for those layers that are connected to several further layers to each have different quantizations, preferably quantizations with fewer than 8 bits, in particular 4 to 7 bits.

[0051] After the resolution of the parameters has been reduced, the parameters with reduced resolution are stored in the memory (200) in step 34.

[0052] This concludes the process (30). It should be noted that steps 32 to 34 can be repeated cyclically several times in succession until a predefined termination criterion is met. Optionally, when using multiple batches of training data, after completing step 34, the machine learning system can be retrained with the newly stored parameters, for example, for another batch, starting with step 32.

[0053] In a further embodiment, after step 32, a reduction in the resolutions, i.e. a quantization, of the parameters and / or the activations of the layers can be carried out additionally or alternatively depending on whether a bridging connection is present or not. In the event that the machine learning system (12) has at least one bridging connection, the resolutions are selected to be lower than in the case where it has no bridging connection. Additionally or alternatively, a suitable method can be selected from a multitude of different methods for reducing the resolution, each with different properties with regard to accuracy and / or computational effort, depending on the presence of the bridging connection. For example, a logarithmic quantization method can be used if no bridging connection is present, and a simpler one, e.g.A linear quantization method can be selected if a bridging connection is present. Advantageously, the parameters and activations are quantized with the same quantization resolution, especially the layers after the input layer that receives the input variable.

[0054] Figure 4shows a schematic representation of a device (40) for generating the machine learning system (12), in particular for carrying out steps 31 to 34 for generating the machine learning system (12). The device (40) comprises a training module (41) and a module to be trained (42). This module (42) to be trained contains the machine learning system (12). The device (40) for training the machine learning system (12) trains the machine learning system (12) depending on output variables of the machine learning system (12) and preferably with predeterminable training data. During the training, parameters of the machine learning system (12) that are stored in the memory (200) are adapted.

Claims

1. Computer-implemented machine learning system (12), which is a deep neural network (20), wherein the machine learning system (12) comprises a plurality of layers, each of which is connected to at least one further layer of the plurality of layers on the input side, wherein the layers are each configured to determine an output variable depending on activations, wherein the layers are furthermore each configured to determine the activations depending on at least one input variable of the respective layer and depending on parameters assigned to the respective layer, wherein a resolution of the parameters is dependent on whether or not one of the layers is connected to a plurality of layers directly on the input side, wherein a number of bits by means of which the parameters are stored, in particular in a memory, characterizes the resolution, wherein the resolution of the parameters of that layer which is connected to a plurality of layers directly on the input side is in each case at least more than one bit lower than the resolution of the parameters of those layers which are connected to exactly one layer directly on the input side, wherein the machine learning system (12) comprises an input layer and an output layer, which outputs an output variable, wherein the input layer receives a sensor variable of a sensor, and either the output layer is connected to a control unit and the control unit determines a control variable for controlling a physical actuator depending on the output variable or the output variable classifies the sensor variable.

2. Machine learning system (12) according to Claim 1 or 2, wherein the machine learning system (12) has at least one layer which is connected to exactly one layer directly on the input side, wherein the parameters or activations of those layers which are connected to exactly one layer directly on the input side each have a variable from a first list comprising a first predefinable number of quantized variables, and the parameters or activations of that layer which is connected to a plurality of layers directly on the input side each have a variable from a second list comprising a second predefinable number of quantized variables, wherein a quantization error of the variables of the first list is smaller than the quantization error of the variables of the second list.

3. Machine learning system (12) according to Claim 1 or 2, wherein if the output layer is connected to the control unit, depending on the determined control variable, an at least partially autonomous robot and / or a manufacturing system and / or a personal assistant and / or an access system and / or a monitoring system or a medical imaging system are / is controlled.

4. Computer-implemented method for creating a compressed machine learning system (12), wherein the created compressed machine learning system (12), which is a deep neural network (20), comprises a plurality of layers, each of which is connected to at least one further layer of the plurality of layers on the input side, wherein the layers are each configured to determine an output variable depending on activations, wherein the layers are furthermore each configured to determine the activations depending on at least one input variable of the respective layer and depending on parameters assigned to the respective layer, wherein a resolution of the parameters is dependent on whether or not one of the layers is connected to a plurality of layers directly on the input side, wherein a number of bits by means of which the parameters are stored, in particular in a memory, characterizes the resolution, wherein the resolution of the parameters of that layer which is connected to a plurality of layers directly on the input side is in each case at least more than one bit lower than the resolution of the parameters of those layers which are connected to exactly one layer directly on the input side, wherein the machine learning system (12) comprises an input layer and an output layer, which outputs an output variable, wherein the input layer receives a sensor variable of a sensor, and either the output layer is connected to a control unit and the control unit determines a control variable for controlling a physical actuator depending on the output variable or the output variable classifies the sensor variable, wherein the compressed machine learning system (12) is created from a provided machine learning system (12) with the following step: selecting a resolution of parameters of the compressed machine learning system (12) by reducing the corresponding resolution of parameters of the provided machine learning system (12), wherein the resolution of the parameters of the compressed machine learning system (12) is selected depending on whether or not one of the layers of the provided machine learning system (12) is connected to a plurality of layers directly on the input side, wherein the corresponding resolution of the parameters of that layer of the provided machine learning system (12) which is connected to a plurality of layers directly on the input side is reduced to a greater extent than the corresponding resolution of the parameters of those layers of the provided machine learning system (12) which are connected to exactly one layer directly on the input side, wherein the reduction of the resolution is carried out by means of a reduction of the number of bits for representing the variables.

5. Method according to Claim 4, wherein the reduction of the number of bits is carried out depending on a quantization, in particular a number of quantization levels.

6. Method according to either of the preceding Claims 4 and 5, comprising the following further steps: providing training data that respectively comprise input variables and output variables assigned to the input variables, training the provided machine learning system (12), wherein during training the parameters are adapted in such a way that, respectively depending on the input variables of the training data, the machine learning system (12) respectively determines their assigned output variable.

7. Computer program comprising instructions which, when they are executed by means of a computer, cause the computer to carry out the method (30) according to any of Claims 4 to 6.

8. Machine-readable storage medium (15) on which the computer program according to Claim 7 is stored.

9. Device (40) configured to carry out the method according to Claims 4 to 6.