Processing unit for training a deep neural network (DNN) and method for determining a characteristic value of a data set of training data

Abstract

A processing unit (10) for training a deep neural network (DNN) using data sets of training data in the high-bit floating-point format, comprising a memory unit (30); a processor chip (20) having a processor unit (22) and having a memory controller unit (30) that comprises a PHY controller (32) for communication with the memory unit (30) and an interface (36) for communication with the processor unit (22). The memory controller unit (30) has a quantization unit (40) for changing the floating-point format of the training data of the data sets, and an estimation unit (42) for estimating a characteristic value of the training data of the data sets. The memory controller unit (30) is designed and configured to reduce the floating-point format of the training data on the basis of the characteristic value of the training data and to perform a data transmission of the reduced-format training data to and from the memory unit (30).

Description

[0001] Rhineland-Palatinate University of Technology Kaiserslautern-Landau

[0002] Processing unit for training a deep neural network (DNN) and method for determining a characteristic value of a data set of training data

[0003] The present invention relates to a processing unit for training a so-called deep neural network (DNN) with data sets of training data in high-bit floating-point format, comprising a memory unit (DRAM) and a processor module. The processor module has a processor unit and a memory controller unit, which comprises a PHY controller (physical controller) for communicating with the memory unit and an interface for communicating with the processor unit. The invention further relates to a method for determining a characteristic value of a data set of training data for a deep neural network (DNN).

[0004] Deep neural network (DNN) models have evolved rapidly over the past decade and now boast complex context-sensitive learning capabilities with continuously improving accuracy. The training phase of DNN models involves processing large datasets requiring several GB of memory and high computational demands. DNN training on computing platforms such as GPU and TPU accelerators consumes a significant amount of energy compared to inference. A large portion of the total energy is consumed by off-chip memory units, such as DRAMs, located outside the processor chip or chips for memory-intensive DNN tasks on the aforementioned computing platforms. For example, a single 64-bit GDDR6 DRAM transaction consumes 350–480 pJ, while a 32-bit floating-point multiplication (FP32) on the latest 7 nm technology requires only 1.31 pJ.For neural network accelerators such as Cambricon-X, eBrain-11, and NeuroCube, it has been reported that memory energy consumption ranges between 50 and 90% of the total energy. This memory energy consumption has a direct impact on overall energy efficiency.

[0005] Since training DNNs is very memory-intensive, the high energy consumption is particularly noticeable during training. Training deep neural networks (DNNs) requires a significant number of memory accesses, primarily due to the large intermediate results that must be stored in memory after each layer of the DNN. Furthermore, DNN model parameters, known as weights, are often large, especially in newer models or computational models such as transformers. Due to the limited memory capacity of processors, these weights and other training data must also be stored in DRAM. DRAM is a high-density and high-energy memory element, which presents a challenge when training DNNs.

[0006] In particular, the document US 2020 / 0 210 839 A1 can be regarded as the state of the art.

[0007] Furthermore, the term training data includes all data used in connection with the training of a DNN, in particular the weights mentioned.

[0008] The present invention is therefore based on the problem of reducing energy consumption during training of a deep neural network. Within the scope of the invention, several approaches for possible solutions were investigated in detail and further developed in an inventive manner.

[0009] A promising solution to the above problem is to compress data before it is written to DRAM memory.

[0010] One way to achieve this is to use narrow bit-width data formats, which reduce the number of bits allocated to each variable. For example, instead of a standard 32-bit floating-point number, an 8-bit floating-point format can be used, reducing the data size by four. The disadvantage is that a narrower bit width results in information loss due to the smaller range of numbers that can be represented. For example, the 32-bit floating-point data type can represent numbers in the absolute range of 10 ( ' 45) up to 10 (38) while the 8-bit floating point data type can only represent numbers in the absolute range of 10 ( ' 4) up to 10 (5) can represent.

[0011] State-of-the-art DNN training is computed using the standard FP32 and FP16 formats. The FP32 format consists of 1 sign bit, 8 exponent bits, and 23 mantissa bits. Recently, there has been a trend toward computing DNN training using a narrow-bit-width data format (e.g., 8-bit). Quantizing the data to 8 bits instead of storing it in 32-bit format allows for more data words to be packed into the DRAM bus, reducing the total number of transactions by a factor of four compared to FP32. In 2019, the first experiments with hybrid 8-bit floating-point-based DNN training were conducted in the field. This approach relies on assigning different exponent sizes for forward and backward passes based on the requirements of the DNN operation. However, this requires a modification of the conventional accelerator processing units.

[0012] Furthermore, due to the narrow numerical range of FP8, the back-calculation requires multiplying a scaling factor to fit the data into the reduced range representable by FP8. Other modern methods also rely on the use of a scaling factor. However, the determination of the scaling factor is based on offline experiments with a trained model.

[0013] The main problem with offline approaches is that a full training run in the FP32 data format must be performed for each new dataset or DNN topology. However, there are a multitude of datasets for many real-world applications. Therefore, offline-based FP8 quantization effectively results in no net energy reduction. Rather, it significantly increases the time and effort required.

[0014] It was therefore recognized that determining a scaling factor offline is not expedient. This requires online identification of the DNN domain requirements and a corresponding adaptation of the data format. Compatibility with a conventional accelerator architecture must be maintained, and online adaptation of the data format must not delay training or only marginally, and it must not incur significant area / energy overhead.

[0015] The problems and requirements identified are solved with a processing unit for training a deep neural network (DNN) having the features of claim 1 and with a method having the features of claim 11.

[0016] In one aspect, the present invention relates to a processing unit for training a deep neural network (DNN) with data sets of training data in high-bit floating-point format, preferably with a format of >32-bit or >64-bit. The processing unit comprises a processor module having a processor unit and a memory controller unit, and a memory unit arranged external to the processor module or chip. The memory controller unit comprises a PHY controller (physical controller with coding) for communicating with the memory unit and an interface for communicating with the processor unit, e.g., an AXI interface, and preferably a digital channel controller.

[0017] The memory controller unit has a quantization unit for converting or changing the floating-point format of the training data of the data sets. It also has an estimation unit for estimating a characteristic value of the training data of the data sets.

[0018] The memory controller unit is designed and configured to change or reduce the floating-point format of the training data based on the characteristic value of the training data and to transfer the format-reduced training data to and from the memory unit. However, the transfer of the training data from the memory controller unit to the processor unit always takes place in a high-bit floating-point format, e.g., in a 32-bit format.

[0019] In a further aspect, the invention relates to a method for determining a characteristic value of a data set of training data for a deep neural network (DNN) in order to implement a change in the floating-point format of the training data during training of the DNN. This involves converting the training data into the corresponding format. This preferably occurs online or on-the-fly, i.e., during training and not in a preparatory step, i.e., not offline.

[0020] The method according to the invention comprises the steps according to claim 11.

[0021] Further aspects of the invention relate to a deep neural network which has been trained by means of the processing unit according to the invention and to a computer program product corresponding to the method with program code for carrying out the steps of the method when the program code is executed on a computer, as well as to a storage medium on which a computer program is stored which, when executed on a computer, effects execution of the method described herein.

[0022] Preferred embodiments of the invention are described in the dependent claims. It is understood that the features mentioned above and those to be explained below can be used not only in the respective combinations specified, but also in other combinations or on their own, without departing from the scope of the present invention. In particular, the method and the computer program product can be implemented according to the embodiments described for the processing unit in the dependent claims, and vice versa.

[0023] The processing unit according to the invention has a quantization unit that converts the training data from a high-bit floating-point format to a low-bit floating-point format and takes all necessary measures for the conversion, including determining a characteristic value for the training data. Since the quantization unit is included in the memory controller unit, the processor unit (also called processor, CPU, computing core) (within the chip or processor module) remains unchanged, as does the memory unit (e.g., DRAM) outside the chip.

[0024] According to the invention, a processing unit and a methodology are proposed for identifying range requirements during the training phase in ongoing operation. By means of the invention, any given DNN can be automatically trained based on statistical analyses. In the first iterations of the method (epochs), preferably in the first two to three iterations, which are carried out in the conventional high-bit floating-point format, e.g., FP32 format, an analysis of DNN parameters of the training data is carried out. Furthermore, information on range requirements is extracted. The subsequent iterations of training are then carried out with training data in the low-bit floating-point format with format-reduced data, e.g., with FP8 compression, in order to save energy.The invention proposes an online method based on statistical analysis to convert FP32 data into FP8 data with minimal information loss during training. This involves applying a variable "warp" or change to adjust the data range, but with the appropriate "warp value" calculated online, i.e., during training.

[0025] For example, DNN training can consist of three different phases: forward path, backward path, and gradient computation. Each phase prefers its own data in different regions; therefore, the presented methodology is applied separately to each phase, compressing the data differently for each phase.

[0026] The invention preferably uses the median metric in statistical analysis because it is a robust metric. The median of the DNN parameters in the initial epochs is determined and compared with the mean number of FP8 data formats with different exponent biases to find the closest match. The floating-point format has a parameter called exponent bias that determines the position of the data format range. The optimal data format is then selected for compression.

[0027] The precise calculation of the median requires a lot of memory and is very slow. Therefore, an approximate method or estimation is used, i.e., a sufficiently accurate approximation to determine the median value. This method determines the maximum and minimum values ​​of the training data and partitions the range between them, preferably logarithmically. Each data value is then inserted into the appropriate partition (range, value range), and the approximate median value is determined by comparing the partitions or ranges. This method is faster and requires fewer hardware resources. The same process is preferably repeated twice to minimize errors and increase the accuracy of the median determination. All hardware blocks required for compression and finding the median are integrated into the memory controller unit.This means that the operating principle does not require any changes to the processor's hardware architecture and can be used in any existing computer platform.

[0028] In a preferred embodiment, the estimation unit in the memory controller unit according to the invention is designed and configured to determine the characteristic value of the training data in an iterative process. For example, the method according to claim 11 can be used for this purpose.

[0029] Further preferably, the estimation unit is designed and configured to determine the characteristic value of the training data in an iterative process with a maximum of four iteration steps, wherein the deviation from the true characteristic value of the training data corresponds to a maximum of a factor of 2. Consequently, a sufficient quality of the characteristic value of the training data is determined in just four iteration steps, since it has been found within the scope of the invention that a deviation of the characteristic value from the true value of at most 2 has proven sufficiently accurate for training deep neural networks. In a particularly preferred embodiment, the estimation unit is configured to determine the characteristic value with a deviation of at most a factor of 2 from the true characteristic value in a maximum of three iteration steps, particularly preferably in a maximum of two iteration steps.

[0030] Within the scope of the invention, it was found that even increasing the iteration steps by more than four, for example to ten or 15, does not result in an improvement in the characteristic value, which is noticeable in the result when using the training data in practice.

[0031] In a preferred embodiment of the processing unit, the memory controller unit is configured to change and adapt the floating-point format of the training data during processing of the training data and during training or the training of the deep neural network. This has the significant advantage that, on the one hand, training is not delayed by changing the data format of the training data. On the other hand, this results in the processing unit, and in particular the processor module comprising the processor unit and memory controller unit, not requiring more space. Rather, the processor unit and large parts of the memory controller unit remain unchanged.The additional energy requirement is also negligible and is far outweighed by the energy savings resulting from transferring the training data in low-bit floating-point format between the processor chip and the memory unit.

[0032] A preferred embodiment of the processing unit provides that the quantization unit of the memory controller unit includes the estimation unit. This allows for a very compact design. The space requirement is thus very small.

[0033] Particularly preferably, the memory controller unit communicates the training data to the processor unit in high-bit floating-point format, so that no changes need to be made to the processor unit. The processor unit itself remains unchanged. Converting the training data into a different floating-point format does not impair the function of the processor unit. Thus, the inventive method and the inventive design of the processing unit have no influence on the processor unit and can be used with already known and available processor units.

[0034] Particularly preferably, the memory controller unit is configured and designed to convert the training data transmitted from the memory unit to the memory controller unit in low-bit floating-point format into high-bit floating-point format. This ensures that communication with the processor unit takes place in high-bit floating-point format.

[0035] Particularly preferably, the quantization unit of the memory controller unit is configured to convert the training data back and forth between the high-bit floating-point format and the low-bit floating-point format, i.e., to perform an upconversion and a downconversion. Preferably, the training data present in the high-bit floating-point format is present in a 32-bit floating-point format, and the low-bit floating-point format is present in an 8-bit floating-point format. Consequently, the training data is preferably converted, i.e., converted back and forth, between an 8-bit floating-point format and a 32-bit floating-point format. The conversion occurs in both directions.

[0036] In a preferred embodiment, the quantization unit of the processing unit comprises an upconverter and a downconverter, wherein the upconverter is configured to convert training data from a low-bit floating-point format to the high-bit floating-point format. The downconverter is configured to convert training data from a high-bit floating-point format to a low-bit floating-point format. Preferably, an upconversion or downconversion between the 32-bit format and the 8-bit format also occurs here.

[0037] A preferred embodiment of the processing unit provides that the memory unit is a DRAM unit. Such memory units are typically used in modern computer systems, using, for example, DDR3 or DDR4 chips.

[0038] Preferably, the memory controller unit is designed and configured to carry out the method according to the invention as claimed in claim 11 or 12.

[0039] Since memory access to the memory unit, such as DRAM access, plays a significant role in the energy consumption of DNN training, the invention proposes that DRAM data transfers be performed using a low-bit floating-point format (low-bit FP format), for example, an 8-bit format. The training data is thus converted or "quantized" into this format. All data is quantized before being written to the DRAM and converted back to the data format of the accelerator core (processor core or processor unit) (e.g., 16-bit, 32-bit) after being retrieved from the DRAM.

[0040] The invention is preferably used for the DRAM level; however, it is in principle applicable to any level in the memory hierarchy. The inventive approach allows the calculations and training to be retained in the original format: All new hardware units required for converting the training data ("quantization") and identifying the DNN domain requirements are integrated into the memory controller unit. For example, the inventive method is compatible with any computing core (processor unit) without major architectural changes, in particular without modifications.

[0041] The invention preferably uses an FP8 format (8-bit FP format) with 1-bit symbol, 5-bit exponent and 2-bit mantissa or equally preferably an 8-bit FP format with 1-bit symbol, 4-bit exponent and 3-bit mantissa.

[0042] The typical FP8 format range is insufficient for all training data, such as backpropagation and gradient computation. Therefore, changing the floating-point bias is one way to shift the range of the FP8 data format to the range desired by the application.

[0043] According to the invention, the appropriate bias value is calculated during training using statistical analysis. The bias value is determined by accessing the data in FP32 format, preferably in the first iterations, called epochs, (e.g., < 5), which are sampled for the statistical online analysis. The determined bias value is then preferably used in the remaining epochs (e.g., > 5) for FP8 quantification, i.e., the FP8 format conversion. A person skilled in the art will recognize that the teaching disclosed herein preferably relates to a novel lEEE-like floating-point number format. The disclosed data format can provide a variable dynamic range. The dynamic range of the disclosed numerical format can be adjusted by changing a bias value.

[0044] In particular, the disclosure relates to hardware units that analyze data and adapt the data format range to a given range.

[0045] In particular, the invention can be a hardware unit that can be integrated into any accelerator or processor for neural networks, preferably with improved energy efficiency. Quantization can be achieved while maintaining compatibility with a variety of known hardware platforms.

[0046] In particular, the hardware unit has high throughput and can determine a median in a short time. The median can be determined without having to access the entire data spectrum. Preferably, an online estimation of the median can be performed.

[0047] The disclosed teaching essentially relates to new hardware units that have been developed. Methods for DNN training are preferably provided functionalities of the hardware.

[0048] The invention is described and explained in more detail below using selected embodiments in conjunction with the accompanying drawings. They show:

[0049] Figure 1 shows a processing unit according to the invention with a memory unit and a processor module which comprises a processor unit and a memory controller unit;

[0050] Figure 2 is a detailed drawing of the processor module from Figure 1; Figure 3 is a detailed drawing of an estimation unit of a memory controller unit from Figure 2;

[0051] Figure 4 is a detailed drawing from Figure 3 relating to an investigation unit;

[0052] Figure 5 shows a detailed drawing for determining a median value; and

[0053] Figure 6 is a schematic flow chart of the method according to the invention.

[0054] Figure 1 shows a processing unit 10 of a deep neural network. The processing unit is designed and configured to train the deep neural network with data sets of training data in high-bit floating-point format. A high-bit floating-point format refers to a data format for floating-point data that comprises at least 32 bits. Accordingly, the term low-bit floating-point format refers to a data format for floating-point data whose bit count is less than or equal to 16. A low-bit floating-point format is preferably an 8-bit floating-point format.

[0055] The processing unit 10 comprises a memory unit 12, which is preferably embodied as a DRAM memory 14. The processing unit further comprises a processor module 20 with a processor unit 22 and a memory controller unit 30.

[0056] The memory controller unit 30 includes a PHY controller 32 for communication with the memory unit 12, an optional channel controller 34 and an interface 36 for communication with the processor unit 22, which may be implemented as an AXI interface.

[0057] The memory controller unit 30 further comprises a quantization unit 40 for changing the floating-point format of the training data of the data sets for the deep neural network (DNN). The quantization unit 40 is thus used to convert the data format of the training data. The memory controller unit 30 is further configured and set up to determine or estimate a characteristic value of the training data of the data sets. In this case, a recursive or iterative determination of the characteristic value can be performed, in which the training data of the data sets are evaluated and processed.

[0058] Figure 2 shows a detailed drawing of the processor module 20 with the processor unit 22 and the memory controller unit 30. The quantization unit 40 includes an estimation unit 42, which is used to determine the characteristic value of the training data of the data sets. The quantization unit 40 further has an upconverter 44, which is configured to convert training data from a low-bit floating-point format into the high-bit floating-point format. A downconverter 46 is also part of the quantization unit 40 and is configured to convert or transform training data from the high-bit floating-point format into a low-bit floating-point format. This involves a conversion from the 32-bit format to an 8-bit format and vice versa.

[0059] The characteristic value of the training data is preferably a bias value, which is used to convert or transform the training data from a high-bit floating-point format to a low-bit floating-point format and vice versa. The bias value thus serves to adapt the value range of the low-bit floating-point format to the value range of the high-bit floating-point format, so that as little information as possible is lost.

[0060] According to the invention, the bias value or characteristic value of the training data is determined by preferably determining the median value of the training data in 32-bit format. The value range of the 8-bit format is then shifted such that the median value of the training data is also included in the 8-bit format and its value range. For this conversion and transformation, a bias value is used which ensures that the value range of the 8-bit floating-point format covers the majority of the data in 32-bit floating-point format and in particular the median value of the data in 32-bit floating-point format. This can be done, for example, by using a table or a calculation rule or a formula to assign the bias value as the characteristic value to the median value.

[0061] The quantization unit 40 is therefore primarily responsible for converting write data from FP32 format to FP8 format and read data from FP8 format to FP32 format. This unit 40 is designed to operate in the background without blocking the computing core (processor unit 22) or its data accesses. The quantization unit 40 therefore does not require any additional memory accesses for statistical analysis.

[0062] Figure 3 shows a detailed drawing of the estimation unit 42 for determining the median value, on which the estimate of the bias value or the characteristic value is based. In the preferred embodiment according to Figure 3, the estimation unit 42 comprises a min-max determiner 50, a range unit 60, and a determination unit 70 for determining the median value of the training data. The estimation unit 42 further has a bias unit 48, in which the characteristic value, in this case the bias value, for converting the floating-point format is determined from the median value present at the output of the determination unit 70. The min-max determiner 50 according to Figure 3 determines the minimum value and the maximum value, as well as the value in between, from the training data. These values ​​are determined in the first iteration of the method according to the invention.

[0063] Based on the minimum and maximum of the training data, the total value range is determined in the range unit 60, which is shown in detail in Figure 4. Here, the minimum and maximum values ​​are subtracted from each other in a subtractor 62. A subsequent divider 64 divides the total value range into a plurality M of ranges. The ranges are also referred to as value ranges, intervals, or sometimes as bins. In a next iteration step, the incoming training data in high-bit floating-point format is divided into the existing ranges 68 by a demultiplexer 66. The ranges 68 are designed such that, in their entirety, they reflect the total value range of the training data.

[0064] Figure 5 shows the determination unit 70 in detail, which compares the incoming training data with the respective ranges. This takes place in a plurality of comparators 72, whereby the training data is compared with the value range of the individual ranges. If a training data item lies within the corresponding value range, a counter 74 is incremented. The counter 74 is followed by a testing unit 76, in which the value of the counter is compared with half (N / 2) of the total number of training data items in the value range. If the total number in counter 74 exceeds half of the total number N / 2 of training data items in the value range, this range 68 is determined as the median value. If necessary, the mean value of the training data present in the range can optionally be calculated. This mean value is then used to determine the characteristic value of the training data.This can be done in various ways, for example, by comparing it with a table containing such values, which can be determined experimentally. Alternatively, the characteristic value can be determined from the median value using a formula or calculation rule, or by other methods known to those skilled in the art.

[0065] The counter reading of counter 74 in the testing unit 76 is preferably only checked once all training data have been distributed across the individual ranges 68. If the counter reading of counter 74 of the first range is below the threshold value N / 2 (half of the total number of training data items in the value range), this value is fed to an adder 78 and linked there with the counter reading of counter 74 of the next range 68. This sum of these values ​​is then also checked against the threshold value N / 2 in the testing unit 76 of the next range 68. The check is initially performed on the ranges with the smallest value range. In a preferred embodiment, the value ranges of the ranges 68 are logarithmically distributed.

[0066] In the next iteration step, the values ​​from the range containing the median value are then defined as the new value range. These values ​​are then fed to the determination unit 70. In this way, an iteration is performed that outputs a very precise median value. Even just four iterations, preferably three iterations, leads to a result with only a very small deviation from the actual median value. For example, the deviation of the median value with four iterations from the median value determined with 50 iterations leads to an error in the characteristic value of a maximum of 2. However, such an error is well tolerable and does not represent a change in the training data that is relevant in practice.

[0067] Figure 6 shows the schematic process flow of the method according to the invention for determining the characteristic value of a data set of training data for a deep neural network. The characteristic value is used to change the floating-point format of the training data during training of the deep neural network. In a first step S10, the training data is received in a high-bit floating-point format, preferably from a storage unit. The data is preferably in 32-bit floating-point format.

[0068] In a second step S12, the minimum value and the maximum value of the training data are determined in a first iteration step.

[0069] In a third step S14, the value range is divided between minimum and maximum values ​​for a predetermined number M of ranges, i.e., sub-value ranges or intervals into which the total value range is divided. The number or number of ranges is predetermined and can depend, for example, on the training data or the deep neural network. In a fourth step S16, the total number N of training data is determined. The training data is assigned to individual ranges and the number W of training data contained in a range is counted in a fifth step S18. This preferably takes place in a further iteration step, epoch.

[0070] In a sixth step S20, the numbers W of data present in the ranges are added, starting with the smallest range, until the sum of the numbers W exceeds a threshold. This threshold is determined from half of the total number N of training data in the value range. This range is also determined as the median range in the sixth step S20.

[0071] In a seventh step S22, the median range is selected. Here, the value range of the median range is divided into the specified number M of ranges.

[0072] In an eighth step S24, these values ​​are used to repeat the fourth to seventh steps, i.e., steps S16 to S22, but this time with the training data and values ​​determined in the median range of the last iteration step. This iteration loop can be run multiple times, resulting in an improvement in the determined median range.

[0073] In a ninth step S26, the mean of the median rank after the last iteration step is determined. The number of iterations performed is preferably between two and four.

[0074] In a tenth step S28, the characteristic value of the training data is determined based on the mean of the median rank. This can be done, as described above, using a table or calculation rule, or in another way. In a preferred embodiment of the method, the characteristic value is a bias value, on the basis of which the floating-point format of the training data is converted.

[0075] The invention shown in the figures is described again in other words.

[0076] The figures show, among other things, a novel hardware design for online median calculation, preferably implemented as a bias estimator. The design processes the incoming data in an optimized manner without blocking the processing core (processor unit) or requiring additional memory unit or DRAM accesses. A preferred hardware design for median determination is shown in Figures 2 to 5. The proposed design requires multiple iterations to calculate the median, with each iteration corresponding to a so-called epoch. This multi-iteration calculation of the median is preferably based on the premise that the variation of the median between epochs is minimal. In the first epoch, the so-called Min / Max block (Min-Max Determiner 50) compares the FP32 data arriving at the memory controller with the previous value to determine the minimum and maximum values ​​(Min-Max values).

[0077] At the end of the first epoch, the min-max values ​​are used to identify the data range and divide the range into a predetermined or predefined number M of ranges, also called "bins." The division of the bins and their ranges is preferably done in the range unit 60 and also preferably on a logarithmic scale.

[0078] In the second epoch, all incoming FP32 data from the memory controller is compared with the bin ranges, and the bin counter (counter 74) is incremented if the data is within the bin range. At the end of the second epoch, the bin counter values ​​are added sequentially from lowest to highest to determine the bin (range) at which the "N / 2" transition occurs (hereafter referred to as the half-bin), where "N" is preferably the total number of samples. The median unit (determination unit 70) uses a separate counter to count "N" in the respective epoch. The range in which this intersection occurs is the median value, and the average of its range is approximately equal to the median.

[0079] To increase the accuracy of the median value, the range of the "half-bin" of the second epoch is preferably further divided into "M" bins.

[0080] In the third epoch, the bins (ranges) are configured to the new values, and the bin counting is repeated to fine-tune the median value. This method preferably requires a total of four epochs to determine the median value with a low error, i.e., a negligible error in practice. Practical implementation has shown that this unit prefers to process the data in the background and also prefers to add only a maximum of 40 cycles (equivalent to 40 ns for a 1 GHz frequency) of delay at the end of the epoch for range allocation and half-life identification. This latency is very low compared to the 360 ​​ns required by the memory controller unit for each DRAM refresh operation. The DRAM refresh operation preferably occurs at an interval (period) of 7.8 ps, while the epoch interval is preferably in the ms range.

[0081] Preferably, the bias estimator (estimation unit 42) calculates four bias values, one for each type of training data, i.e., forward-pass activation, backward-pass error, weight gradient, and weight. The bias estimator has individual min-max blocks, range registers, and bin counter registers for each training phase, with only one set shown in the figures. The FP32 data (training data in high-bit floating-point format) is separated and processed accordingly based on a user signal of the AXI interface bits indicating the type of data. Within the scope of the invention, the actual median was compared with the approximate median at the end of the 4th iteration. The resulting deviation by the approximate median is a maximum of two. The approximate median of the 4th epoch compared to the actual median of the 50th epoch.Epoch (with more epochs, no significant change could be detected) also leads to a deviation of a maximum of two, resulting in a slight loss of accuracy. Additionally, synthetically generated extreme data samples were introduced in the first epoch (i.e., 10). 40 and 1 O 10 ) to negatively influence the min-max value and compare the median error after four epochs.

[0082] Example of an evaluation:

[0083] PyTorch and QPyTorch were used to evaluate the accuracy results of the libraries. The QpyTorch library was modified to support floating-point data format with variable bias. After the bias estimation epochs, the modified library was used to convert the DNN data (training data) to FP8 (8-bit floating-point format) based on the calculated bias. The median calculation was performed based on a software-level simulation model of the proposed median hardware unit. The hardware implementation of all blocks (units) of the adaptive quantization unit—the bias estimator, the upconverter, and the downconverter—was designed in Verilog, and the results were extracted by Synopsis' Design Compiler using Global Foundry's FDSOI, 22-nm technology.Two open-source tools were used to calculate the DRAM energy and latency reduction of the FP8 data format. First, SCALE-Sim, a cycle-accurate open-source systolic array simulator, was used to generate the data access requirements to the DRAM memory controller for different types of training data, i.e., forward-pass activation, backward-pass error, and weight gradient. The topology configuration of SCALE-Sim was updated according to the training phase (i.e., forward-pass or backpropagation) to obtain data access requirements for each type. Second, DRAMSys, a DRAM subsystem (memory controller + DRAM), was used to determine the energy and latency of DRAM accesses. The evaluations were performed only for DDR3 and DDR4, since the open-source power tool (DRAMPower) integrated with DRAMSys only supports these devices.

[0084] In the following, the accuracy of the validation set after training according to the invention, the area and power consumption of the (adaptive) quantization unit as well as the results of the energy reduction in DRAM data access are briefly presented.

[0085] A. Accuracy

[0086] DNN training accuracy was investigated for several image datasets using the inventive methodology in comparison to various standard and state-of-the-art data formats. The other formats used for comparison were IEEE-754 FP32, FP16, BFIoat16, TinyFloat12, and FP8, which is based on manual bias selection. The accuracy results were obtained after 50 training epochs using an ADAM optimizer with a learning rate of 0.001. The resulting accuracy reduction of FP8 compared to the reference data format FP32 was minimal (i.e., <1% on average). Furthermore, the new online bias detection method achieved similar or, in some cases, even better results than the state-of-the-art manual bias selection.

[0087] B. Adaptive quantization unit: area and power

[0088] The total area of ​​the (adaptive) preferred quantization unit is 0.019 mm 2 in 22 nm technology. The area overhead of this unit compared to the area of ​​the digital logic of an exemplary memory controller unit is 6.3%, with the area of ​​the digital logic of the memory controller unit being 0.890 mm 2 in 65 nm technology, which was linearly scaled to 22 nm technology (0.301 mm 2All blocks (units) are designed to operate at a frequency of 1 GHz to meet the data access requirements of the compute core (processor unit) without additional delays. The bias estimator consumes the highest power of the three blocks at 10.8 mW. However, this unit is only active during the bias calculation epochs (i.e., 4 epochs or iterations) and is deactivated for the rest of the training (i.e., 46 epochs). Therefore, the average power consumption of the bias estimator is 0.864 mW for 50 epochs. This is very low compared to the typical power consumption of DDR4 DRAM devices (i.e., 1-2 W, as known in the state of the art) and is negligible in practical applications.

[0089] C. DRAM access analysis

[0090] DRAM energy consumption was investigated for the following number of requests: forward = 13925992, reverse = 13862080, and gradient = 13926181 requests, each with different data widths. The data requests were generated by the SCALE-Sim tool for Resnet18. SCALE-Sim's systolic array and on-chip memory configurations correspond to the two well-known hardware accelerators, namely Google TPUv4 and Qualcomm Cloud AI 100. DRAM energy consumption decreases linearly with data width.

[0091] For example, in a TPU-like accelerator with DDR4 DRAM, the forward pass energy is reduced from 5.35 mJ to 1.53 mJ. According to the invention, with four epochs (iterations) in FP32 and 46 epochs in FP8 format, the total DRAM energy was reduced by a factor of 3.07 compared to conventional FP32 data format training for 50 epochs. Within the scope of the invention, energy results were only determined for DDR3 and DDR4 DRAM. This was due to the limitation of the DRAM Power tool required in DRAMSys for energy calculation. In summary, it can be stated that with the inventive conversion of the 32-bit FP training data into 8-bit FP training data using the inventive processing unit, the energy consumption of the memory unit, preferably the DRAM, during DNN training can be significantly reduced.The invention is based on varying the bias of the training data in the FP8 format to adapt the range to the DNN-specific domain. The solution is based on a statistical online analysis approach to determine the appropriate bias value during training. In a preferred embodiment, the median was chosen as the statistical metric due to its robustness.

[0092] To be compatible with any DNN processing core (processor unit) without major architectural changes, the quantization unit according to the invention is integrated into the memory controller unit. For example, the total energy consumption of the memory unit (DRAM) could be reduced by 3.07 times compared to conventional FP32 transactions, resulting in only a slight deterioration in accuracy of <1%. The invention can thus also be used for a wide range of deep neural networks (DNNs), other AI tools, machine learning instances, or other DNN domains such as NLP and federated learning, i.e., parallel learning on multiple devices.

[0093] The invention has been comprehensively described and explained with reference to the drawings and the description. The description and explanation are to be understood as exemplary and not restrictive. The invention is not limited to the disclosed embodiments. Other embodiments or variations will become apparent to those skilled in the art upon use of the present invention and upon careful analysis of the drawings, the disclosure, and the following claims.

[0094] In the claims, the words "comprising" and "having" do not exclude the presence of further elements or steps. The undefined article "a" or "an" does not exclude the presence of a plurality. A single element or unit can perform the functions of several of the units recited in the claims. An element, unit, device, and system can be partially or completely implemented in hardware and / or software. The mere mention of some measures in several different dependent claims should not be understood to mean that a combination of these measures cannot also be used advantageously. A computer program can be stored / distributed on a non-volatile data carrier, for example on an optical memory or on a solid-state drive (SSD).A computer program may be distributed together with hardware and / or as part of hardware, for example, via the Internet or via wired or wireless communication systems. Reference signs in the patent claims are not to be construed as limiting.

Claims

Patent claims 1 . Processing unit for training a deep neural network (DNN) with sets of training data in high-bit floating-point format, comprising: - a storage unit (12); - a processor module (20) with a processor unit (22) and a memory controller unit (30) comprising a PHY controller (32) for communication with the memory unit (12) and an interface (36) for communication with the processor unit (22); wherein - the memory controller unit (30) has a quantization unit (40) for changing the floating point format of the training data of the data sets and an estimation unit (42) for estimating a characteristic value of the training data of the data sets; - the memory controller unit (30) is designed and configured to reduce the floating point format of the training data based on the characteristic value of the training data and to carry out a data transfer of the format-reduced training data to and from the memory unit (12).

2. Processing unit according to claim 1, characterized in that the estimation unit (42) of the memory controller unit (30) is designed and configured to determine the characteristic value of the training data in an iterative process.

3. Processing unit according to the preceding claim, characterized in that the estimation unit (42) is designed and is set up to determine the characteristic value of the training data in an iterative process with a maximum of 4 iteration steps with a deviation of a maximum of a factor of 2 from the true characteristic value of the training data, preferably with a maximum of 3 iteration steps.

4. Processing unit according to one of the preceding claims, characterized in that the memory controller unit (30) is designed and configured to change the floating point format of the training data during the processing of the training data and in particular during the training of the DNN.

5. Processing unit according to one of the preceding claims, characterized in that the quantization unit (40) of the memory controller unit (30) comprises the estimation unit (42).

6. Processing unit according to one of the preceding claims, characterized in that the memory controller unit (30) communicates the training data in the high-bit floating-point format to the processor unit (22), wherein preferably the memory controller unit (30) converts training data in the low-bit floating-point format into the high-bit floating-point format.

7. Processing unit according to one of the preceding claims, characterized in that the quantization unit (40) is configured to convert training data back and forth between the high-bit floating-point format and the low-bit floating-point format, preferably between a 32-bit FP format and an 8-bit FP format.

8. Processing unit according to one of the preceding claims, characterized in that the quantization unit (40) comprises an up-converter (44) and a down-converter (46), wherein the up-converter (44) is designed to convert training data from a low-bit floating-point format into the high-bit floating-point format. convert, and the down converter (46) is arranged to convert training data from a high-bit floating point format to a low-bit floating point format.

9. Processing unit according to one of the preceding claims, characterized in that the memory unit (12) is a DRAM unit (14).

10. Processing unit, characterized in that the memory controller unit (30) is designed and configured to carry out the method according to claim 11 or 12.

11. A method for determining a characteristic value of a data set of training data for a deep neural network (DNN) in order to change the floating point format of the training data of the data sets during training of the deep neural network, comprising the following steps: a) receiving training data in a high-bit floating point format, preferably from a storage unit (12); b) determining a minimum value and a maximum value of the training data in a first iteration step; c) dividing the total value range between the minimum value and the maximum value into a predetermined plurality M of ranges (68); d) determining the total number N of training data; e) assigning the training data to the individual ranges (68) and counting the number W of training data contained in a range (68);f) adding the numbers W in the ranges (68) starting with the smallest range (68) until the sum of the numbers W exceeds half of the total number N of the training data in the value range and setting this range (68) as the median range; g) selecting the median rank and dividing the range of values ​​of the median rank into the predetermined plurality M of ranges (68); h) repeating steps d) to g) for a predetermined number of iterations; i) determining the mean of the median rank; j) determining the characteristic value based on the mean of the median rank.

12. Method according to the preceding claim, characterized in that the characteristic value is a bias value on the basis of which a conversion of the floating point format of the training data takes place.

13. Deep neural network trained with a processing unit according to one of claims 1 to 10.

14. A computer program product comprising program code for carrying out the steps of the method according to one of claims 11 or 12.

Images