Neural network data compression device and method

Abstract

The invention provides an artificial neural network data compression device and method. A floating point data acquisition module is used for acquiring floating point data of an artificial neural network; the floating point data storage module is used for storing floating point data existing in a shared index floating point format; the floating point data operation module is used for performing artificial neural network forward operation on floating point data existing in a shared index floating point format; wherein the shared index floating point format is specifically characterized in that n pieces of floating point data are encoded into a block, each piece of data in the block is composed of an E part, an S part and an M part which respectively represent an index part, a symbol part and a mantissa part, and the data are shifted rightwards according to the difference edf between the E and a shared index Es. According to the device, a high data compression rate can be realized by utilizing a shared index, meanwhile, good DNN model precision is kept in a low-precision format, and compared with an existing low-precision floating-point format, the device is more suitable for realizing a product circuit of end-side equipment.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a neural network data compression device and method. Background Art

[0002] In recent years, deep learning algorithms represented by deep neural networks (DNNs) have achieved remarkable results in solving high-level abstract cognitive problems, such as target detection, image retrieval, language recognition, and intelligent question and answer. However, as the scale of deep learning network models continues to increase, the model depth becomes deeper and deeper, and the complexity of calculations and data and memory requirements also increase. With the development of the mobile Internet era, the popularity of smartphones and other end-side devices has brought great convenience to people's daily communication, information acquisition, learning, and work. Deep learning is increasingly used as a general solution to give electronic devices intelligent attributes, but the storage and calculation of neural network models on end-side devices is still a huge challenge. In order to improve the efficiency and ability of end-side devices to process image and video data, while meeting the limitations of storage space and power consumption, in recent years, the use of low-precision fixed-point numbers has become a widely used neural network inference format on end-side devices. However, this method has certain limitations. First, it is difficult to maintain good accuracy for some neural networks to be quantized using fixed-point operations, such as those involving image super-resolution, image denoising, and other scenarios where there is insufficient conditions for parameter fine-tuning. However, directly deploying a floating-point precision neural network on the end device will result in a large amount of hardware overhead and low computational efficiency. Summary of the invention

[0003] The object of the present invention is to provide a neural network data compression device and method, which includes a new floating point format, called shared exponent floating point (SFP), which aims to combine the advantages of low data precision and shared exponent schemes. It can achieve the same storage density as int8 without sacrificing inference accuracy and without fine-tuning. Specifically, SFP can achieve high data compression rate by using shared exponent while maintaining good DNN model accuracy on low-precision formats. And it is more suitable for product circuit implementation of end-side devices than existing low-precision floating point formats.

[0004] To achieve the above-mentioned purpose, the present invention discloses an artificial neural network data compression device, including a floating point data acquisition module, a floating point data storage module and a floating point data operation module, characterized in that:

[0005] The floating point data acquisition module is used to obtain floating point data of the artificial neural network;

[0006] The floating point data storage module is used to store floating point data in a shared exponent floating point format;

[0007] The floating point data operation module is used to perform artificial neural network forward operation on floating point data in a shared exponent floating point format;

[0008] The shared exponent floating-point format specifically encodes n floating-point data into a block, each data in the block consists of three parts: E, S and M, representing the exponent, sign and mantissa respectively, and is shifted right according to the difference edif between E and the shared exponent Es.

[0009] The present invention also discloses an artificial neural network data compression method, which comprises the following steps:

[0010] S100: Obtain floating point data of the artificial neural network;

[0011] S200: storing the floating point data in a shared exponent floating point format;

[0012] S300: performing an artificial neural network forward operation on the floating point data in a shared exponent floating point format;

[0013] The shared exponent floating-point format specifically encodes n floating-point data into a block, each data in the block consists of three parts: E, S and M, representing the exponent, sign and mantissa respectively, and is shifted right according to the difference edif between E and the shared exponent Es.

[0014] The present invention uses the shared exponent floating point data for a neural network to increase the efficiency of the neural network. The neural network data compression device and method of the present invention have strong scalability, which is mainly reflected in the following points: (1) There can be any n data sets with a shared exponent. Commonly used schemes are 4, 8, 16, etc. (2) Each SFP format data can retain any n bits. Commonly used schemes are 4, 6, 8, 16, 32, etc. (3) Further, each SFP format data, E, C, M can freely retain n bits. (4) The mapping relationship of the C part is free, which realizes the compressed storage of the shift exponent difference, and this compression encoding method is freely designed. (5) The base storage method of the M part is free, and it can fix the decimal point or explicitly store the decimal point. In addition, the computing device and method of the present invention have a wide range of applications, mainly in the following aspects: (1) The SFP data format can be used for memory data compression storage. (2) The SFP data format can be used for AI field calculations. According to the data flow required for dedicated computing, the SFP data format can be specially adapted and adjusted to simplify its circuit complexity and better adapt to the edge computing hardware platform. (3) The SFP data format can be used for AI computing, including convolutional neural network reasoning, convolutional neural network training, transform reasoning and training, and can also be used for data compression and training acceleration of large models. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a structural diagram of an artificial neural network data compression device provided in one embodiment of the present invention;

[0016] Figure 2 It is a specific schematic diagram of the shared exponent floating point format SFP provided in one embodiment of the present disclosure;

[0017] Figure 3 is a schematic diagram of the structure of a floating-point data type provided in one embodiment of the present disclosure;

[0018] Figure 4 is a schematic diagram of the SFP8 format provided in one embodiment of the present disclosure;

[0019] Figure 5 is a schematic diagram of the SFP8_124 format provided in one embodiment of the present disclosure;

[0020] Figure 6 It is a schematic diagram of the SFP4 format provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following is combined with Figures 1 to 6The embodiments of the present invention are described in detail. It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Those skilled in the art should know that the following specific embodiments or specific implementations are a series of optimized settings listed by the present invention to further explain the specific content of the invention, and these settings can be combined or used in association with each other, unless the present invention clearly states that some or a specific embodiment or implementation cannot be associated or used together with other embodiments or implementations. At the same time, the following specific embodiments or implementations are only used as the most optimized settings, and are not to be understood as limiting the scope of protection of the present invention.

[0023] See also Figure 1 In one embodiment, the present invention discloses an artificial neural network data compression device, including a floating point data acquisition module, a floating point data storage module and a floating point data operation module, characterized in that:

[0024] The floating point data acquisition module is used to obtain floating point data of the artificial neural network;

[0025] The floating point data storage module is used to store floating point data in a shared exponent floating point format;

[0026] The floating point data operation module is used to perform artificial neural network forward operation on floating point data in a shared exponent floating point format;

[0027] The shared exponent floating-point format specifically encodes n floating-point data into a block, each data in the block consists of three parts: E, S and M, representing the exponent, sign and mantissa respectively, and is shifted right according to the difference edif between E and the shared exponent Es.

[0028] In this embodiment, the shared-exponential floating point format is called SFP (Shared-exponential Floating Point). Figure 2 The definition of the SFP format is described. This format considers encoding n floating point numbers as a block, such as Figure 2 As shown in (a) in the figure, each data in the block consists of three parts: E, S and M, which represent the exponent, sign and mantissa respectively. S and M constitute the decimal part of the number, and E is the exponent part. Figure 2 As shown in (b), when a number adjusts its E to Es, its M must be shifted right according to the difference between E and Es. We define the exponential difference between E and Es as edif.

[0029] Floating point format is the basis of floating point arithmetic. IEEE 754 standard specifies the representation of 32-bit floating point numbers (float 32). A 32-bit floating point number consists of three parts: 1-bit sign bit, 8-bit exponent part and 23-bit mantissa part. The value of a 32-bit floating point number can be expressed as follows:

[0030]

[0031] Where s represents the sign bit used to indicate the positive or negative of the floating-point number. When s=0, V is a positive number; when s=1, V is a negative number. E represents the exponent bit used to identify the exponent of the floating-point number. The mantissa part M is used to identify the mantissa of the floating-point number, which is greater than or equal to 1 and less than 2.

[0032] Take the representation of the floating point number 0.15625 as an example to explain its storage method. When a floating point number is stored, its highest bit is the sign bit, followed by the 8-bit exponent part, and finally the 23-bit mantissa part. When 0.15625 is stored as a floating point number, its binary representation is:

[0033] 0b0011 1110 0010 0000 0000 0000 0000 0000

[0034] The low-bit standard floating-point data format is mainly divided into half-precision (FP16, BF16) and 8-bit precision (FP8) according to the floating-point precision. Figure 3 shown.

[0035] The first is FP16. FP16 follows the standard floating-point format, but adjusts the number of data bits in each part to reduce the floating-point precision. FP16 has a total of 16 bits, including 1 sign bit, 5 exponent bits, and 10 mantissa bits.

[0036] Brain Float 16, proposed by Google Brain, is also designed for machine learning. It consists of 1 sign bit, 8 exponent bits (consistent with FP32) and 7 decimal bits (lower than FP16). So the accuracy is lower than FP16, but the representation range is consistent with FP32, and it is easy to convert between FP32 and FP32.

[0037] FP8 is an 8-bit floating point representation. FP8 uses two representations: E4M3 and E5M2, where E stands for the exponent bit (Exponent) and M stands for the mantissa bit (Mantissa). Within the representation range, E4M3 consists of 1 sign bit, 4 exponent bits and 3 mantissa bits, and its representation is more accurate, while E5M2 consists of 1 sign bit, 5 exponent bits and 2 mantissa bits, and it has a wider dynamic range. Compared with traditional FP16 (16-bit floating point numbers) and FP32 (32-bit floating point numbers), it significantly reduces storage and improves computing throughput.

[0038] In another embodiment, the shared index is obtained by selecting the maximum index value in the block.

[0039] For this embodiment, by sharing the exponent within a block, we can effectively compress the size of the floating point data. Specifically, we select the maximum exponent value in the block as the shared exponent Es.

[0040] The shared exponent takes the maximum value to ensure that other numbers in a group are less than the maximum value. The exponent of a smaller number can be increased by shifting the mantissa to the right (dividing the mantissa by the power of 2). If the shared exponent does not take the maximum value, then when the exponent is greater than the exponent shared by the group, the mantissa needs to be shifted to the left. Since the mantissa width is fixed, there is no space for shifting to the left. Therefore, the shared exponent of each group can only select the maximum exponent in the group.

[0041] In another embodiment, the shared exponent floating-point format may also encode n floating-point data into a block, wherein the block consists of an e-bit exponent and n elements, each element consisting of a c-bit correction component, a 1-bit sign component and an m-bit mantissa component.

[0042] In this embodiment, the above method leads to two problems. First, since the leading derivative of M is not always equal to 1, the leading derivative of M must be stored explicitly, resulting in additional storage overhead. Second, due to the right shift, the precision of the smaller numbers in a block will be greatly reduced. Therefore, further, such as Figure 2 As shown in (c), the leading part of M is encoded as a value C, which represents the number of zeros in the leading derivative of M. This C value is called the correction bit. In this way, S, C, and M follow the form of floating point format.

[0043] Formally, the SFP format is defined as: n floating point numbers are represented as a block. An SFP block consists of an e-bit exponent and n elements, each of which consists of a c-bit correction component, a 1-bit sign component, and an m-bit mantissa component.

[0044] In another embodiment, the value Vi of any floating point number in the block is calculated by the following formula:

[0045]

[0046] Among them, S i Represents the sign bit of any floating point number. Dec represents the difference between the exponent E of a number encoded with the correction bit C and the maximum exponent value Es of the block. C i Represents the correction bit of any floating point number, M i Represents the mantissa of any floating-point number, and n represents the number of floating-point numbers to be shared.

[0047] For this embodiment, one of the key innovations of the SFP format is the definition of a correction bit C, which is used to encode the difference between the exponent E of a number and the maximum exponent value Es of the block. dec is an abbreviation for decode and represents a way of encoding the difference between the exponent E of a number and the maximum exponent value Es of the block using a correction bit C.

[0048] In another embodiment, the correction bit C represents the number of zeros in the leading derivative of M.

[0049] In another embodiment, the present invention also discloses an artificial neural network data compression method, the method comprising the following steps:

[0050] S100: Obtain floating point data of the artificial neural network;

[0051] S200: storing the floating point data in a shared exponent floating point format;

[0052] S300: performing an artificial neural network forward operation on the floating point data in a shared exponent floating point format;

[0053] The shared exponent floating-point format specifically encodes n floating-point data into a block, each data in the block consists of three parts: E, S and M, representing the exponent, sign and mantissa respectively, and is shifted right according to the difference edif between E and the shared exponent Es.

[0054] In another embodiment, a bit explicit decimal point scheme is proposed. Based on the above analysis, we can further compress the encoding method of the correction bit C to achieve hardware-oriented customization optimization. We can let two floating-point numbers share a C encoding. Since each number has three edif cases, there are a total of 9 combinations of two numbers. In our design, a 3-bit Ccomb is used to encode these 9 combinations. Specifically, the C value of each number is defined by the following formula:

[0055]

[0056] According to the analysis of the distribution of edif, we use the first three C=0, 1, and 2 to represent edif = 0, edif = 1, and edif > 1, respectively. In addition, a special case C = 3 is proposed, that is, edif ≥ 1, which is a combination of C = 1 and C = 2. According to the formula According to the definition of , we use 3-bit Ccomb to encode the values ​​of C0 and C1 of two floating point numbers. As shown in Table 1, we merge the cases of (C0 = 1, C1 = 2) and (C0 = 2, C1 = 1) into (C0 = 3, C1 = 3), so that 3-bit Ccomb can cover all combinations of (C0, C1).

[0057]

[0058] Table 1

[0059] The above format is called SFP8, such as Figure 4 As described, an SFP block contains 16 floating point numbers which are further split into 8 groups of two numbers. All numbers in the block share an 8-bit exponent Es. Each group of two numbers has a 3-bit Ccomb for the combined correction part shared by the two numbers (1.5 bits are allocated per number on average). In addition, each number has a 1-bit sign and 6-bit mantissa. Therefore, an SFP8 block contains a total of 128 bits. In this case, the overall density of the proposed format is equivalent to INT8 and FP8, as 16 numbers are stored in 128 bits. Therefore, this new format is called 8-bit equivalent SFP (SFP8).

[0060] In another embodiment, an 8-bit fixed decimal point scheme is proposed. The core difference between the 8-bit fixed decimal point scheme and the 8-bit explicit decimal point scheme lies in whether the decimal point and the leading derivative 1 of the floating point number are explicitly stored. In the 8-bit explicit decimal point scheme, since the mantissa M part needs to be shifted along with the exponent difference edif, while the leading derivative 1 is not explicitly stored in the 8-bit fixed decimal point scheme, it also means that the mantissa M part does not need to be shifted along with the exponent difference edif. We store edif by encoding the correction bit C. For data with a larger edif, that is, data smaller than the representation range of edif, we discard it as 0. The specific scheme is shown as follows: Figure 5 shown.

[0061] The above format is called SFP8_124, such as Figure 5As described, an SFP block contains 8 floating-point numbers, and all numbers in the block share an 8-bit exponent Es. Each number has a 2-bit C for storing edif. The representation range of edif is 0~3, which means that data with edif less than or equal to 3 can be represented, and each data maintains the implicitly stored leading 1 plus the 4-bit mantissa part, totaling 5 bits of mantissa precision. Therefore, an SFP8 block contains a total of 64 bits. The overall density of the format is still equivalent to INT8 and FP8, because 8 numbers are stored in 64 bits, and because E, C, and M, the three parts occupy 1bit, 2bits, and 4bits respectively. Therefore, this new format is called SFP8_124.

[0062] In this solution, we further improve the data representation accuracy within the range of edif less than or equal to 3, but correspondingly discard the data representation range where edif is greater than 3.

[0063] In another embodiment, a 4-bit fixed decimal point solution is provided. Compared with the 8-bit decimal point solution, the core difference between the two is that this solution is equivalent to 4 bits, and since the bit width is half of 8 bits, the bit widths of the corresponding C and M parts are reduced accordingly.

[0064] The above format is called SFP4, such as Figure 6 As described, an SFP block contains 8 floating-point numbers, and all numbers in the block share an 8-bit exponent Es. Each number has a 1-bit C for storing edif, and the representation range of edif is 0~1, which means that data with edif less than or equal to 1 can be represented, and each data maintains the implicitly stored leading 1 plus 1-bit mantissa part, totaling 2 bits of mantissa precision, so an SFP4 block contains a total of 32 bits. The overall density of the format is still equivalent to INT4 and FP4, because 8 numbers are stored in 32 bits, so this format is called SFP4.

[0065] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields, and the above specific embodiments are only illustrative and instructive, rather than restrictive. A person of ordinary skill in the art can also make many forms under the guidance of this specification and without departing from the scope of protection of the claims of the present invention, all of which belong to the protection of the present invention.

Claims

1. An artificial neural network data compression device, comprising a floating point data acquisition module, a floating point data storage module and a floating point data compression module, characterized in that: The floating point data acquisition module is used to obtain floating point data of the artificial neural network; The floating point data storage module is used to store floating point data in a shared exponent floating point format; The floating point data operation module is used to perform artificial neural network forward operation on floating point data in a shared exponent floating point format; The shared exponent floating-point format specifically encodes n floating-point data into a block, each data in the block consists of three parts: E, S and M, representing the exponent, sign and mantissa respectively, and is shifted right according to the difference edif between E and the shared exponent Es.

2. The device according to claim 1, preferably, the sharing index is obtained by selecting the maximum index value in the block.

3. According to the device according to claim 1, the shared exponent floating-point format can also be to encode n floating-point data into a block, wherein the block consists of an e-bit exponent and n elements, each element consists of a c-bit correction component, a 1-bit sign component and an m-bit mantissa component.

4. The apparatus according to claim 3, wherein the value V of any floating point number in the block i Calculated by the following formula: ， in, S i Represents the sign bit of any floating point number. Dec represents the difference between the exponent E of a number encoded with the correction bit C and the maximum exponent value Es of the block. C i Represents the correction bit of any floating point number, M i Represents the mantissa of any floating-point number, and n represents the number of floating-point numbers to be shared.

5. The apparatus according to claim 4, wherein the correction bit C represents the number of zeros in the leading derivative of M.

6. An artificial neural network data compression method, the method comprising the following steps: S100: Obtain floating point data of the artificial neural network; S200: storing the floating point data in a shared exponent floating point format; S300: performing an artificial neural network forward operation on the floating point data in a shared exponent floating point format; in, The shared exponent floating-point format specifically encodes n floating-point data into a block, each data in the block consists of three parts: E, S and M, which represent the exponent, sign and mantissa respectively, and is shifted right according to the difference edif between E and the shared exponent Es. The method according to claim 6 , wherein the shared index is obtained by selecting a maximum index value in a block.

8. According to the method of claim 6, the shared exponent floating-point format can also be to encode n floating-point data into a block, wherein the block consists of an e-bit exponent and n elements, each element consists of a c-bit correction component, a 1-bit sign component and an m-bit mantissa component.

9. The method according to claim 8, wherein the value Vi of any floating point number in the block is calculated by the following formula: ， in, S i Represents the sign bit of any floating point number. Dec represents the difference between the exponent E of a number encoded with the correction bit C and the maximum exponent value Es of the block. C i Represents the correction bit of any floating point number, M i Represents the mantissa of any floating-point number, and n represents the number of floating-point numbers to be shared.

10. The method according to claim 9, wherein the correction bit C represents the number of zeros in the leading derivative of M.

Images