Floating point representation method for efficient training and reasoning of intelligent model
By dividing the floating interval and Huffman coding to optimize floating point representation, the problems of complex floating point format conversion and high resource consumption are solved, efficient training and inference of intelligent models are realized, and the efficiency and adaptability of floating point data representation are improved. Provides new opportunities for hybrid precision optimization.
Patent Information
- Application Number
- CN202411114268.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-12-03
AI Technical Summary
In intelligent model training and inference, existing floating-point formats have problems such as complex conversion of different precisions, tight coupling of calculation and storage formats, and uneven distribution of floating-point data on the digital axis, resulting in high resource consumption and low efficiency.
By dividing the number axis into multiple floating intervals, allocating bits according to the density of the floating point number, and assigning unique identifiers to each interval using Huffman encoding, it realizes efficient representation of floating point numbers, and supports dynamic adjustment of the number of bits and precision conversion. Use table lookup mechanism instead of floating-point operations to decouple computation and storage formats.
It realizes rapid conversion between different precisions, improves the efficiency and adaptability of floating-point data representation, reduces resource consumption, breaks through the limitations of existing floating-point formats, and provides new opportunities for hybrid accuracy optimization of intelligent models.
Smart Images

Figure CN119067219A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of intelligent models, and in particular relates to a floating-point representation method for efficient training and reasoning of intelligent models. Background Art
[0002] From traditional machine learning models and shallow neural networks to later deep neural networks, Transformer models, and now large language models, the field of artificial intelligence has undergone tremendous changes. It has performed well in many fields such as image recognition, computer vision, speech processing, and natural language processing, and occupies an irreplaceable position in today's society. As people continue to pursue smart models with better performance, the complexity and number of parameters of mainstream smart models are showing an upward trend, which puts higher requirements on the storage and computing resources of the hardware platform and the efficiency of the software algorithm. Therefore, how to reduce the demand for storage and computing resources in the process of training and reasoning of smart models and improve the efficiency of training and reasoning is one of the current research hotspots.
[0003] Mixed precision optimization introduces low-precision floating-point representation during training and reasoning, partially or completely replacing the traditional single-precision floating-point representation (FP32) to participate in the calculation and storage during model training and reasoning. It is an effective means to optimize the model training and reasoning process, and can fundamentally reduce the amount of calculation and storage during model training and reasoning.
[0004] A floating point unit (FPU) is a separate processor or an important part of an integrated processor, implemented as a hardware circuit, that performs floating-point calculations. While early FPUs were separate processors, most are now integrated into a computer's CPU. The integrated FPUs in modern CPUs are very complex because they perform high-precision floating-point calculations while ensuring compliance with the rules governing these calculations as specified in the IEEE floating-point standard (IEEE 754).
[0005] The representation efficiency of the floating-point format is crucial to the effectiveness of mixed-precision optimization. In addition to the existing IEEE-754 standard floating-point formats FP16, FP32, and FP64, researchers have proposed new low-precision floating-point formats, such as FP24 proposed by AMD, BFloat16 (BF16) proposed by Google, and FP8 proposed by Intel. Although these low-precision floating-point formats have been used for training and reasoning optimization of intelligent models, they use a representation similar to the "sign bit + exponent + mantissa" defined in the IEEE-754 floating-point standard, and there are the following problems in model training and reasoning:
[0006] 1. Conversion between different floating-point formats is complicated. When the number of exponent bits of two floating-point formats is different, whether converting from high precision to low precision or from low precision to high precision, it is necessary to calculate the exponent value in the new floating-point format and adjust the mantissa field.
[0007] 2. The calculation format and storage format of floating points are tightly coupled. In order to support mixed-precision calculations, different precision representations of the same data need to be obtained during the calculation process. The existing IEEE-754-like floating-point representation method cannot support on-demand loading and calculation of the same floating-point data with different precisions. Instead, it needs to be loaded into memory in a certain format and converted to another format online, or multiple copies of a piece of data with different precisions need to be stored to meet the needs of mixed-precision calculations, which increases the conversion or storage overhead of floating-point data.
[0008] 3. Floating-point numbers are unevenly distributed on the number axis. In the IEEE-754-like representation method, floating-point numbers near the origin of the number axis are denser, while they become increasingly sparse away from the origin. The floating-point numbers generated and processed in the actual intelligent model are not necessarily all near the origin, and some data are concentrated in a certain area far from the origin. Figure 1 The probability distribution diagram of the weight and activation value of a network layer in the large language model ChatGLM-6B is shown. It can be observed that both tensors show highly clustered numerical distributions, but the positions of the two tensors on the number axis are different. The tensor on the left is mainly concentrated in the interval [2.0,3.5], while the tensor on the right is concentrated near zero. The IEEE-754-like floating-point format has higher representation accuracy for the right tensor than for the left tensor. Summary of the invention
[0009] In order to solve the above technical problems, the present invention proposes a floating-point representation method for efficient training and reasoning of intelligent models to solve the problems existing in the prior art.
[0010] To achieve the above object, the present invention provides a floating point representation method for efficient training and reasoning of intelligent models, comprising:
[0011] Based on the numerical distribution characteristics of floating-point numbers in the intelligent model, the number axis is divided into multiple floating intervals;
[0012] Allocating a determined number of bits to each floating interval based on the floating point number density of each floating interval;
[0013] Based on the number of bits, assigning a unique interval identifier to each floating interval by Huffman coding;
[0014] The representation of each floating point number consists of the interval identifier and the specific position within the floating interval to achieve optimization of intelligent model training and reasoning.
[0015] Preferably, the numerical distribution feature is a multi-modal clustered distribution of tensor data.
[0016] Preferably, the floating point representation based on the floating interval supports the representation of data distributed at any position on the number axis.
[0017] Preferably, the number of bits is dynamically adjusted according to the floating point number density of the floating interval.
[0018] In order to achieve the above technical objectives, the present invention also provides a floating-point calculation method for training and reasoning of an intelligent model. Based on the floating-point representation method for efficient training and reasoning of intelligent models, a table lookup mechanism is used to replace floating-point operations.
[0019] Preferably, the process of using a table lookup mechanism to replace floating-point operations includes:
[0020] Concatenate two floating-point numbers represented based on floating intervals into a query index;
[0021] Use the query index to find the corresponding operation results in the pre-calculated and cached result table.
[0022] The present invention also provides an operator optimization method for an intelligent model, characterized in that the operator calculation optimization is performed using the floating-point representation method for efficient training and reasoning of the intelligent model.
[0023] The present invention also provides a training and reasoning system for an intelligent model, wherein the training and reasoning system is used to implement the floating-point calculation method for training and reasoning of the intelligent model.
[0024] Compared with the prior art, the present invention has the following advantages and technical effects:
[0025] The present invention provides a floating point representation method for efficient training and reasoning of intelligent models, comprising: dividing a number axis into a plurality of floating intervals based on the numerical distribution characteristics of floating point numbers in the intelligent model; allocating a certain number of bits to each floating interval based on the floating point number density of each floating interval; allocating a unique interval identifier to each floating interval through Huffman coding based on the number of bits; the representation of each floating point number consists of the interval identifier and a specific position in the floating interval, so as to achieve optimization of intelligent model training and reasoning.
[0026] The present invention is based on the floating-point representation of the floating interval, which can change the range and precision by adjusting the interval size and the position coding bit width, and realize the rapid conversion between different precisions. Among them, the floating interval representation can realize the use of different representation densities at different positions on the number axis, and has better representation adaptability and flexibility. In addition, the floating interval representation supports the rapid conversion of precision, thereby supporting the decoupling of the calculation format and the storage format. Therefore, compared with the existing IEEE-754 floating-point format, the floating-point representation method for efficient training and reasoning of intelligent models of the present invention breaks through the limitations of the existing floating-point format, solves the problems of the existing floating-point format, such as high overhead of different precision conversion, tight coupling of storage and calculation, and uneven distribution of floating-point data on the number axis, further improves the efficiency of floating-point data representation in intelligent models, and provides new opportunities for mixed precision optimization of intelligent models. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0028] Figure 1 It is a probability distribution diagram of the weight and activation value of a certain network layer in ChatGLM-6B of an embodiment of the present invention;
[0029] Figure 2 This is a schematic diagram of interval division according to an embodiment of the present invention;
[0030] Figure 3 A schematic diagram of a floating point format representation of an embodiment of the present invention;
[0031] Figure 4 A schematic diagram of interval Huffman coding according to an embodiment of the present invention;
[0032] Figure 5 Schematic diagram of a table lookup mechanism for floating-point operations according to an embodiment of the present invention. DETAILED DESCRIPTION
[0033] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0034] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0035] Embodiment 1
[0036] This embodiment provides a floating point representation method for efficient training and reasoning of intelligent models, including:
[0037] Based on the numerical distribution characteristics of floating-point numbers in the intelligent model, the number axis is divided into multiple floating intervals;
[0038] Allocating a determined number of bits to each floating interval based on the floating point number density of each floating interval;
[0039] Based on the number of bits, assigning a unique interval identifier to each floating interval by Huffman coding;
[0040] The representation of each floating point number consists of the interval identifier and the specific position within the floating interval to achieve optimization of intelligent model training and reasoning.
[0041] Specifically, the number axis is divided and encoded according to the floating point number distribution characteristics in the intelligent model. For areas with a higher floating point number density, more bits are used to form a higher precision representation; for areas with a lower floating point number density, relatively fewer bits are used. Figure 2 As shown, the numerical distribution area corresponding to the weight matrix in the intelligent model is divided into five areas, and more bits are used to represent the intervals I1, I2 and I3 where the numerical distribution is more concentrated, while I4 and I5 can be represented by fewer bits.
[0042] In the representation method based on floating intervals, the representation of each floating point number consists of two parts: an interval identifier and a specific position within the interval. In order to maximize the savings of the bits occupied by the interval identifier, the present invention proposes to use Huffman coding to encode and identify different intervals. The higher the density of the interval, the shorter the interval identifier corresponding to the interval, and the more bits used to characterize the position encoding, that is, the higher the accuracy of the floating point representation, thereby achieving the purpose of making full use of limited bits.
[0043] Figure 3 Shown Figure 2 The corresponding floating point representation based on the floating interval, Figure 4 An example of interval Huffman coding is shown. Taking 8-bit floating point representation as an example, I1 only uses 1 bit to represent the interval, and the remaining 7 bits are all used to represent the position within the interval.
[0044] Furthermore, the numerical distribution feature is a multi-modal clustered distribution of tensor data.
[0045] Furthermore, the floating point representation based on the floating interval supports the representation of data distributed at any position on the number axis and provides higher representation accuracy.
[0046] Furthermore, the number of bits is dynamically adjusted according to the floating point number density of the floating interval.
[0047] Beneficial effects of this embodiment:
[0048] The present invention breaks through the limitations of existing floating-point formats and proposes a floating-point representation method for efficient training and reasoning of intelligent models. It solves the problems of high overhead of conversion between different precisions, tight coupling of storage and calculation, and uneven distribution of floating-point data on the number axis in existing floating-point formats. It is suitable for floating-point number representation with multi-peak clustered numerical characteristics in intelligent models, improves the efficiency of floating-point data representation in intelligent models, and provides new opportunities for mixed-precision optimization of intelligent models.
[0049] Embodiment 2
[0050] This embodiment also provides a floating-point calculation method for intelligent model training and reasoning, which uses a table lookup mechanism to replace floating-point operations based on the floating-point representation method for efficient intelligent model training and reasoning described in Example 1.
[0051] Furthermore, the process of using a table lookup mechanism to replace floating point operations includes:
[0052] Concatenate two floating-point numbers represented based on floating intervals into a query index;
[0053] Use the query index to find the corresponding operation result in the pre-calculated result table.
[0054] Specifically, taking the most typical matrix multiplication operator GEMM as an example, its calculation process is usually multiplication and addition of floating-point numbers. The results of summing or multiplication of two floating-point numbers represented based on floating intervals can be calculated in advance and retained in the memory. During the actual calculation, the calculation result can be obtained by looking up the table. Figure 5 The table lookup mechanism when two 8-bit floating-point data based on floating intervals (abbreviated as FR8) participate in the operation is demonstrated. Regardless of the bit width of the interval code and the position code in FR8, one FR8 can represent a maximum of 256 floating-point numbers. The table lookup mechanism proposed in the present invention first splices the two FR8s involved in the calculation into a 16-bit number as the query index of the result table. The result table of each operation only occupies 64KB of storage space. In the case of limited resources, it is possible to choose to implement table lookup for some areas with dense data distribution, while other intervals are converted to traditional methods for calculation.
[0055] Embodiment 3
[0056] This embodiment provides an operator optimization method for an intelligent model, which uses the floating-point representation method for efficient training and reasoning of intelligent models described in Embodiment 1 to optimize floating-point operations.
[0057] Embodiment 4
[0058] This embodiment provides a training and reasoning system for an intelligent model, and the training and reasoning system is used to implement the floating-point calculation method for training and reasoning of the intelligent model described in Embodiment 2.
[0059] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A floating point representation method for efficient training and reasoning of intelligent models, It is characterized in that The following steps are involved: Based on the numerical distribution characteristics of floating-point numbers in the intelligent model, the number axis is divided into multiple floating intervals; Allocating a determined number of bits to each floating interval based on the floating point number density of each floating interval; Based on the number of bits, assigning a unique interval identifier to each floating interval by Huffman coding; The representation of each floating point number consists of the interval identifier and the specific position within the floating interval to achieve optimization of intelligent model training and reasoning.
2. The floating point representation method for efficient training and reasoning of intelligent models according to claim 1, It is characterized in that The numerical distribution feature is a multi-modal clustered distribution of tensor data.
3. The floating point representation method for efficient training and reasoning of intelligent models according to claim 1, It is characterized in that Floating point representation based on floating interval supports the representation of data distributed anywhere on the number axis.
4. The floating point representation method for efficient training and reasoning of intelligent models according to claim 1, It is characterized in that The number of bits is dynamically adjusted according to the floating point number density of the floating interval.
5. A floating-point calculation method for intelligent model training and reasoning, It is characterized in that Based on the floating-point representation method for efficient training and reasoning of intelligent models described in any one of claims 1-4, a table lookup mechanism is used to replace floating-point operations.
6. The floating point calculation method for intelligent model training and reasoning according to claim 5, It is characterized in that The process of using table lookup mechanism to replace floating point operations includes: Concatenate two floating-point numbers represented based on floating intervals into a query index; Use the query index to find the corresponding operation results in the pre-calculated and cached result table.
7. An operator optimization method for an intelligent model, It is characterized in that The floating-point representation method for efficient training and reasoning of intelligent models described in any one of claims 1 to 4 is used to optimize the operation of operators.
8. A training and reasoning system for intelligent models, It is characterized in that The training and reasoning system is used to implement the floating-point computing method for training and reasoning of the intelligent model described in any one of claims 5-6.
