Operation method, processor and computing device

By using the weight vector integer in the AI ​​model as a floating point number with an exponent of 0 and the activation vector, the impact of quantitative compression on the accuracy of the AI ​​model is solved, and more efficient floating point number calculation and performance improvement are achieved.

CN119065630APending Publication Date: 2024-12-03HUAWEI TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202311092509.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-30
Filing Date
2023-08-28
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Quantitative compression will reduce model accuracy in AI models, especially because the numerical range of activation data is large and outliers exist, resulting in the impact of quantization on the accuracy of AI models than the weighted data.

Method used

By inputting the integer element in the weight vector as a floating point number with an exponent of 0, it is used to perform calculations with the floating point activation vector, avoiding the exponent addition processing in floating point multiplication, reducing the power consumption of the data operation unit, and Multiplexing exponential alignment module improves efficiency.

Benefits of technology

In the process of quantization and compression of AI models, only the weighted data is quantified, the impact on model accuracy is reduced, the efficiency of floating-point number calculations is improved, and the performance of data computing units is improved through lower hardware costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065630A_ABST
    Figure CN119065630A_ABST
Patent Text Reader

Abstract

The embodiment of the invention belongs to the technical field of computing, and particularly relates to an operation method, a processor and computing equipment. The method is executed by a processor comprising a data carrying unit and a data operation unit, and comprises the following steps: the data carrying unit obtains an activation vector in an AI model and a quantized weight vector, the weight vector is an integer number, and the activation vector is a floating-point number; and the data carrying unit inputs the sign bit and the mantissa bit of each element in the weight vector as the sign bit and the mantissa bit of the floating-point number with the index of 0 into the data operation unit, and inputs each element in the activation vector into the data operation unit. And the data operation unit performs vector operation on the input weight vector and the activation vector to obtain an operation result. By adopting the method and the device, the operation of the integer number and the floating-point number can be realized in the data operation unit, so that when the quantitative compression is performed on the AI model, only the weight data can be compressed, and the influence of the quantitative compression on the precision of the AI model can be reduced.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application claims priority to Chinese patent application No. 202310633793.X, filed on May 30, 2023, with invention name “A Data Calculation Method”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computing technology, and in particular to a computing method, a processor, and a computing device. Background Art

[0003] Quantization compression of artificial intelligence (AI) models refers to a method of quantizing the weight data and activation data in the AI ​​model from floating point numbers to integers, thereby improving the running speed of the AI ​​model.

[0004] Although quantization compression can improve the running speed of AI models, it will also reduce the accuracy of AI models. Among them, because the value range of activation data is larger and there is a certain proportion of outliers, the impact of activation data quantization on AI model accuracy is greater than that of weight data quantization on AI model accuracy. Summary of the invention

[0005] The embodiments of the present application provide a computing method, a processor, and a computing device, which can implement integer and floating-point computing in a data computing unit, and can reduce the decrease in model accuracy when quantizing a model. The corresponding technical solutions are as follows:

[0006] In a first aspect, a computing method is provided. The computing method may be executed by a processor including a data handling unit and a data computing unit. The computing method executed by the processor includes:

[0007] The data handling unit obtains the quantized weight vector in the artificial intelligence AI model and the activation vector used for vector operations with the weight vector, wherein the elements in the weight vector are integers and the elements in the activation vector are floating-point numbers. The data handling unit inputs the sign bit and mantissa of each element in the weight vector as the sign bit and mantissa of a floating-point number with an exponent of 0 into the data operation unit, and inputs each element in the activation vector into the data operation unit. The data operation unit performs vector operations on the input weight vector and activation vector to obtain the operation results.

[0008] In the scheme shown in the present application, for the elements of the weight vector that belong to integers, the element can be used as a floating-point input value data operation unit with an exponent of 0 to operate with the activation vector of the floating-point data. Among them, the sign bit of the element of the integer is the sign bit of the floating-point number with an exponent of 0, and the mantissa of the element of the integer is the mantissa of the floating-point number with an exponent of 0. In addition, since the exponent bit in the floating-point number with an exponent of 0 is 0, it can not participate in the operation of the floating-point number. It can be seen that the scheme shown in the present application can realize the floating-point operation of the weight vector of the quantized integer and the activation vector of the floating-point number. In this way, on the one hand, when the AI ​​model is quantized and compressed, only the weight data can be quantized, so that the impact of quantization compression on the accuracy of the AI ​​model can be reduced. On the other hand, in the process of performing floating-point operations on integers and floating-point numbers, there is no need to convert the integer to a floating-point number, thereby improving the efficiency of floating-point operations.

[0009] In one implementable manner, the data operation unit includes a floating-point multiplication module and a floating-point addition module, the vector operation is a vector inner product, and the data operation unit performs vector operation on the input weight vector and activation vector to obtain an operation result, including: the floating-point multiplication module performs mantissa multiplication on the mantissa of each element (weight data) in the weight vector and the mantissa of the corresponding element (activation data) in the activation vector to obtain the mantissa multiplication result. The floating-point addition module performs a floating-point addition operation on the exponent bit of each element in the activation vector and the mantissa multiplication result to obtain the operation result.

[0010] In the scheme shown in the present application, since the exponent bit of the floating-point number with an exponent of 0 is 0, the exponent bit in the activation data is the sum of the exponent bits corresponding to the activation data and the weight data. Therefore, the floating-point multiplication module can only perform mantissa multiplication on the mantissa bits corresponding to the activation data and the weight data. In this way, the exponential addition process in the floating-point multiplication operation can be avoided, thereby reducing the power consumption of the data operation unit. When performing a floating-point addition operation, the floating-point addition module can directly obtain the exponent bit of the activation data as the sum of the exponent bits corresponding to the activation data and the weight data, and then perform a floating-point addition operation in combination with the mantissa multiplication result to obtain the operation result.

[0011] In an achievable manner, the data operation unit further includes an exponent alignment module, and the floating-point addition module includes a mantissa accumulation submodule and a normalization submodule. The above-mentioned operation method also includes: the exponent alignment module performs exponent alignment on the exponent bit of each element in the activation vector to obtain an exponent alignment result. The floating-point addition module performs floating-point addition operation on the exponent bit of each element in the activation vector and the mantissa multiplication result to obtain an operation result, including: the mantissa accumulation submodule performs accumulation processing on the mantissa multiplication result to obtain a mantissa addition result. The normalization submodule performs normalization processing on the exponent alignment result and the mantissa addition result to obtain an operation result.

[0012] Among them, the floating-point addition operation includes exponent alignment, mantissa accumulation, and normalization of the exponent after exponent alignment and the mantissa after mantissa accumulation. In the scheme shown in the present application, since the exponent corresponding to the weight data is 0, when performing exponent alignment, the exponent in the activation data can be only aligned through the exponent alignment module. And because the same activation vector may perform vector operations with multiple weight vectors during the operation of the AI ​​model, in the present application, the result of exponential alignment of the activation data by the exponent alignment module can be reused, thereby improving the efficiency of vector operations.

[0013] In one implementable manner, the floating-point addition module includes a plurality of mantissa accumulation submodules and a plurality of normalization submodules.

[0014] In the scheme shown in the present application, since the exponent alignment result of the exponent alignment module can be reused, if the performance of the data operation unit needs to be doubled, it is only necessary to increase the number of the mantissa accumulation submodule and the normalization submodule. Therefore, the operation method provided by the present application can improve the performance of the data operation unit at a lower hardware cost.

[0015] In one implementable manner, the data operation unit includes a floating-point multiplication module and a floating-point addition module, and the vector operation is a vector outer product. The data operation unit performs vector operations on the input weight vector and activation vector to obtain an operation result, including: the data handling unit inputs the exponent bit and mantissa bit of each element in the accumulation matrix obtained in the vector outer product process into the floating-point addition module. The floating-point multiplication module performs mantissa multiplication on the mantissa bit of each element in the weight vector with the mantissa bit of each element in the activation vector to obtain the mantissa multiplication result. The floating-point addition module inputs the exponent bit of each element in the activation vector, the mantissa multiplication result, and the exponent bit and mantissa bit of each element in the accumulation matrix into the floating-point addition module to obtain the operation result.

[0016] In the scheme shown in the present application, when the data operation unit performs the vector outer product on the weight vector and the activation vector, since the weight data is input into the data operation unit as a floating point number with an exponent of 0, the exponent bit in the activation data is the sum of the exponent bits corresponding to the activation data and the weight data. Therefore, the floating-point multiplication module can only perform mantissa multiplication on the mantissa bits corresponding to the activation data and the weight data. In this way, the exponential addition process in the floating-point multiplication operation can be avoided, thereby reducing the power consumption of the data operation unit. When the floating-point addition module performs a floating-point addition operation, the exponent bits of the activation data and the corresponding elements in the accumulation matrix can be directly aligned, and the mantissa bits of the mantissa multiplication result and the corresponding elements in the accumulation matrix can be accumulated to obtain the operation result.

[0017] In one implementable manner, the data handling unit inputs the sign bit and mantissa bit of each element in the weight vector as the sign bit and mantissa bit of a floating-point number with an exponent of 0 to the data calculation unit, including: the data handling unit inputs the mantissa bit of each element in the weight vector as the sign bit and mantissa bit of a floating-point number with an exponent of 0, and inputs 0 as the exponent bit of the floating-point number with an exponent of 0 to the data calculation unit.

[0018] In the scheme shown in the present application, the data handling unit can input the sign bit of each element in the weight vector as the sign bit of a floating point number with a value of 0, the mantissa bit of each element in the weight vector as the mantissa bit of a floating point number with a value of 0, and 0 as the exponent bit of a floating point number with an exponent of 0 into the data operation unit, thereby enabling the data operation unit to implement floating-point operations on the weight data of integers and the activation data of floating-point numbers. In this way, on the one hand, when quantizing and compressing the AI ​​model, only the weight data can be quantized, which can reduce the impact of quantization compression on the accuracy of the AI ​​model. On the other hand, in the process of performing floating-point operations on integers and floating-point numbers, there is no need to convert the integers into floating-point numbers, thereby improving the efficiency of floating-point operations.

[0019] In one implementable manner, the data handling unit inputs the sign bit and mantissa bits of each element in the weight vector as the sign bit and mantissa bits of a floating-point number with an exponent of 0 to the data calculation unit, including: the data handling unit inputs the sign bit and mantissa bits of each element in the weight vector as the sign bit and mantissa bits of a floating-point number with an exponent of 0 and eight bits to the data calculation unit.

[0020] In the scheme shown in the present application, the data handling unit can input each element in the weight vector as an eight-bit floating point number with an exponent of 0 to the data operation unit for floating point operation with the activation data of the floating point number. The first bit of the eight-bit floating point number with an exponent of 0 is the sign bit of the floating point number, and the last seven bits are the mantissa bits of the floating point number. This form of floating point number can be called FP8.

[0021] In one achievable manner, when the elements in the weight vector are less than eight bits, the method further includes: padding 0 between the sign bit and the mantissa bit of the elements in the weight vector; or padding 0 after the mantissa bit of the elements in the weight vector.

[0022] In the scheme shown in the present application, the elements in the weight vector are less than eight bits, indicating that the elements in the weight vector are IN4 or INT6. Therefore, the mantissa bits of the elements in the weight vector can be padded with 0s first, the elements in the weight vector can be padded to eight bits and then input into the data operation unit as FP8 for calculation.

[0023] In one achievable manner, the data operation unit performs vector operation on the input weight vector and activation vector to obtain an operation result, including: the data operation unit performs vector operation on the input weight vector and obtains an intermediate result. The data operation unit performs floating point multiplication operation on the intermediate result and the quantization coefficient to obtain the operation result, and the quantization coefficient is used to reduce the numerical error generated after the elements in the weight vector are used as floating point numbers.

[0024] In the scheme shown in the present application, the mantissa of the integer is directly input into the data operation unit as the mantissa of the floating point number with an exponent of 0 for operation, and the obtained operation result will have errors. Therefore, after the data operation unit performs vector operations on the input weight vector and activation vector, the obtained intermediate result can be multiplied by the quantization coefficient for eliminating the error to obtain the operation result after the error is eliminated. This can improve the accuracy of the data operation unit's operation on the weight data and activation data.

[0025] In one achievable manner, before the data handling unit obtains the quantized weight vector and the activation vector used for performing vector operations with the weight vector, the method further includes: the data operation unit receiving an asymmetric operation instruction.

[0026] In the scheme shown in the present application, before the data operation unit executes the above-mentioned operation method, an asymmetric operation instruction can be sent to the data operation unit to instruct the data operation unit to execute the above-mentioned operation method, thereby realizing floating-point operations corresponding to the weight data of integer numbers and the activation data of floating-point numbers.

[0027] In a second aspect, a processor is provided, the processor comprising a data handling unit and a data operation unit, wherein:

[0028] A data handling unit, used to obtain a quantized weight vector and an activation vector used to perform vector operations with the weight vector, wherein the elements in the weight vector are integers and the elements in the activation vector are floating point numbers;

[0029] A data handling unit, used to input the sign bit and the mantissa bit of each element in the weight vector as the sign bit and the mantissa bit of the floating point number with an exponent of 0 into the data operation unit, and input each element in the activation vector into the data operation unit;

[0030] The data operation unit is used to perform vector operations on the input weight vector and activation vector to obtain operation results.

[0031] In one implementation, the data operation unit includes a floating-point multiplication module and a floating-point addition module, and the vector operation is a vector inner product. The floating-point multiplication module is used to perform a mantissa multiplication on the mantissa of each element in the weight vector and the mantissa of the corresponding element in the activation vector to obtain a mantissa multiplication result. The floating-point addition module is used to perform a floating-point addition operation on the exponent bit of each element in the activation vector and the mantissa multiplication result to obtain an operation result.

[0032] In one implementation, the data operation unit further includes an exponent alignment module, and the floating-point addition module includes a mantissa accumulation submodule and a normalization submodule. The exponent alignment module is used to perform exponent alignment on the exponent bit of each element in the activation vector to obtain an exponent alignment result. The mantissa accumulation submodule is used to perform accumulation processing on the mantissa multiplication result to obtain a mantissa addition result. The normalization submodule is used to perform normalization processing on the exponent alignment result and the mantissa addition result to obtain an operation result.

[0033] In one implementation, the floating-point addition module includes a plurality of mantissa accumulation submodules and a plurality of normalization submodules.

[0034] In one implementation, the data operation unit includes a floating-point multiplication module and a floating-point addition module, and the vector operation is a vector outer product. The data handling unit is used to input the exponent bit and mantissa bit of each element in the accumulation matrix obtained in the vector outer product process into the floating-point addition module. The floating-point multiplication module is used to perform mantissa multiplication on the mantissa bit of each element in the weight vector with the mantissa bit of each element in the activation vector to obtain the mantissa multiplication result. The floating-point addition module is used to input the exponent bit of each element in the activation vector, the mantissa multiplication result, and the exponent bit and mantissa bit of each element in the accumulation matrix into the floating-point addition module to obtain the operation result.

[0035] In one implementation, the data handling unit is used to input the mantissa bits of each element in the weight vector as the sign bit and mantissa bits of a floating point number with an exponent of 0, and input 0 as the exponent bit of the floating point number with an exponent of 0 to the data operation unit.

[0036] In one implementation, the data handling unit is used to input the sign bit and mantissa bit of each element in the weight vector into the data operation unit as the sign bit and mantissa bit of a floating point number with an exponent of 0 and eight bits.

[0037] In one implementation, when the elements in the weight vector are less than eight bits, the data handling unit is further used to fill 0 between the sign bit and the mantissa bit of the elements in the weight vector; or, fill 0 after the mantissa bit of the elements in the weight vector.

[0038] In one implementation, the data operation unit is used to perform vector operations on the input weight vector and obtain an intermediate result; perform floating-point multiplication operations on the intermediate result and the quantization coefficient to obtain an operation result, and the quantization coefficient is used to reduce the numerical error generated after the elements in the weight vector are treated as floating-point numbers.

[0039] In one implementation, the data operation unit is further configured to receive an asymmetric operation instruction.

[0040] In a third aspect, a computing device is provided, which includes a memory and a processor as described in the second aspect, and the processor can be used to implement the calculation method as described in the first aspect.

[0041] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program code. When the computer program code is executed by a computing device, the computing device executes the operation method described in the first aspect.

[0042] In a fifth aspect, a computer program product comprising instructions is provided. When the computer program product is run on a computing device, the computing device is enabled to execute the operation method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0044] Figure 2 is a schematic diagram of a processor structure provided in an embodiment of the present application;

[0045] Figure 3 is a flow chart of a calculation method provided in an embodiment of the present application;

[0046] Figure 4 It is a structural diagram of an integer and a floating point number provided in an embodiment of the present application;

[0047] Figure 5 is a schematic diagram of a vector inner product operation provided in an embodiment of the present application;

[0048] Figure 6 is a schematic diagram of a data operation unit performing an operation method provided in an embodiment of the present application;

[0049] Figure 7 is a schematic diagram of a data operation unit performing an operation method provided in an embodiment of the present application;

[0050] Figure 8 is a schematic diagram of a data operation unit performing an operation method provided in an embodiment of the present application;

[0051] Fig. 9 is a schematic diagram of a vector outer product operation provided in an embodiment of the present application;

[0052] Fig.10 It is a schematic diagram of a data operation unit performing an operation method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0054] At the beginning of AI model design, in order to improve the accuracy of AI models, weight data and activation data will be set to higher-precision floating-point numbers, such as FP32 and FP16. Floating-point numbers occupy more bits, such as FP32 floating-point numbers occupy 32 bits, and FP16 floating-point numbers occupy 16 bits. Therefore, floating-point numbers require more storage and computing resources when stored and calculated. Especially in some large models, such as large language models (LLM), the number of parameters is huge, and the storage of weight data alone requires hundreds of GB of storage space.

[0055] in:

[0056] The AI ​​model can be any trained artificial intelligence model, such as a large language model, an image recognition model, etc.

[0057] Weight data includes the weight parameters of the AI ​​model after training. In the actual operation of the AI ​​model, weight data is generally expressed in the form of vectors or matrices.

[0058] Activation data can be the output data of the middle layer of the AI ​​model, which is related to the sample data input into the AI ​​model. In the actual operation of the AI ​​model, the activation data can also be expressed in the form of vectors or matrices, and often performs vector or matrix operations with weight data.

[0059] In order to improve the universality of AI models, quantization compression will be used to reduce the storage space required to store AI models and improve the running speed of AI models. Quantization compression refers to a method of quantizing the weight data and activation data in the AI ​​model from floating point numbers to integers, thereby improving the running speed of the AI ​​model. For example, converting FP16 weight data and activation data into INT8 integers. In this way, the storage of weight data is reduced from 16 bits to 8 bits.

[0060] However, when converting floating-point numbers to integers, the precision of the values ​​will decrease, and there will be certain errors in the conversion process. The larger the range of floating-point numbers, the greater the error in converting floating-point numbers to integers. Therefore, quantization compression will have a greater impact on the accuracy of AI models.

[0061] Since in the AI ​​model, the numerical range of the activation data is generally larger than the numerical range of the weight data, and there is a certain proportion of outliers in the activation data, the quantization of the activation data causes a larger error, and the impact on the accuracy of the AI ​​model is greater than the quantization of the weight data. In addition, the applicant found that in some AI models, especially in large generative models, the amount of weight data is much larger than the amount of activation data. Therefore, quantizing the weight data can bring better quantization effects to the AI ​​model, and the impact on the accuracy of the AI ​​model is less than the quantization of the activation data.

[0062] Therefore, the applicant discovered that if the impact of quantization compression on the accuracy of the AI ​​model is reduced, in theory only the weight data can be quantized. However, since the current hardware cannot support operations on floating-point numbers and integers, that is, it cannot support operations on unquantized activation data and quantized weight data, this theory cannot be directly implemented using existing technology.

[0063] The embodiment of the present application provides a calculation method that can realize the calculation of floating-point numbers and integer numbers on the existing data calculation unit. In this way, when the AI ​​model is quantized and compressed, the weight data in the AI ​​model can be quantized to integers offline. In the actual operation of the AI ​​model, the activation data may not be quantized, but the calculation method provided by the present application may be used to realize the calculation of floating-point activation data and integer weight data. In this way, by quantizing the weight data, the storage space occupied by the AI ​​model can be reduced and the operating efficiency of the AI ​​model can be improved. Because the activation data is not quantized, the impact of quantization compression on the accuracy of the AI ​​model is reduced.

[0064] An embodiment of the present application provides a calculation method that can be implemented by a computing device. Figure 1 Schematic diagram of a computing device provided in an embodiment of the present application. Figure 1As shown, the computing device 100 includes: a bus 102, a processor 104, a memory 106, and the optional computing device 100 may also include a communication interface 108. The processor 104, the memory 106 and the communication interface 108 communicate through the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 100. The computing device 100 can be a device for running an AI model, a terminal or a server. When the computing device 100 is a terminal, the computing device 100 includes but is not limited to a desktop computer, a mobile phone, a notebook, a tablet computer, etc. When the computing device 100 is a server, the computing device 100 can be a separate server, or a server cluster composed of multiple servers, can be a physical machine, or can be a virtual machine, a container, etc. virtualized by virtual technology.

[0065] The bus 102 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 1 The bus 102 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 102 may include a path for transmitting information between various components of the computing device 100 (eg, the memory 106, the processor 104, and the communication interface 108).

[0066] The processor 104 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP) or a digital signal processor (DSP). Among them, the processor 104 may further include a data handling unit and a data operation unit. The data handling unit is used to implement the data handling function of the processor, such as direct memory access (DMA). The data operation unit is used to implement the operation of floating-point activation data and integer weight data, such as the data operation unit can be a matrix operation unit, such as a matrix acceleration engine (cube in NPU, tensor core of GPU), etc.

[0067] The memory 106 may include a volatile memory, such as a random access memory (RAM). The memory 106 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0068] The communication interface 108 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 100 and other devices or a communication network.

[0069] Figure 2 is a schematic diagram of a processor structure provided in an embodiment of the present application, and the processor may be the above-mentioned Figure 2 The processor 104 in the computing device, such as Figure 2 As shown, the processor includes a data handling unit and a data computing unit. The processor can execute the computing method provided in the embodiment of the present application. Figure 3 is a flowchart of a calculation method provided in an embodiment of the present application, see Figure 3 , the operation method executed by the processor includes:

[0070] Step 301: The data handling unit obtains a quantized weight vector in the artificial intelligence AI model and an activation vector used to perform vector operations with the weight vector, wherein the elements in the weight vector are integers and the elements in the activation vector are floating-point numbers.

[0071] During the operation of the AI ​​model, weight data and activation data are generally represented in the form of matrices or vectors. During the operation, even if the weight data and activation data are represented as matrices, they are first converted into vectors for operation. In this application, weight vectors and activation vectors are vectors composed of weight data and activation data to be operated. In one example, the data handling unit can read weight vectors and activation vectors from memory.

[0072] Among them, each element in the weight vector can be quantized weight data. The quantization method of the weight data can adopt uniform quantization or non-uniform quantization, and its specific quantization method will not be introduced in detail in this application. In one example, the weight data can be quantized from FP16 floating point numbers to INT8 integers by uniform quantization. The elements in the activation vector can be the activation data output by the middle layer of the model during the operation of the model. The activation data is a floating point number. In one example, the activation data does not need to be quantized.

[0073] Step 302: The data handling unit inputs the sign bit and mantissa bit of each element in the weight vector as the sign bit and mantissa bit of a floating point number with an exponent of 0 into the data operation unit, and inputs each element in the activation vector into the data operation unit.

[0074] like Figure 4 As shown, an integer number includes a sign bit and a mantissa, where the sign bit is the first bit in the integer number, and the mantissa is the other bits in the integer number except the first bit. For example, in an INT8 integer number, the first bit is the sign bit, and the next seven bits are the mantissa. A floating-point number includes a sign bit, an exponent bit, and a mantissa, where the sign bit is the first bit in the integer number, and the exponent bit is between the sign bit and the mantissa. For example, in an FP16 floating-point number, the first bit is the sign bit, the next ten bits are the mantissa, and the remaining five bits between the sign bit and the mantissa are the exponent bits.

[0075] Since floating point numbers can be expressed as f=s*(t+1)*2 i , where f represents a floating point number, s is the value corresponding to the sign bit (±), t+1 is the value corresponding to the mantissa bit, and i is the value corresponding to the exponent bit. When the exponent in the floating point number f is 0, 2 i =1, therefore, the floating point number can be expressed as f=s*(t+1)*1, i.e., f=s*(t+1), that is, the floating point number with exponent 0 can be represented by the mantissa and the sign bit. For integers, it can be expressed as I=s*m, where I represents an integer, s is the value corresponding to the sign bit, and m is the mantissa. It can be seen that when the exponent of the floating point number is 0, the floating point number and the integer number can have the same representation. Therefore, in the present application, the weight data of the integer number can be used as a floating point number with exponent 0 to perform floating point operations with the activation data.

[0076] Treat each element in the weight vector as a floating point number with an exponent of 0, that is, treat the integer as a floating point number with an exponent of 0. Figure 4 As shown, the sign bit of the integer is the sign bit of the floating point number with an exponent of 0, and the mantissa bit of the integer is the mantissa bit of the floating point number with an exponent of 0. Since the exponent of the floating point number is 0, that is, the exponent bit is 0, the exponent bit of the floating point number may not be set temporarily.

[0077] In one achievable manner, when an integer is required to be input into a data operation unit as a floating point number with an exponent of 0, the data handling unit can directly input the mantissa of the integer into the data operation unit as the mantissa of the floating point number with an exponent of 0. The sign bit of the integer is input into the data operation unit as the sign bit of the floating point number with an exponent of 0. Although there is no exponent bit in an integer, since the exponent of the floating point number is 0, in one example, the exponent bit corresponding to the floating point number with an exponent of 0 can be input into the data operation unit by padding with 0.

[0078] Step 303: The data operation unit performs vector operation on the input weight vector and activation vector to obtain an operation result.

[0079] The data handling unit inputs each element in the weight vector as a floating point number with an exponent of 0 and each element in the activation vector to the data operation unit to perform vector operations on the input weight vector and activation vector to obtain the operation result. The further operation process will not be described in detail here.

[0080] Since the mantissa bits in the floating point number and the mantissa bits in the integer number correspond to the same binary string, but the values ​​represented are different, for example, the mantissa bits in the floating point number and the mantissa bits in the integer number are both "0101", and the mantissa bits in the integer number are "0101" can represent the decimal 5, but the mantissa bits in the floating point number are "0101" can represent a certain error between the numerical value and the decimal 5, and the error is a linear error, which can be compensated by the quantization coefficient (scale). Therefore, in step 303, after the data operation unit performs vector operation on the input weight vector and activation vector, error compensation processing can be performed. The corresponding processing is as follows: the data operation unit performs vector operation on the input weight vector and obtains an intermediate result, and performs floating point multiplication operation on the intermediate result and the quantization coefficient to obtain the operation result.

[0081] The quantization coefficient is a floating point number, which is used to reduce the numerical error caused by the elements in the weight vector as floating point numbers. For the quantization coefficient, scale = a*scale1. Scale1 is the quantization parameter obtained by quantizing the original floating point weight data in the AI ​​model to the integer weight data, and a is the minimum value represented by the floating point number with an exponent of 0.

[0082] In one example, in step 303, after the data operation unit performs vector operation on the input weight vector and activation vector, the result obtained can be called an intermediate result. After obtaining the intermediate result, the intermediate result floating point number can be multiplied by a floating point number with a quantization coefficient to obtain the final operation result of the vector operation of the weight vector and the activation vector. This can improve the accuracy of the data operation unit's operation on the weight data and activation data. Among them, the floating point multiplication of the intermediate result floating point number and the quantization coefficient can be implemented by a hardware circuit in the data operation unit.

[0083] In addition, in an embodiment of the present application, the weight data is quantized to INT8. In this application, an integer with an exponent of 0 as INT8 may be referred to as FP8. That is, FP8 is a floating point number including a sign bit and seven mantissa bits. When the elements in the weight vector are less than eight bits, it means that the elements in the weight vector are IN4 or INT6, so the mantissa bits of the elements in the weight vector can be padded with 0 first, and then input into the data operation unit as FP8 for operation.

[0084] In the embodiments provided by the present application, the weight data of an integer can be input into the data operation unit as a floating point number with an exponent of 0, and during the input process, the weight data of the integer does not need to be converted, so that the operation of integers and floating point numbers can be realized without increasing the conversion steps between integers and floating point numbers. In this way, using the operation method provided by the present application, when quantizing and compressing the AI ​​model, only the weight data can be quantized, and the activation data is not quantized. In this way, the operation speed of the AI ​​model can be improved through quantization compression, and the impact of quantization compression on the accuracy of the AI ​​model can be reduced.

[0085] The following is a further introduction to the calculation method of the integer weight vector and the floating point activation vector provided by the present application in conjunction with the accompanying drawings:

[0086] In one achievable manner, the weight vector and the activation vector are a row or a column of data included in the weight matrix and the activation matrix. A matrix inner product operation needs to be performed between the weight matrix and the activation matrix, and the matrix inner product operation is implemented by the inner product of the vectors.

[0087] Correspondingly, the data operation unit may include a floating-point multiplication module and a floating-point addition module. In this case, the processing of the above steps 302 and 303 can be replaced by: the data handling unit inputs the mantissa bits of each element in the weight vector and the mantissa bits of the corresponding elements in the activation vector into the floating-point multiplication module, and inputs the exponent bits of each element in the activation vector and the mantissa multiplication result into the floating-point addition module. The floating-point multiplication module performs mantissa multiplication on the mantissa bits of each element in the weight vector and the mantissa bits of the corresponding elements in the activation vector to obtain the mantissa multiplication result; performs floating-point addition operation on the exponent bits of each element in the activation vector and the mantissa multiplication result to obtain the operation result.

[0088] like Figure 5 As shown, the inner product of the weight vector and the activation vector includes multiplying the corresponding elements in the weight vector and the activation vector, and then accumulating the multiplication results of the corresponding elements to obtain the inner product result of the weight vector and the activation vector. For example, the activation vector is [a, b, c, d], and the weight vector is [e, f, g, h], where a*e=i, b*f=j, c*g=k, d*e=l, i+j=m, k+l=n, and a~n are all floating point numbers. Then the inner product of the activation vector and the weight vector can be expressed as: F=a*e+b*f+c*g+d*e. It can be seen that the inner product operation of the weight vector and the activation vector includes floating point multiplication and floating point addition.

[0089] Figure 6 Schematic diagram of a data operation unit performing an operation method provided in an embodiment of the present application. Figure 6 As shown, the data operation unit may include a floating-point multiplication module and a floating-point addition module, and the multiplication of the weight vector and the activation vector for the corresponding elements may be implemented by the floating-point multiplication module. The multiplication of floating-point numbers includes the addition of exponent bits and the multiplication of mantissa bits, and the addition of exponent bits may be implemented by the exponent addition submodule in the floating-point multiplication module, and the multiplication of mantissa bits may be implemented by the mantissa multiplication submodule in the floating-point multiplication module.

[0090] In implementation, the data handling unit may input the mantissa bits of the corresponding elements in the activation vector and the weight vector into the mantissa multiplication submodule in the floating-point multiplication module for mantissa multiplication. That is, the mantissa bits of the floating-point elements in the activation vector and the mantissa bits of the integer elements in the weight vector are input into the mantissa multiplication submodule for mantissa multiplication to obtain the mantissa multiplication result. If the number of bits of the mantissa bits of the integer elements is less than the number of bits required to be input into the mantissa multiplication submodule, the mantissa bits of the integer elements may be padded with 0.

[0091] In one example, since the exponent in the weight vector is 0, that is, the value of the exponent bit is 0, the value of the exponent bit of the floating-point element in the activation vector and the exponent bit in the weight vector are added, and the value of the exponent bit of the floating-point element in the activation vector is still equal to the value of the exponent bit of the floating-point element in the activation vector. Therefore, in an embodiment of the present application, a bypass function can be set to omit the processing of the exponent addition submodule in the floating-point multiplication module, that is, a direct path can be set between the exponent alignment submodule in the data handling unit and the floating-point addition unit, and the data handling unit can directly input the exponent bit in the activation data into the exponent alignment submodule. Among them, the exponent bit in the activation data is equal to the exponential addition result of the exponent bit in the activation data and the exponent bit in the weight data. For the bypass function, the data operation unit can provide a configuration interface (configuration switch) for controlling the opening of the bypass function. In one example, an asymmetric calculation instruction can be sent to the data operation unit, and then the opening of the bypass function can be realized through the configuration interface.

[0092] Continue to see Figure 6 , the addition of floating-point multiplication results can be implemented by a floating-point addition module. Among them, the addition of floating-point numbers includes exponent alignment of the exponent bits of the activation data and the weight data, mantissa accumulation of the mantissa bits of the activation data and the weight data, and normalization of the exponent alignment results and mantissa accumulation. The exponent alignment can be implemented by the exponent alignment submodule in the floating-point addition module, the mantissa accumulation can be implemented by the mantissa accumulation submodule in the floating-point addition module, and the normalization processing is implemented by the normalization submodule in the floating-point addition module.

[0093] After the exponential addition result and the mantissa multiplication result of the activation data and the weight data are obtained by the floating-point multiplication module, the exponential addition results obtained can be input into the exponent alignment submodule in pairs for exponent alignment to obtain the exponent alignment result, and the corresponding mantissa multiplication results can be input into the mantissa multiplication submodule in pairs for mantissa multiplication to obtain the mantissa multiplication result, and then the exponent alignment result and the mantissa multiplication result are input into the normalization submodule, and the normalization submodule performs normalization processing, thereby obtaining the operation result after normalization processing.

[0094] For the above bypass function, the exponent addition result input to the exponent alignment submodule can be the exponent bit of the activation data input through the bypass function. In addition, the calculation of the activation data and the sign bit of the weight data is the same as that of the prior art, and will not be repeated in the embodiment of the present application.

[0095] In the embodiment of the present application, on the one hand, floating-point operations of integer weight data and floating-point activation data can be realized, which can reduce the impact of quantization compression on the accuracy of the AI ​​model. On the other hand, in the process of realizing the inner product operation of the weight vector and the activation vector, the processing of the exponential addition submodule in the floating-point multiplication module can be omitted through the bypass function, which can reduce the power consumption of the data operation unit.

[0096] Figure 7 It is a schematic diagram of a data operation unit performing an operation method provided in an embodiment of the present application. Figure 7 The method shown can be used in Figure 6 Based on the method shown, exponent alignment is achieved through an exponent alignment module included in the data operation unit.

[0097] During the operation of the model, the same activation vector needs to be operated with multiple weight vectors, and the elements in the weight vector will be treated as floating point numbers with an exponent of 0. The results of the index alignment of the activation data in the same activation vector and the weight data in multiple weight vectors are all the results of the index alignment of the activation data in the same activation vector. In this way, it is only necessary to perform index alignment on the exponent bits of the elements in the activation vector once, and the index alignment result can be reused multiple times. Therefore, in the embodiment of the present application, a bypass function can be designed, and the index alignment module ( Figure 7 (not directly shown) implements the exponent alignment processing of the exponent bits of the elements in the activation vector, and the exponent alignment module may not be subordinate to the matrix addition module.

[0098] In one example, when the exponent bits of the elements in the activation vector are aligned for the first time, the alignment can be done through the exponent alignment module. After the exponent alignment result is obtained, the exponent alignment result can be stored, and when the exponent bits of the elements in the activation vector need to be aligned again, the exponent alignment result can be directly obtained. In this way, the number of times the exponent alignment unit in the floating-point addition module performs the exponent alignment process can be reduced, thereby reducing the computing power consumption.

[0099] because Figure 6 , Figure 7 The method shown omits the exponential addition processing of the exponential addition submodule and the exponential alignment processing repeatedly performed by the exponent alignment submodule, thereby reducing the power consumption of the data operation unit.

[0100] In the solution shown in the present application, since the data operation unit can omit the exponential addition processing of the exponential addition submodule when performing the vector inner product, the exponential alignment result of the exponential alignment module can be reused. Figure 8 As shown, if the performance of the data operation unit needs to be doubled, it is only necessary to increase the number of mantissa accumulation submodules and normalization submodules, without increasing the number of exponent addition submodules and exponent alignment submodules in the data operation unit. Therefore, the operation method provided by the present application can improve the performance of the data operation unit at a lower hardware cost.

[0101] In one achievable manner, the weight vector and the activation vector are a row or a column of data included in the weight matrix and the activation matrix. A matrix outer product is required between the weight matrix and the activation matrix, and the matrix multiplication is implemented by the outer product of the vectors.

[0102] Correspondingly, the data operation unit may include a floating-point multiplication module and a floating-point addition module. In this case, the processing of the above steps 302 and 303 can be replaced by: the data handling unit inputs the exponent bit and mantissa bit of each element in the accumulation matrix obtained in the vector outer product process to the floating-point addition module. The floating-point multiplication module performs mantissa multiplication on the mantissa bit of each element in the weight vector with the mantissa bit of each element in the activation vector to obtain the mantissa multiplication result. The floating-point addition module inputs the exponent bit of each element in the activation vector, the mantissa multiplication result, and the exponent bit and mantissa bit of each element in the accumulation matrix to the floating-point addition module to obtain the operation result.

[0103] The outer product operation of the weight matrix and the activation matrix includes multiplying each element of the weight vector with each element of the activation vector to obtain an intermediate matrix, and then adding the intermediate matrix to the accumulation matrix obtained during the outer product operation to obtain the updated accumulation matrix. The accumulation matrix is ​​initially a 0 matrix. Fig. 9 As shown, each i-th element of the weight vector is multiplied by the j-th element in the activation vector to obtain the element in the i-th row and j-th column of the intermediate matrix, and then the element in the i-th row and j-th column of the intermediate matrix can be added to the element in the i-th row and j-th column of the accumulation matrix. It can be seen that the outer product operation between the weight matrix and the activation matrix also requires floating-point multiplication and addition.

[0104] Fig.10 Schematic diagram of a data operation unit performing an operation method provided in an embodiment of the present application. Fig.10 As shown, the data operation unit may include a floating-point multiplication module and a floating-point addition module, and the multiplication of the weight vector and the activation vector for the corresponding elements may be implemented by the floating-point multiplication module. The multiplication of floating-point numbers includes the addition of exponent bits and the multiplication of mantissa bits, and the addition of exponent bits may be implemented by the exponent addition submodule in the floating-point multiplication module, and the multiplication of mantissa bits may be implemented by the mantissa multiplication submodule in the floating-point multiplication module.

[0105] In implementation, the data handling unit may input the mantissa bits of the corresponding elements in the activation vector and the weight vector into the mantissa multiplication submodule in the floating-point multiplication module for mantissa multiplication. That is, the mantissa bits of the floating-point elements in the activation vector and the mantissa bits of the integer elements in the weight vector are input into the mantissa multiplication submodule for mantissa multiplication to obtain the mantissa multiplication result. If the number of bits of the mantissa bits of the integer elements is less than the number of bits required to be input into the mantissa multiplication submodule, the mantissa bits of the integer elements may be padded with 0.

[0106] Since the exponent in the weight vector is 0, that is, the value of the exponent bit is 0, the sum of the exponent bit of the floating-point element in the activation vector and the exponent bit in the weight vector is still equal to the value of the exponent bit of the floating-point element in the activation vector. Fig.10 In the operation method shown, similarly, a bypass function can also be set, and the processing of the exponential addition submodule in the floating-point multiplication module is omitted, and a direct path can be set between the data handling unit and the exponent alignment submodule in the floating-point addition unit, and the data handling unit can directly input the exponent bit in the activation data into the exponent alignment submodule. Among them, the exponent bit in the activation data is equal to the exponential addition result of the exponent bit in the activation data and the exponent bit in the weight data. For the bypass function, the data operation unit can provide a configuration interface (configuration switch) for controlling the opening of the bypass function. In one example, the bypass function can be turned on by sending an asymmetric calculation instruction to the data operation unit, and then realizing the opening of the bypass function through the configuration interface.

[0107] Continue to see Fig.10 , the addition of floating-point multiplication results can be implemented by a floating-point addition module. Among them, the addition of floating-point numbers includes exponent alignment of the exponent bits of the activation data and the weight data, mantissa accumulation of the mantissa bits of the activation data and the weight data, and normalization of the exponent alignment results and mantissa accumulation. The exponent alignment can be implemented by the exponent alignment submodule in the floating-point addition module, the mantissa accumulation can be implemented by the mantissa accumulation submodule in the floating-point addition module, and the normalization processing is implemented by the normalization submodule in the floating-point addition module.

[0108] After the floating-point multiplication module obtains the exponential addition result and the mantissa multiplication result of the activation data and the weight data, the exponential addition result and the exponent bit of the corresponding element in the accumulation matrix can be input into the exponent alignment submodule for exponent alignment to obtain the exponent alignment result, and the corresponding mantissa multiplication result and the mantissa bit of the corresponding element in the accumulation matrix are input into the mantissa multiplication submodule for mantissa multiplication to obtain the mantissa multiplication result, and then the exponent alignment result and the mantissa multiplication result are input into the normalization submodule, and the normalization submodule performs normalization processing to obtain the exponent bit and mantissa bit of the corresponding element in the updated accumulation matrix.

[0109] Corresponding to the above bypass function, the exponent addition result input to the exponent alignment submodule can be the exponent bit of the activation data input through the bypass function. In addition, the calculation of the activation data and the sign bit of the weight data is the same as that of the prior art, and will not be repeated in the embodiments of the present application.

[0110] In the embodiment of the present application, on the one hand, floating-point operations of integer weight data and floating-point activation data can be realized, which can reduce the impact of quantization compression on the accuracy of the AI ​​model. On the other hand, in the process of realizing the outer product operation of the weight vector and the activation vector, the processing of the exponential addition submodule in the floating-point multiplication module can be omitted through the bypass function, which can reduce the power consumption of the data operation unit.

[0111] The present application also provides a processor, which can be as follows: Figure 3 The processor shown includes a data handling unit and a data computing unit. The processor can be used to execute the computing method provided in the embodiment of the present application, including:

[0112] The data handling unit is used to obtain the quantized weight vector and the activation vector used to perform vector operations with the weight vector, wherein the elements in the weight vector are integer numbers and the elements in the activation vector are floating point numbers.

[0113] The data handling unit is used to input the sign bit and the mantissa bit of each element in the weight vector as the sign bit and the mantissa bit of the floating point number with an exponent of 0 into the data operation unit, and input each element in the activation vector into the data operation unit.

[0114] The data operation unit is used to perform vector operations on the input weight vector and activation vector to obtain operation results.

[0115] In one implementation, the data operation unit includes a floating-point multiplication module and a floating-point addition module, and the vector operation is a vector inner product. The floating-point multiplication module is used to perform a mantissa multiplication on the mantissa of each element in the weight vector and the mantissa of the corresponding element in the activation vector to obtain a mantissa multiplication result. The floating-point addition module is used to perform a floating-point addition operation on the exponent bit of each element in the activation vector and the mantissa multiplication result to obtain an operation result.

[0116] In one implementation, the data operation unit further includes an exponent alignment module, and the floating-point addition module includes a mantissa accumulation submodule and a normalization submodule. The exponent alignment module is used to perform exponent alignment on the exponent bit of each element in the activation vector to obtain an exponent alignment result. The mantissa accumulation submodule is used to perform accumulation processing on the mantissa multiplication result to obtain a mantissa addition result. The normalization submodule is used to perform normalization processing on the exponent alignment result and the mantissa addition result to obtain an operation result.

[0117] In one implementation, the floating-point addition module includes a plurality of mantissa accumulation submodules and a plurality of normalization submodules.

[0118] In one implementation, the data operation unit includes a floating-point multiplication module and a floating-point addition module, and the vector operation is a vector outer product. The data handling unit is used to input the exponent bit and mantissa bit of each element in the accumulation matrix obtained in the vector outer product process into the floating-point addition module. The floating-point multiplication module is used to perform mantissa multiplication on the mantissa bit of each element in the weight vector with the mantissa bit of each element in the activation vector to obtain the mantissa multiplication result. The floating-point addition module is used to input the exponent bit of each element in the activation vector, the mantissa multiplication result, and the exponent bit and mantissa bit of each element in the accumulation matrix into the floating-point addition module to obtain the operation result.

[0119] In one implementation, the data handling unit is used to input the mantissa bits of each element in the weight vector as the sign bit and mantissa bits of a floating point number with an exponent of 0, and input 0 as the exponent bit of the floating point number with an exponent of 0 to the data operation unit.

[0120] In one implementation, the data handling unit is used to input the sign bit and mantissa bit of each element in the weight vector into the data operation unit as the sign bit and mantissa bit of a floating point number with an exponent of 0 and eight bits.

[0121] In one implementation, when the elements in the weight vector are less than eight bits, the data handling unit is further used to fill 0 between the sign bit and the mantissa bit of the elements in the weight vector; or, fill 0 after the mantissa bit of the elements in the weight vector.

[0122] In one implementation, the data operation unit is used to perform vector operations on the input weight vector and obtain an intermediate result; perform floating-point multiplication operations on the intermediate result and the quantization coefficient to obtain an operation result, and the quantization coefficient is used to reduce the numerical error generated after the elements in the weight vector are treated as floating-point numbers.

[0123] In one implementation, the data operation unit is further configured to receive an asymmetric operation instruction.

[0124] The processor provided in the embodiment of the present application can realize floating-point operations by using quantized integer weight vectors and floating-point activation vectors. In this way, on the one hand, when quantizing and compressing the AI ​​model, only the weight data can be quantized, which can reduce the impact of quantization compression on the accuracy of the AI ​​model. On the other hand, in the process of performing floating-point operations on integers and floating-point numbers, there is no need to convert the integers into floating-point numbers, thereby improving the efficiency of floating-point operations.

[0125] The specific steps of the processor provided in the embodiment of the present application to execute the integer and floating-point calculation method are consistent with the execution steps of the calculation method provided in the above embodiment. Please refer to the content of the above embodiment for details, and will not be repeated in the embodiment of the present application.

[0126] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the computing method provided in the present application.

[0127] The present application embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the computing method provided in the present application embodiment.

[0128] In this application, the terms "first", "second", etc. are used to distinguish between identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first" and "second", nor is the quantity and execution order limited. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another element. In this application, the term "at least one" means one or more, and the term "multiple" means two or more.

[0129] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A calculation method, characterized in that: The method is executed by a processor, wherein the processor includes a data handling unit and a data computing unit, and the method includes: The data handling unit obtains a quantized weight vector in the artificial intelligence AI model and an activation vector used to perform vector operations with the weight vector, wherein the elements in the weight vector are integers and the elements in the activation vector are floating-point numbers; The data handling unit inputs the sign bit and the mantissa bit of each element in the weight vector as the sign bit and the mantissa bit of the floating point number with an exponent of 0 to the data operation unit, and inputs each element in the activation vector to the data operation unit; The data operation unit performs vector operation on the input weight vector and the activation vector to obtain an operation result.

2. The method according to claim 1, characterized in that The data operation unit includes a floating-point multiplication module and a floating-point addition module, the vector operation is a vector inner product, and the data operation unit performs a vector operation on the input weight vector and the activation vector to obtain an operation result, including: The floating-point multiplication module performs mantissa multiplication on the mantissa bits of each element in the weight vector and the mantissa bits of the corresponding elements in the activation vector to obtain a mantissa multiplication result; The floating-point addition module performs a floating-point addition operation on the multiplication result of the exponent bit of each element in the activation vector and the mantissa to obtain the operation result.

3. The method according to claim 2, characterized in that The data operation unit further includes an exponent alignment module, and the floating point addition module includes a mantissa accumulation submodule and a normalization submodule; The method further comprises: the exponent alignment module performs exponent alignment on the exponent bit of each element in the activation vector to obtain an exponent alignment result; The floating-point addition module performs a floating-point addition operation on the multiplication result of the exponent bit of each element in the activation vector and the mantissa to obtain the operation result, including: The mantissa accumulation submodule accumulates the mantissa multiplication results to obtain a mantissa addition result; The normalization submodule normalizes the exponent alignment result and the mantissa addition result to obtain a calculation result.

4. The method according to claim 3, characterized in that The floating point addition module includes a plurality of mantissa accumulation submodules and a plurality of normalization submodules.

5. The method according to claim 1, characterized in that The data operation unit includes a floating-point multiplication module and a floating-point addition module, and the vector operation is a vector outer product; The method further comprises: the data handling unit inputting the exponent bit and the mantissa bit of each element in the accumulation matrix obtained in the vector outer product process into the floating point addition module; The data operation unit performs vector operation on the input weight vector and the activation vector to obtain an operation result, including: The floating-point multiplication module performs mantissa multiplication on the mantissa bits of each element in the weight vector and the mantissa bits of each element in the activation vector to obtain a mantissa multiplication result; The floating-point addition module inputs the exponent bit of each element in the activation vector, the mantissa multiplication result, and the exponent bit and mantissa bit of each element in the accumulation matrix into the floating-point addition module to obtain the operation result.

6. The method according to claim 1, characterized in that The data handling unit inputs the sign bit and the mantissa bit of each element in the weight vector as the sign bit and the mantissa bit of a floating point number with an exponent of 0 to the data operation unit, including: The data handling unit uses the mantissa of each element in the weight vector as the sign bit and mantissa of a floating point number with an exponent of 0, and inputs 0 as the exponent of the floating point number with an exponent of 0 to the data operation unit.

7. The method according to any one of claims 1 to 6, characterized in that: The data handling unit inputs the sign bit and the mantissa bit of each element in the weight vector as the sign bit and the mantissa bit of a floating point number with an exponent of 0 to the data operation unit, including: The data handling unit inputs the sign bit and the mantissa bit of each element in the weight vector into the data operation unit as the sign bit and the mantissa bit of a floating point number with an exponent of 0 and eight bits.

8. The method according to claim 7, characterized in that In the case where the elements in the weight vector are less than eight bits, the method further comprises: Filling 0 between the sign bit and the mantissa bit of the elements in the weight vector; or, The tail digits of the elements in the weight vector are padded with 0.

9. The method according to any one of claims 1 to 8, characterized in that: The data operation unit performs vector operation on the input weight vector and the activation vector to obtain an operation result, including: The data operation unit performs vector operation on the input weight vector and the vector to obtain an intermediate result; The data operation unit performs a floating-point multiplication operation on the intermediate result and the quantization coefficient to obtain the operation result, and the quantization coefficient is used to reduce the numerical error generated when the elements in the weight vector are treated as floating-point numbers.

10. The method according to any one of claims 1 to 9, characterized in that: Before the data operation unit performs vector operation on the input weight vector and the activation vector, the data operation unit further includes: The data operation unit receives an asymmetric operation instruction corresponding to the weight vector and the activation vector.

11. A processor, characterized in that: The processor includes a data handling unit and a data computing unit, wherein: The data handling unit is used to obtain a quantized weight vector in the artificial intelligence AI model and an activation vector used to perform vector operations with the weight vector, wherein the elements in the weight vector are integers and the elements in the activation vector are floating-point numbers; The data handling unit is used to input the sign bit and the mantissa bit of each element in the weight vector as the sign bit and the mantissa bit of the floating point number with an exponent of 0 to the data operation unit, and input each element in the activation vector to the data operation unit; The data operation unit is used to perform vector operation on the input weight vector and the activation vector to obtain an operation result.

12. The processor according to claim 11, characterized in that The data operation unit includes a floating-point multiplication module and a floating-point addition module, and the vector operation is a vector inner product; The floating-point multiplication module is used to perform mantissa multiplication on the mantissa bits of each element in the weight vector and the mantissa bits of the corresponding elements in the activation vector to obtain a mantissa multiplication result; The floating-point addition module is used to perform a floating-point addition operation on the multiplication result of the exponent bit of each element in the activation vector and the mantissa to obtain the operation result.

13. The processor according to claim 12, characterized in that The data operation unit further includes an exponent alignment module, and the floating point addition module includes a mantissa accumulation submodule and a normalization submodule; The exponent alignment module is used to perform exponent alignment on the exponent bit of each element in the activation vector to obtain an exponent alignment result; The mantissa accumulation submodule is used to accumulate the mantissa multiplication results to obtain a mantissa addition result; The normalization submodule is used to perform normalization processing on the exponent alignment result and the mantissa addition result to obtain a calculation result.

14. The processor according to claim 13, characterized in that The floating point addition module includes a plurality of mantissa accumulation submodules and a plurality of normalization submodules.

15. The processor according to claim 11, characterized in that The data operation unit includes a floating-point multiplication module and a floating-point addition module, and the vector operation is a vector outer product; The data handling unit is used to input the exponent bit and the mantissa bit of each element in the accumulation matrix obtained in the vector outer product process into the floating point addition module; The floating-point multiplication module is used to perform mantissa multiplication on the mantissa bits of each element in the weight vector and the mantissa bits of each element in the activation vector to obtain a mantissa multiplication result; The floating-point addition module is used to input the exponent bit of each element in the activation vector, the mantissa multiplication result, and the exponent bit and mantissa bit of each element in the accumulation matrix into the floating-point addition module to obtain the operation result.

16. The processor according to claim 11, characterized in that The data handling unit is used for the data handling unit to use the mantissa bits of each element in the weight vector as the sign bit and mantissa bits of a floating point number with an exponent of 0, and to input 0 as the exponent bit of the floating point number with an exponent of 0 to the data calculation unit.

17. The processor according to any one of claims 11 to 16, characterized in that: The data handling unit is used to input the sign bit and the mantissa bit of each element in the weight vector into the data operation unit as the sign bit and the mantissa bit of a floating point number with an exponent of 0 and eight bits.

18. The processor according to claim 17, characterized in that In the case where the elements in the weight vector are less than eight bits, the method further includes: Filling 0 between the sign bit and the mantissa bit of the elements in the weight vector; or, The tail digits of the elements in the weight vector are padded with 0.

19. The processor according to any one of claims 11 to 18, characterized in that: The data operation unit is used to perform vector operation on the input weight vector and the vector to obtain an intermediate result; The data operation unit is used to perform a floating-point multiplication operation on the intermediate result and the quantization coefficient to obtain the operation result, and the quantization coefficient is used to reduce the numerical error generated when the elements in the weight vector are treated as floating-point numbers.

20. The processor according to any one of claims 11 to 19, characterized in that: The data operation unit is further used to receive asymmetric operation instructions corresponding to the weight vector and the activation vector.

21. A computing device, characterized in that The computing device comprises a memory and a processor as claimed in any one of claims 10 to 18 above.

Citation Information

Cited By

  • Tensor kernel operation component

    CN119576274A