Operation method, processor, and computing device

Abstract

Embodiments disclosed in the present application relate to the technical field of computing, and in particular to an operation method, a processor, and a computing device. The method is executed by a processor comprising a data transfer unit and a data operation unit, and comprises: the data transfer unit acquiring an activation vector and a quantized weight vector in an AI model, the weight vector being an integer, and the activation vector being a floating point number; the data transfer unit inputting to the data operation unit a sign bit and a mantissa bit of each element in the weight vector as a sign bit and a mantissa bit of the floating point number having an index of 0, and inputting each element in the activation vector to the data operation unit; the data operation unit performing a vector operation on the inputted weight vector and activation vector, to obtain an operation result. Using the present application, operations can be performed on an integer number and a floating point number in the data operation unit. Thus, when an AI model is quantitatively compressed, it is possible to only compress the weight data, and the impact of quantitative compression on the precision of the AI model can be reduced.

Description

Computing method, processor and computing device

[0001] This application claims priority to Chinese patent application No. 202310633793.X, filed on May 30, 2023, with invention name “A data calculation method”, and priority to Chinese patent application No. 202311092509.9, filed on August 28, 2023, with invention name “Operation method, processor and computing device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computing technology, and in particular to an operation method, a processor, and a computing device. Background Art

[0003] Quantization compression of artificial intelligence (AI) models refers to a method of quantizing the weight data and activation data in the AI ​​model from floating-point numbers to integers, thereby improving the running speed of the AI ​​model.

[0004] While quantization compression can increase the speed of AI models, it also reduces their accuracy. Because activation data has a wider range of values ​​and contains a certain proportion of outliers, the impact of activation data quantization on AI model accuracy is greater than that of weight data quantization.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a calculation method, a processor, and a computing device that can implement integer and floating-point operations in a data operation unit, and can reduce the decrease in model accuracy when quantizing a model. The corresponding technical solutions are as follows:

[0007] In a first aspect, a computing method is provided. The computing method can be executed by a processor including a data handling unit and a data computing unit. The computing method executed by the processor includes:

[0008] The data handling unit obtains the quantized weight vector from the artificial intelligence (AI) model and the activation vector used to perform vector operations on the weight vector. The elements in the weight vector are integers, and the elements in the activation vector are floating-point numbers. The data handling unit inputs the sign and mantissa bits of each element in the weight vector into the data operation unit as if they were the sign and mantissa bits of a floating-point number with an exponent of 0. It also inputs each element in the activation vector into the data operation unit. The data operation unit performs vector operations on the input weight vector and activation vector to obtain the operation results.

[0009] In the solution shown in the present application, for the elements in the weight vector that are integers, the elements can be used as floating-point input value data operation units with an exponent of 0 to perform operations with the activation vector of the floating-point data. Among them, the sign bit of the integer element is the sign bit of the floating-point number with an exponent of 0, and the mantissa bit of the integer element is the mantissa bit of the floating-point number with an exponent of 0. In addition, since the exponent bit in the floating-point number with an exponent of 0 is 0, it does not participate in the floating-point operation. It can be seen that the solution shown in the present application can realize floating-point operations with the weight vector of the quantized integer and the activation vector of the floating-point number. In this way, on the one hand, when quantizing and compressing the AI ​​model, only the weight data can be quantized, which can reduce the impact of quantization compression on the accuracy of the AI ​​model. On the other hand, in the process of performing floating-point operations on integers and floating-point numbers, there is no need to convert the integer to a floating-point number, thereby improving the efficiency of floating-point operations.

[0010] In one implementable manner, the data operation unit includes a floating-point multiplication module and a floating-point addition module. The vector operation is a vector inner product. The data operation unit performs a vector operation on the input weight vector and activation vector to obtain an operation result, including: the floating-point multiplication module performs a mantissa multiplication on the mantissa of each element (weight data) in the weight vector and the mantissa of the corresponding element (activation data) in the activation vector to obtain a mantissa multiplication result. The floating-point addition module performs a floating-point addition operation on the exponent bit of each element in the activation vector and the mantissa multiplication result to obtain an operation result.

[0011] In the scheme shown in the present application, since the exponent bit of the floating-point number with an exponent of 0 is 0, the exponent bit in the activation data is the sum of the exponent bits corresponding to the activation data and the weight data. Therefore, the floating-point multiplication module can only perform mantissa multiplication on the mantissa bits corresponding to the activation data and the weight data. In this way, the exponential addition process in the floating-point multiplication operation can be avoided, thereby reducing the power consumption of the data operation unit. When performing a floating-point addition operation, the floating-point addition module can directly obtain the exponent bit of the activation data as the sum of the exponent bits corresponding to the activation data and the weight data, and then perform a floating-point addition operation in combination with the mantissa multiplication result to obtain the operation result.

[0012] In one achievable manner, the data operation unit further includes an exponent alignment module, and the floating-point addition module includes a mantissa accumulation submodule and a normalization submodule. The above-mentioned operation method further includes: the exponent alignment module performs exponent alignment on the exponent bit of each element in the activation vector to obtain an exponent alignment result. The floating-point addition module performs a floating-point addition operation on the exponent bit of each element in the activation vector and the mantissa multiplication result to obtain an operation result, including: the mantissa accumulation submodule performs accumulation processing on the mantissa multiplication result to obtain a mantissa addition result. The normalization submodule performs normalization processing on the exponent alignment result and the mantissa addition result to obtain an operation result.

[0013] Among them, the floating-point addition operation includes exponent alignment, mantissa accumulation, and normalization of the exponent after exponent alignment and the mantissa after mantissa accumulation. In the scheme shown in the present application, since the exponent corresponding to the weight data is 0, when performing exponent alignment, the exponent in the activation data can be only aligned through the exponent alignment module. And since the same activation vector may perform vector operations with multiple weight vectors during the operation of the AI ​​model, in the present application, the result of exponential alignment of the activation data by the exponent alignment module can be reused, thereby improving the efficiency of vector operations.

[0014] In one implementable manner, the floating-point addition module includes a plurality of mantissa accumulation submodules and a plurality of normalization submodules.

[0015] In the solution presented in this application, since the exponent alignment results of the exponent alignment module can be reused, doubling the performance of the data operation unit requires only increasing the number of mantissa accumulation submodules and normalization submodules. Therefore, the computation method provided in this application can improve the performance of the data operation unit at a lower hardware cost.

[0016] In one implementable manner, the data operation unit includes a floating-point multiplication module and a floating-point addition module, and the vector operation is a vector outer product. The data operation unit performs vector operation on the input weight vector and activation vector to obtain an operation result, including: the data handling unit inputs the exponent bit and mantissa bit of each element in the accumulation matrix obtained in the vector outer product process into the floating-point addition module. The floating-point multiplication module performs mantissa multiplication on the mantissa bit of each element in the weight vector with the mantissa bit of each element in the activation vector to obtain a mantissa multiplication result. The floating-point addition module inputs the exponent bit of each element in the activation vector, the mantissa multiplication result, and the exponent bit and mantissa bit of each element in the accumulation matrix into the floating-point addition module to obtain an operation result.

[0017] In the scheme shown in the present application, when the data operation unit performs the vector outer product on the weight vector and the activation vector, since the weight data is input into the data operation unit as a floating-point number with an exponent of 0, the exponent bit in the activation data is the sum of the exponent bits corresponding to the activation data and the weight data. Therefore, the floating-point multiplication module can only perform mantissa multiplication on the mantissa bits corresponding to the activation data and the weight data. In this way, the exponential addition process in the floating-point multiplication operation can be avoided, thereby reducing the power consumption of the data operation unit. When performing a floating-point addition operation, the floating-point addition module can directly align the exponent bits of the activation data and the corresponding elements in the accumulation matrix, and perform mantissa accumulation on the mantissa multiplication result and the mantissa bits of the corresponding elements in the accumulation matrix to obtain the operation result.

[0018] In one implementable manner, the data handling unit inputs the sign bit and mantissa bit of each element in the weight vector as the sign bit and mantissa bit of a floating-point number with an exponent of 0 to the data operation unit, including: the data handling unit inputs the mantissa bit of each element in the weight vector as the sign bit and mantissa bit of a floating-point number with an exponent of 0, and inputs 0 as the exponent bit of the floating-point number with an exponent of 0 to the data operation unit.

[0019] In the solution shown in the present application, the data handling unit can input the sign bit of each element in the weight vector as the sign bit of a floating-point number with a number of 0, the mantissa bit of each element in the weight vector as the mantissa bit of a floating-point number with a number of 0, and 0 as the exponent bit of a floating-point number with an exponent of 0 into the data operation unit, thereby enabling the data operation unit to perform floating-point operations on the weight data of integers and the activation data of floating-point numbers. In this way, on the one hand, when quantizing and compressing the AI ​​model, only the weight data can be quantized, which can reduce the impact of quantization compression on the accuracy of the AI ​​model. On the other hand, in the process of performing floating-point operations on integers and floating-point numbers, there is no need to convert the integers into floating-point numbers, thereby improving the efficiency of floating-point operations.

[0020] In one implementable manner, the data handling unit inputs the sign bit and mantissa bits of each element in the weight vector as the sign bit and mantissa bits of a floating-point number with an exponent of 0 to the data operation unit, including: the data handling unit inputs the sign bit and mantissa bits of each element in the weight vector as the sign bit and mantissa bits of a floating-point number with an exponent of 0 and eight bits to the data operation unit.

[0021] In the solution shown in this application, the data handling unit can input each element in the weight vector as an eight-bit floating-point number with an exponent of 0 to the data operation unit for floating-point operation with the activation data of the floating-point number. The first bit of the eight-bit floating-point number with an exponent of 0 is the sign bit of the floating-point number, and the last seven bits are the mantissa bits of the floating-point number. This form of floating-point number can be called FP8.

[0022] In one achievable manner, when the elements in the weight vector are less than eight bits, the method further includes: padding 0 between the sign bit and the mantissa bit of the elements in the weight vector; or padding 0 after the mantissa bit of the elements in the weight vector.

[0023] In the scheme shown in this application, the elements in the weight vector are less than eight bits, which means that the elements in the weight vector are IN4 or INT6. Therefore, the mantissa bits of the elements in the weight vector can be padded with 0 first, and the elements in the weight vector can be padded to eight bits and then input into the data operation unit as FP8 for calculation.

[0024] In one achievable embodiment, the data operation unit performs a vector operation on the input weight vector and activation vector to obtain an operation result, including: performing a vector operation on the input weight vector and activation vector to obtain an intermediate result. The data operation unit performs a floating-point multiplication operation on the intermediate result and a quantization coefficient to obtain an operation result, where the quantization coefficient is used to reduce numerical errors caused by treating elements in the weight vector as floating-point numbers.

[0025] In the scheme shown in the present application, the mantissa of the integer number is directly input into the data operation unit as the mantissa of the floating-point number with an exponent of 0 for operation, and the obtained operation result will have errors. Therefore, after the data operation unit performs vector operations on the input weight vector and activation vector, the intermediate result obtained can be multiplied by the quantization coefficient that eliminates the error to obtain the operation result after the error is eliminated. This can improve the accuracy of the data operation unit's operations on the weight data and activation data.

[0026] In one achievable manner, before the data handling unit obtains the quantized weight vector and the activation vector used to perform vector operations with the weight vector, the method further includes: the data operation unit receiving an asymmetric operation instruction.

[0027] In the scheme shown in the present application, before the data operation unit executes the above-mentioned operation method, an asymmetric operation instruction can be sent to the data operation unit to instruct the data operation unit to execute the above-mentioned operation method, thereby realizing floating-point operations corresponding to the weight data of integer numbers and the activation data of floating-point numbers.

[0028] In a second aspect, a processor is provided, the processor including a data handling unit and a data operation unit, wherein:

[0029] A data handling unit, configured to obtain a quantized weight vector and an activation vector for performing vector operations on the weight vector, wherein the elements in the weight vector are integers and the elements in the activation vector are floating-point numbers;

[0030] A data handling unit, configured to input the sign bit and mantissa bit of each element in the weight vector as the sign bit and mantissa bit of a floating-point number with an exponent of 0 into the data operation unit, and input each element in the activation vector into the data operation unit;

[0031] The data operation unit is used to perform vector operations on the input weight vector and activation vector to obtain the operation results.

[0032] In one implementation, the data operation unit includes a floating-point multiplication module and a floating-point addition module, and the vector operation is a vector inner product. The floating-point multiplication module is configured to perform a mantissa multiplication on the mantissa of each element in the weight vector with the mantissa of the corresponding element in the activation vector to obtain a mantissa multiplication result. The floating-point addition module is configured to perform a floating-point addition operation on the exponent of each element in the activation vector and the mantissa multiplication result to obtain a calculation result.

[0033] In one implementation, the data operation unit further includes an exponent alignment module, and the floating-point addition module includes a mantissa accumulation submodule and a normalization submodule. The exponent alignment module is configured to align the exponent bit of each element in the activation vector to obtain an exponent alignment result. The mantissa accumulation submodule is configured to accumulate the mantissa multiplication results to obtain a mantissa addition result. The normalization submodule is configured to normalize the exponent alignment result and the mantissa addition result to obtain a calculation result.

[0034] In one implementation, the floating-point addition module includes a plurality of mantissa accumulation submodules and a plurality of normalization submodules.

[0035] In one implementation, the data operation unit includes a floating-point multiplication module and a floating-point addition module, and the vector operation is a vector outer product. The data handling unit is used to input the exponent bit and mantissa bit of each element in the accumulation matrix obtained in the vector outer product process into the floating-point addition module. The floating-point multiplication module is used to perform mantissa multiplication on the mantissa bit of each element in the weight vector with the mantissa bit of each element in the activation vector to obtain the mantissa multiplication result. The floating-point addition module is used to input the exponent bit of each element in the activation vector, the mantissa multiplication result, and the exponent bit and mantissa bit of each element in the accumulation matrix into the floating-point addition module to obtain the operation result.

[0036] In one implementation, the data handling unit is configured to input the mantissa bit of each element in the weight vector as the sign bit and mantissa bit of a floating-point number with an exponent of 0, and input 0 as the exponent bit of the floating-point number with an exponent of 0 to the data operation unit.

[0037] In one implementation, the data handling unit is configured to input the sign bit and mantissa bit of each element in the weight vector as the sign bit and mantissa bit of an eight-bit floating point number with an exponent of 0 to the data operation unit.

[0038] In one implementation, when the elements in the weight vector are less than eight bits, the data handling unit is further configured to fill 0 between the sign bit and the mantissa bit of the elements in the weight vector; or fill 0 after the mantissa bit of the elements in the weight vector.

[0039] In one implementation, the data operation unit is used to perform vector operations on the input weight vector and obtain an intermediate result; perform floating-point multiplication operations on the intermediate result and the quantization coefficient to obtain an operation result, and the quantization coefficient is used to reduce the numerical error generated after the elements in the weight vector are treated as floating-point numbers.

[0040] In one implementation, the data operation unit is further configured to receive an asymmetric operation instruction.

[0041] In a third aspect, a computing device is provided, which includes a memory and a processor as described in the second aspect, and the processor can be used to implement the operation method as described in the first aspect.

[0042] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program code. When the computer program code is executed by a computing device, the computing device executes the operation method described in the first aspect above.

[0043] In a fifth aspect, a computer program product comprising instructions is provided, which, when run on a computing device, enables the computing device to execute the operation method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] FIG1 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0045] FIG2 is a schematic diagram of a processor structure provided in an embodiment of the present application;

[0046] FIG3 is a flow chart of a calculation method provided in an embodiment of the present application;

[0047] FIG4 is a schematic diagram of the structure of an integer and a floating-point number provided in an embodiment of the present application;

[0048] FIG5 is a schematic diagram of a vector inner product operation provided in an embodiment of the present application;

[0049] FIG6 is a schematic diagram of a method for executing a calculation by a data calculation unit provided in an embodiment of the present application;

[0050] FIG7 is a schematic diagram of a method for executing a calculation by a data calculation unit provided in an embodiment of the present application;

[0051] FIG8 is a schematic diagram of a method for executing a calculation by a data calculation unit according to an embodiment of the present application;

[0052] FIG9 is a schematic diagram of a vector outer product operation provided in an embodiment of the present application;

[0053] FIG10 is a schematic diagram of a method for executing a calculation by a data calculation unit provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0055] At the beginning of AI model design, in order to improve the accuracy of the AI ​​model, the weight data and activation data will be set to higher-precision floating-point numbers, such as FP32 and FP16. Floating-point numbers occupy more bits, such as FP32 floating-point numbers occupy 32 bits, and FP16 floating-point numbers occupy 16 bits. Therefore, floating-point numbers require more storage and computing resources when stored and calculated. Especially in some large models, such as large language models (LLM), the number of parameters is huge, and the storage of weight data alone requires hundreds of GB of storage space. Among them:

[0056] The AI ​​model can be any trained artificial intelligence model, such as a large language model, an image recognition model, etc.

[0057] Weight data includes the weight parameters of the AI ​​model after training. During the actual operation of the AI ​​model, weight data is generally represented in the form of vectors or matrices.

[0058] Activation data can be the output data of the AI ​​model's intermediate layers, related to the sample data input to the AI ​​model. During the actual operation of the AI ​​model, activation data can also be represented in the form of vectors or matrices, and often performs vector or matrix operations with weight data.

[0059] To improve the universality of AI models, quantization compression is used to reduce the storage space required to store AI models and increase their speed. Quantization compression is a method for quantizing the weight and activation data in AI models from floating-point numbers to integers, thereby increasing their speed. For example, FP16 weight and activation data can be converted to INT8 integers. This reduces the storage of weight data from 16 bits to 8 bits.

[0060] However, converting floating-point numbers to integers reduces the precision of the numbers, and certain errors are introduced during the conversion process. The larger the range of floating-point numbers, the greater the error in converting them to integers. Therefore, quantization compression can significantly impact the accuracy of AI models.

[0061] Since in AI models, the numerical range of activation data is generally larger than the numerical range of weight data, and there is a certain proportion of outliers in the activation data, the error caused by the quantization of activation data is greater, and the impact on the accuracy of the AI ​​model is greater than the quantization of weight data. In addition, the applicant found that in some AI models, especially in large generative models, the amount of weight data is much larger than the amount of activation data. Therefore, quantizing the weight data can bring better quantization effects to the AI ​​model, and the impact on the accuracy of the AI ​​model is less than the quantization of activation data.

[0062] Therefore, the applicant found that if the impact of quantization compression on the accuracy of the AI ​​model is reduced, in theory only the weight data can be quantized. However, since the current hardware cannot support operations on floating-point numbers and integer numbers, that is, it cannot support operations on unquantized activation data and quantized weight data, this theory cannot be directly implemented using existing technology.

[0063] The embodiment of the present application provides a calculation method that can realize the calculation of floating-point numbers and integer numbers on the existing data calculation unit. In this way, when the AI ​​model is quantized and compressed, the weight data in the AI ​​model can be quantized into integers offline. During the actual operation of the AI ​​model, the activation data may not be quantized, but the calculation method provided by the present application may be used to realize the calculation of floating-point activation data and integer weight data. In this way, by quantizing the weight data, the storage space occupied by the AI ​​model can be reduced and the operating efficiency of the AI ​​model can be improved. Because the activation data is not quantized, the impact of quantization compression on the accuracy of the AI ​​model is reduced.

[0064] The embodiment of the present application provides a calculation method that can be implemented by a computing device. Figure 1 is a schematic diagram of the structure of a computing device provided by an embodiment of the present application. As shown in Figure 1, the computing device 100 includes: a bus 102, a processor 104, and a memory 106. The computing device 100 may optionally also include a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other via the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 100. The computing device 100 can be a device that runs an AI model, a terminal or a server. When the computing device 100 is a terminal, the computing device 100 includes but is not limited to a desktop computer, a mobile phone, a notebook, a tablet computer, etc. When the computing device 100 is a server, the computing device 100 can be a single server, or a server cluster consisting of multiple servers, and can be a physical machine, or a virtual machine or container virtualized by virtualization technology.

[0065] Bus 102 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG1 illustrates a single bus line, but this does not imply a single bus or type of bus. Bus 102 may include a path for transmitting information between various components of computing device 100 (e.g., memory 106, processor 104, and communication interface 108).

[0066] The processor 104 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP). The processor 104 may further include a data handling unit and a data operation unit. The data handling unit is used to implement the data handling function of the processor, such as direct memory access (DMA). The data operation unit is used to implement the operation of floating-point activation data and integer weight data. For example, the data operation unit may be a matrix operation unit, such as a matrix acceleration engine (cube in NPU, tensor core of GPU), etc.

[0067] The memory 106 may include volatile memory, such as random access memory (RAM). The memory 106 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0068] The communication interface 108 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 100 and other devices or a communication network.

[0069] FIG2 is a schematic diagram of a processor structure provided in an embodiment of the present application. The processor may be the processor 104 in the computing device of FIG2 . As shown in FIG2 , the processor includes a data handling unit and a data operation unit. The processor can execute the operation method provided in an embodiment of the present application. FIG3 is a flow chart of an operation method provided in an embodiment of the present application. Referring to FIG3 , the operation method executed by the processor includes:

[0070] Step 301: The data handling unit obtains a quantized weight vector in the artificial intelligence (AI) model and an activation vector for performing vector operations with the weight vector, wherein the elements in the weight vector are integers and the elements in the activation vector are floating-point numbers.

[0071] During the operation of the AI ​​model, weight data and activation data are generally represented in the form of matrices or vectors. During the operation process, even if the weight data and activation data are represented as matrices, they are first converted into vectors for operation. In this application, the weight vector and activation vector are vectors composed of the weight data and activation data to be operated, respectively. In one example, the data handling unit can read the weight vector and activation vector from the memory.

[0072] Among them, each element in the weight vector can be quantized weight data. The quantization method of the weight data can adopt uniform quantization or non-uniform quantization, and its specific quantization method will not be introduced in detail in this application. In one example, uniform quantization can be used to quantize the weight data from FP16 floating point numbers to INT8 integer numbers. The elements in the activation vector can be the activation data output by the middle layer of the model during the operation of the model. The activation data is a floating point number. In one example, the activation data does not need to be quantized.

[0073] Step 302: The data handling unit inputs the sign bit and mantissa bit of each element in the weight vector as the sign bit and mantissa bit of a floating-point number with an exponent of 0 into the data operation unit, and inputs each element in the activation vector into the data operation unit.

[0074] As shown in Figure 4, an integer number includes a sign bit and a mantissa, where the sign bit is the first bit in the integer number, and the mantissa is the bits in the integer number other than the first bit. For example, in an INT8 integer number, the first bit is the sign bit, and the next seven bits are the mantissa. A floating-point number includes a sign bit, an exponent bit, and a mantissa, where the sign bit is the first bit in the integer number, and the exponent bit is located between the sign bit and the mantissa. For example, in an FP16 floating-point number, the first bit is the sign bit, the next ten bits are the mantissa, and the remaining five bits between the sign bit and the mantissa are the exponent bits.

[0075] Since floating point numbers can be expressed as f=s*(t+1)*2 i , where f represents a floating point number, s is the value corresponding to the sign bit (±), t+1 is the value corresponding to the mantissa bit, and i is the value corresponding to the exponent bit. When the exponent in the floating point number f is 0, 2 i =1, therefore, the floating-point number can be expressed as f=s*(t+1)*1, i.e., f=s*(t+1), that is, the floating-point number with an exponent of 0 can be represented by the mantissa and the sign bit. For integers, it can be expressed as I=s*m, where I represents an integer, s is the value corresponding to the sign bit, and m is the mantissa. It can be seen that when the exponent of a floating-point number is 0, the floating-point number and the integer can have the same representation. Therefore, in this application, the weight data of the integer can be used as a floating-point number with an exponent of 0 to perform floating-point operations with the activation data.

[0076] Each element in the weight vector is treated as a floating-point number with an exponent of 0, that is, the integer is treated as a floating-point number with an exponent of 0. As shown in Figure 4, the sign bit of the integer is the sign bit of the floating-point number with an exponent of 0, and the mantissa bits of the integer are the mantissa bits of the floating-point number with an exponent of 0. Since the exponent of this floating-point number is 0, that is, the exponent bit is 0, the exponent bit of this floating-point number can be temporarily not set.

[0077] In one achievable manner, when an integer is required to be input into the data operation unit as a floating-point number with an exponent of 0, the data handling unit can directly input the mantissa of the integer into the data operation unit as the mantissa of the floating-point number with an exponent of 0. The sign bit of the integer is input into the data operation unit as the sign bit of the floating-point number with an exponent of 0. Although there is no exponent bit in an integer, since the exponent of the floating-point number is 0, in one example, the exponent bit corresponding to the floating-point number with an exponent of 0 can be input into the data operation unit by padding with 0.

[0078] Step 303: The data operation unit performs vector operation on the input weight vector and activation vector to obtain an operation result.

[0079] The data handling unit treats each element of the weight vector as a floating-point number with an exponent of 0, and each element of the activation vector as a floating-point number. The data operation unit then performs a vector operation on the weight vector and activation vector to produce the result. The further operation process is not described in detail here.

[0080] Since the mantissa bits in the floating-point number and the mantissa bits in the integer number correspond to the same binary string, but the values ​​represented are different, for example, the mantissa bits in the floating-point number and the mantissa bits in the integer number are both "0101", and the mantissa bits in the integer number are "0101" can represent decimal 5, but the mantissa bits in the floating-point number are "0101" can represent a certain error between the numerical value and the decimal 5, and the error is a linear error, which can be compensated by the quantization coefficient (scale). Therefore, in step 303, after the data operation unit performs vector operation on the input weight vector and activation vector, the error compensation process can be performed. The corresponding processing is as follows: the data operation unit performs vector operation on the input weight vector and obtains an intermediate result, performs floating-point multiplication operation on the intermediate result and the quantization coefficient to obtain the operation result.

[0081] The quantization coefficient is a floating-point number used to reduce the numerical error caused by converting the elements in the weight vector into floating-point numbers. For this quantization coefficient, scale = a * scale1. scale1 is the quantization parameter obtained by quantizing the original floating-point weight data in the AI ​​model to integer weight data, and a is the minimum value represented by a floating-point number with an exponent of 0.

[0082] In one example, in step 303, after the data operation unit performs vector operations on the input weight vector and activation vector, the result obtained can be called an intermediate result. After obtaining the intermediate result, the intermediate result floating-point number can be multiplied by the quantization coefficient to obtain the final result of the vector operation of the weight vector and the activation vector. This can improve the accuracy of the data operation unit's operations on the weight data and activation data. Among them, the floating-point multiplication of the intermediate result floating-point number and the quantization coefficient can be implemented by the hardware circuit in the data operation unit.

[0083] In addition, in an embodiment of the present application, the weight data is quantized to INT8. In this application, an integer with an exponent of 0 using INT8 may be referred to as FP8. That is, FP8 is a floating point number including a sign bit and seven mantissa bits. When the elements in the weight vector are less than eight bits, it means that the elements in the weight vector are IN4 or INT6, so the mantissa bits of the elements in the weight vector can be padded with 0 first, and then input into the data operation unit as FP8 for operation.

[0084] In the embodiment provided by the present application, the weight data of an integer can be input into the data operation unit as a floating-point number with an exponent of 0, and during the input process, the weight data of the integer does not need to be converted, so that the operation of integers and floating-point numbers can be realized without adding the conversion steps between integers and floating-point numbers. In this way, when the operation method provided by the present application is used to quantize and compress the AI ​​model, only the weight data can be quantized, and the activation data is not quantized. In this way, the operation speed of the AI ​​model can be improved through quantization compression, and the impact of quantization compression on the accuracy of the AI ​​model can be reduced.

[0085] The following is a further introduction to the calculation method of the integer weight vector and the floating-point activation vector provided by this application with reference to the accompanying drawings:

[0086] In one possible implementation, the weight vector and activation vector are a row or column of data included in the weight matrix and activation matrix. A matrix inner product operation is required between the weight matrix and the activation matrix, and the matrix inner product operation is implemented by the inner product of the vectors.

[0087] Correspondingly, the data operation unit may include a floating-point multiplication module and a floating-point addition module. In this case, the processing of the above steps 302 and 302 can be replaced by: the data handling unit inputs the mantissa bits of each element in the weight vector and the mantissa bits of the corresponding elements in the activation vector into the floating-point multiplication module, and inputs the result of multiplying the exponent bits of each element in the activation vector with the mantissa bits into the floating-point addition module. The floating-point multiplication module performs mantissa multiplication on the mantissa bits of each element in the weight vector and the mantissa bits of the corresponding elements in the activation vector to obtain the mantissa multiplication result; performs a floating-point addition operation on the result of multiplying the exponent bits of each element in the activation vector with the mantissa bits to obtain the operation result.

[0088] As shown in Figure 5, the inner product of the weight vector and the activation vector involves multiplying the corresponding elements in the weight vector and the activation vector, and then accumulating the corresponding element-wise multiplication results to obtain the inner product of the weight vector and the activation vector. For example, the activation vector is [a, b, c, d] and the weight vector is [e, f, g, h], where a*e = i, b*f = j, c*g = k, d*e = l, i + j = m, k + l = n, and a through n are all floating-point numbers. The inner product of the activation vector and the weight vector can be expressed as: F = a*e + b*f + c*g + d*e. This shows that the inner product operation of the weight vector and the activation vector involves floating-point multiplication and addition.

[0089] Figure 6 is a schematic diagram of a data operation unit performing an operation method provided by an embodiment of the present application. As shown in Figure 6, the data operation unit may include a floating-point multiplication module and a floating-point addition module, and the multiplication of the corresponding elements of the weight vector and the activation vector can be implemented by the floating-point multiplication module. Among them, the multiplication of floating-point numbers includes the addition of exponent bits and the multiplication of mantissa bits. The addition of exponent bits can be implemented by the exponent addition submodule in the floating-point multiplication module, and the multiplication of mantissa bits can be implemented by the mantissa multiplication submodule in the floating-point multiplication module.

[0090] In implementation, the data handling unit may input the mantissa bits of the corresponding elements in the activation vector and the weight vector into the mantissa multiplication submodule in the floating-point multiplication module for mantissa multiplication. That is, the mantissa bits of the floating-point elements in the activation vector and the mantissa bits of the integer elements in the weight vector are input into the mantissa multiplication submodule for mantissa multiplication to obtain a mantissa multiplication result. If the number of bits of the mantissa bits of the integer element is less than the number of bits required by the mantissa multiplication submodule, the mantissa bits of the integer element may be padded with 0.

[0091] In one example, since the exponent in the weight vector is 0, that is, the value of the exponent bit is 0, the value of the exponent bit of the floating-point element in the activation vector and the exponent bit in the weight vector added together is still equal to the value of the exponent bit of the floating-point element in the activation vector. Therefore, in an embodiment of the present application, a bypass function can be set to omit the processing of the exponential addition submodule in the floating-point multiplication module, that is, a direct path can be set between the exponent alignment submodule in the data handling unit and the floating-point addition unit, and the data handling unit can directly input the exponent bit in the activation data into the exponent alignment submodule. Wherein, the exponent bit in the activation data is equal to the exponential addition result of the exponent bit in the activation data and the exponent bit in the weight data. For the bypass function, the data operation unit can provide a configuration interface (configuration switch) for controlling the opening of the bypass function. In one example, an asymmetric calculation instruction can be sent to the data operation unit, and then the opening of the bypass function can be realized through the configuration interface.

[0092] Continuing to refer to Figure 6, the addition of the floating-point multiplication results can be implemented by the floating-point addition module. Among them, the addition of floating-point numbers includes exponent alignment of the exponent bits of the activation data and the weight data, mantissa accumulation of the mantissa bits of the activation data and the weight data, and normalization of the exponent alignment results and mantissa accumulation. The exponent alignment can be implemented by the exponent alignment submodule in the floating-point addition module, the mantissa accumulation can be implemented by the mantissa accumulation submodule in the floating-point addition module, and the normalization processing is implemented by the normalization submodule in the floating-point addition module.

[0093] After the floating-point multiplication module obtains the exponential addition result and the mantissa multiplication result of the activation data and the weight data, the exponential addition results obtained can be input into the exponent alignment submodule in pairs for exponential alignment to obtain the exponent alignment result, and the corresponding mantissa multiplication results can be input into the mantissa multiplication submodule in pairs for mantissa multiplication to obtain the mantissa multiplication result. Then, the exponential alignment result and the mantissa multiplication result are input into the normalization submodule, and the normalization submodule performs normalization processing to obtain the operation result after normalization processing.

[0094] For the above-mentioned bypass function, the exponential addition result input to the exponent alignment submodule can be the exponent bit of the activation data input through the bypass function. In addition, the calculation of the sign bit of the activation data and the weight data is the same as that of the prior art and will not be repeated in the embodiments of the present application.

[0095] In the embodiments of the present application, on the one hand, floating-point operations can be implemented for integer weight data and floating-point activation data, which can reduce the impact of quantization compression on the accuracy of the AI ​​model. On the other hand, in the process of implementing the inner product operation of the weight vector and the activation vector, the processing of the exponential addition submodule in the floating-point multiplication module can be omitted through the bypass function, which can reduce the power consumption of the data operation unit.

[0096] Figure 7 is a schematic diagram of a method for performing an operation by a data operation unit according to an embodiment of the present application. The method shown in Figure 7 can implement exponent alignment through an exponent alignment module included in the data operation unit based on the method shown in Figure 6.

[0097] Because in the process of running the model, the same activation vector needs to be operated with multiple weight vectors, and because the elements in the weight vector will be treated as floating-point numbers with an exponent of 0. The result of exponential alignment of the activation data in the same activation vector and the weight data in multiple weight vectors is the result of exponential alignment of the activation data in the same activation vector. In this way, it is only necessary to perform exponential alignment on the exponential bits of the elements in the activation vector once, and the exponential alignment result can be reused many times. Therefore, in an embodiment of the present application, a bypass function can be designed to realize the exponential alignment of the exponential bits of the elements in the activation vector by an exponential alignment module (not directly shown in FIG7 ), and the exponential alignment module may not be subordinate to the matrix addition module.

[0098] In one example, when the exponent bits of elements in the activation vector are initially aligned, this can be accomplished by the exponent alignment module. After obtaining the exponent alignment result, it can be stored. When the exponent bits of elements in the activation vector need to be aligned again, the exponent alignment result can be directly retrieved. This reduces the number of exponent alignment operations performed by the exponent alignment unit in the floating-point addition module, thereby reducing computational power consumption.

[0099] Since the methods shown in FIG6 and FIG7 omit the exponential addition processing of the exponential addition submodule and the exponential alignment processing repeatedly performed by the exponent alignment submodule, the power consumption of the data operation unit can be reduced.

[0100] In the solution shown in the present application, since the data operation unit performs the vector inner product, the exponential addition processing of the exponential addition submodule and the exponential alignment result of the exponential alignment module can be omitted and reused. Therefore, as shown in Figure 8, if it is necessary to double the performance of the data operation unit, it is only necessary to increase the number of mantissa accumulation submodules and normalization submodules, without increasing the exponential addition submodule and exponential alignment submodule in the data operation unit. Therefore, by using the calculation method provided by the present application, the performance of the data operation unit can be improved at a lower hardware cost.

[0101] In one possible implementation, the weight vector and activation vector are a row or column of data included in the weight matrix and activation matrix. A matrix outer product is required between the weight matrix and the activation matrix, and the matrix multiplication is implemented by the outer product of the vectors.

[0102] Correspondingly, the data operation unit may include a floating-point multiplication module and a floating-point addition module. In this case, the processing of the above steps 302 and 303 can be replaced by: the data handling unit inputs the exponent bit and mantissa bit of each element in the accumulation matrix obtained in the vector outer product process into the floating-point addition module. The floating-point multiplication module performs mantissa multiplication on the mantissa bit of each element in the weight vector and the mantissa bit of each element in the activation vector to obtain the mantissa multiplication result. The floating-point addition module inputs the exponent bit of each element in the activation vector, the mantissa multiplication result, and the exponent bit and mantissa bit of each element in the accumulation matrix into the floating-point addition module to obtain the operation result.

[0103] The outer product operation of the weight matrix and the activation matrix includes multiplying each element of the weight vector by each element of the activation vector to obtain an intermediate matrix, and then adding the intermediate matrix to the accumulation matrix obtained during the outer product operation to obtain the updated accumulation matrix. Among them, the accumulation matrix is ​​initially a 0 matrix. As shown in Figure 9, each i element of the weight vector is multiplied by the jth element in the activation vector to obtain the element in the i-th row and j-th column of the intermediate matrix, and then the element in the i-th row and j-th column of the intermediate matrix can be added to the element in the i-th row and j-th column of the accumulation matrix. It can be seen that the outer product operation between the weight matrix and the activation matrix also requires floating-point multiplication and addition.

[0104] Figure 10 is a schematic diagram of a data operation unit performing an operation method provided by an embodiment of the present application. As shown in Figure 10, the data operation unit may include a floating-point multiplication module and a floating-point addition module, and the multiplication of the corresponding elements of the weight vector and the activation vector can be implemented by the floating-point multiplication module. Among them, the multiplication of floating-point numbers includes the addition of exponent bits and the multiplication of mantissa bits. The addition of exponent bits can be implemented by the exponent addition submodule in the floating-point multiplication module, and the multiplication of mantissa bits can be implemented by the mantissa multiplication submodule in the floating-point multiplication module.

[0105] In implementation, the data handling unit may input the mantissa bits of the corresponding elements in the activation vector and the weight vector into the mantissa multiplication submodule in the floating-point multiplication module for mantissa multiplication. That is, the mantissa bits of the floating-point elements in the activation vector and the mantissa bits of the integer elements in the weight vector are input into the mantissa multiplication submodule for mantissa multiplication to obtain a mantissa multiplication result. If the number of bits of the mantissa bits of the integer element is less than the number of bits required by the mantissa multiplication submodule, the mantissa bits of the integer element may be padded with 0.

[0106] Since the exponent in the weight vector is 0, that is, the value of the exponent bit is 0, the value of the exponent bit of the floating-point element in the activation vector and the exponent bit in the weight vector added together is still equal to the value of the exponent bit of the floating-point element in the activation vector. Therefore, in the operation method shown in Figure 10, similarly, a bypass function can be set, and the processing of the exponent addition submodule in the floating-point multiplication module is omitted, and a direct path can be set between the exponent alignment submodule in the data handling unit and the floating-point addition unit, and the data handling unit can directly input the exponent bit in the activation data into the exponent alignment submodule. Among them, the exponent bit in the activation data is equal to the exponential addition result of the exponent bit in the activation data and the exponent bit in the weight data. For the bypass function, the data operation unit can provide a configuration interface (configuration switch) for controlling the opening of the bypass function. In one example, the opening of the bypass function can be achieved by sending an asymmetric calculation instruction to the data operation unit and then realizing the opening of the bypass function through the configuration interface.

[0107] Continuing to refer to Figure 10, the addition of the floating-point multiplication results can be implemented by the floating-point addition module. Among them, the addition of floating-point numbers includes exponent alignment of the exponent bits of the activation data and the weight data, mantissa accumulation of the mantissa bits of the activation data and the weight data, and normalization of the exponent alignment results and mantissa accumulation. The exponent alignment can be implemented by the exponent alignment submodule in the floating-point addition module, the mantissa accumulation can be implemented by the mantissa accumulation submodule in the floating-point addition module, and the normalization processing is implemented by the normalization submodule in the floating-point addition module.

[0108] After the floating-point multiplication module obtains the exponential addition result and the mantissa multiplication result of the activation data and the weight data, the exponential addition result and the exponent bit of the corresponding element in the accumulation matrix can be input into the exponent alignment submodule for exponent alignment to obtain the exponent alignment result, and the corresponding mantissa multiplication result and the mantissa bit of the corresponding element in the accumulation matrix are input into the mantissa multiplication submodule for mantissa multiplication to obtain the mantissa multiplication result, and then the exponent alignment result and the mantissa multiplication result are input into the normalization submodule, and the normalization submodule performs normalization processing to obtain the exponent bit and mantissa bit of the corresponding element in the updated accumulation matrix.

[0109] Corresponding to the above-mentioned bypass function, the exponential addition result input to the exponent alignment submodule can be the exponent bit of the activation data input through the bypass function. In addition, the calculation of the sign bit of the activation data and the weight data is the same as that of the prior art and will not be repeated in the embodiments of the present application.

[0110] In the embodiments of the present application, on the one hand, floating-point operations can be implemented for integer weight data and floating-point activation data, which can reduce the impact of quantization compression on the accuracy of the AI ​​model. On the other hand, in the process of implementing the outer product operation of the weight vector and the activation vector, the processing of the exponential addition submodule in the floating-point multiplication module can be omitted through the bypass function, which can reduce the power consumption of the data operation unit.

[0111] The present application also provides a processor, which may be the processor shown in FIG3 , including a data handling unit and a data operation unit. The processor may be used to execute the operation method provided in the present application, including:

[0112] A data handling unit is used to obtain a quantized weight vector and an activation vector for performing vector operations with the weight vector, wherein the elements in the weight vector are integers and the elements in the activation vector are floating-point numbers.

[0113] The data handling unit is used to input the sign bit and mantissa bit of each element in the weight vector as the sign bit and mantissa bit of a floating-point number with an exponent of 0 into the data operation unit, and input each element in the activation vector into the data operation unit.

[0114] The data operation unit is used to perform vector operations on the input weight vector and activation vector to obtain the operation results.

[0115] In one implementation, the data operation unit includes a floating-point multiplication module and a floating-point addition module, and the vector operation is a vector inner product. The floating-point multiplication module is configured to perform a mantissa multiplication on the mantissa of each element in the weight vector with the mantissa of the corresponding element in the activation vector to obtain a mantissa multiplication result. The floating-point addition module is configured to perform a floating-point addition operation on the exponent of each element in the activation vector and the mantissa multiplication result to obtain a calculation result.

[0116] In one implementation, the data operation unit further includes an exponent alignment module, and the floating-point addition module includes a mantissa accumulation submodule and a normalization submodule. The exponent alignment module is configured to align the exponent bit of each element in the activation vector to obtain an exponent alignment result. The mantissa accumulation submodule is configured to accumulate the mantissa multiplication results to obtain a mantissa addition result. The normalization submodule is configured to normalize the exponent alignment result and the mantissa addition result to obtain a calculation result.

[0117] In one implementation, the floating-point addition module includes a plurality of mantissa accumulation submodules and a plurality of normalization submodules.

[0118] In one implementation, the data operation unit includes a floating-point multiplication module and a floating-point addition module, and the vector operation is a vector outer product. The data handling unit is used to input the exponent bit and mantissa bit of each element in the accumulation matrix obtained in the vector outer product process into the floating-point addition module. The floating-point multiplication module is used to perform mantissa multiplication on the mantissa bit of each element in the weight vector with the mantissa bit of each element in the activation vector to obtain the mantissa multiplication result. The floating-point addition module is used to input the exponent bit of each element in the activation vector, the mantissa multiplication result, and the exponent bit and mantissa bit of each element in the accumulation matrix into the floating-point addition module to obtain the operation result.

[0119] In one implementation, the data handling unit is configured to input the mantissa bit of each element in the weight vector as the sign bit and mantissa bit of a floating-point number with an exponent of 0, and input 0 as the exponent bit of the floating-point number with an exponent of 0 to the data operation unit.

[0120] In one implementation, the data handling unit is configured to input the sign bit and mantissa bit of each element in the weight vector as the sign bit and mantissa bit of an eight-bit floating point number with an exponent of 0 to the data operation unit.

[0121] In one implementation, when the elements in the weight vector are less than eight bits, the data handling unit is further configured to fill 0 between the sign bit and the mantissa bit of the elements in the weight vector; or fill 0 after the mantissa bit of the elements in the weight vector.

[0122] In one implementation, the data operation unit is used to perform vector operations on the input weight vector and obtain an intermediate result; perform floating-point multiplication operations on the intermediate result and the quantization coefficient to obtain an operation result, and the quantization coefficient is used to reduce the numerical error generated after the elements in the weight vector are treated as floating-point numbers.

[0123] In one implementation, the data operation unit is further configured to receive an asymmetric operation instruction.

[0124] The processor provided in the embodiment of the present application can implement floating-point operations using quantized integer weight vectors and floating-point activation vectors. Thus, on the one hand, when quantizing and compressing the AI ​​model, only the weight data can be quantized, which can reduce the impact of quantization compression on the accuracy of the AI ​​model. On the other hand, when performing floating-point operations on integers and floating-point numbers, there is no need to convert the integers into floating-point numbers, thereby improving the efficiency of floating-point operations.

[0125] The specific steps of the processor provided in the embodiment of the present application for executing the calculation method of integers and floating-point numbers are consistent with the execution steps of the calculation method provided in the above embodiment. For details, please refer to the content of the above embodiment, and will not be repeated in the embodiment of the present application.

[0126] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the computing method provided in the present application.

[0127] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the operation method provided in the embodiment of the present application.

[0128] In this application, the terms "first", "second", etc. are used to distinguish between identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first" and "second", nor is there a limit on quantity and execution order. It should also be understood that although the following description uses the terms "first", "second", etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. In this application, the term "at least one" means one or more, and the term "plurality" means two or more.

[0129] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A calculation method, characterized in that: The method is executed by a processor, wherein the processor includes a data handling unit and a data computing unit, and the method includes: The data handling unit obtains a quantized weight vector in the artificial intelligence AI model and an activation vector used to perform vector operations with the weight vector, wherein the elements in the weight vector are integers and the elements in the activation vector are floating-point numbers; The data handling unit inputs the sign bit and the mantissa bit of each element in the weight vector as the sign bit and the mantissa bit of the floating point number with an exponent of 0 to the data operation unit, and inputs each element in the activation vector to the data operation unit; The data operation unit performs vector operation on the input weight vector and the activation vector to obtain an operation result.

2. The method according to claim 1, characterized in that: The data operation unit includes a floating-point multiplication module and a floating-point addition module, the vector operation is a vector inner product, and the data operation unit performs a vector operation on the input weight vector and the activation vector to obtain an operation result, including: The floating-point multiplication module performs mantissa multiplication on the mantissa bits of each element in the weight vector and the mantissa bits of the corresponding elements in the activation vector to obtain a mantissa multiplication result; The floating-point addition module performs a floating-point addition operation on the multiplication result of the exponent bit of each element in the activation vector and the mantissa to obtain the operation result.

3. The method according to claim 2, characterized in that The data operation unit further includes an exponent alignment module, and the floating point addition module includes a mantissa accumulation submodule and a normalization submodule; The method further comprises: the exponent alignment module performs exponent alignment on the exponent bit of each element in the activation vector to obtain an exponent alignment result; The floating-point addition module performs a floating-point addition operation on the multiplication result of the exponent bit of each element in the activation vector and the mantissa to obtain the operation result, including: The mantissa accumulation submodule accumulates the mantissa multiplication results to obtain a mantissa addition result; The normalization submodule normalizes the exponent alignment result and the mantissa addition result to obtain a calculation result.

4. The method according to claim 3, characterized in that The floating point addition module includes a plurality of mantissa accumulation submodules and a plurality of normalization submodules.

5. The method according to claim 1, characterized in that The data operation unit includes a floating-point multiplication module and a floating-point addition module, and the vector operation is a vector outer product. The method further comprises: the data handling unit inputting the exponent bit and the mantissa bit of each element in the accumulation matrix obtained in the vector outer product process into the floating point addition module; The data operation unit performs vector operation on the input weight vector and the activation vector to obtain an operation result, including: The floating-point multiplication module performs mantissa multiplication on the mantissa bits of each element in the weight vector and the mantissa bits of each element in the activation vector to obtain a mantissa multiplication result; The floating-point addition module inputs the exponent bit of each element in the activation vector, the mantissa multiplication result, and the exponent bit and mantissa bit of each element in the accumulation matrix into the floating-point addition module to obtain the operation result.

6. The method according to claim 1, characterized in that The data handling unit inputs the sign bit and the mantissa bit of each element in the weight vector as the sign bit and the mantissa bit of a floating point number with an exponent of 0 to the data operation unit, including: The data handling unit uses the mantissa of each element in the weight vector as the sign bit and mantissa of a floating point number with an exponent of 0, and inputs 0 as the exponent of the floating point number with an exponent of 0 to the data operation unit.

7. The method according to any one of claims 1 to 6, characterized in that: The data handling unit inputs the sign bit and the mantissa bit of each element in the weight vector as the sign bit and the mantissa bit of a floating point number with an exponent of 0 to the data operation unit, including: The data handling unit inputs the sign bit and the mantissa bit of each element in the weight vector into the data operation unit as the sign bit and the mantissa bit of a floating point number with an exponent of 0 and eight bits.

8. The method according to claim 7, characterized in that In the case where the elements in the weight vector are less than eight bits, the method further comprises: Filling 0 between the sign bit and the mantissa bit of the elements in the weight vector; or, The tail digits of the elements in the weight vector are padded with 0.

9. The method according to any one of claims 1 to 8, characterized in that: The data operation unit performs vector operation on the input weight vector and the activation vector to obtain an operation result, including: The data operation unit performs vector operation on the input weight vector and the vector to obtain an intermediate result; The data operation unit performs a floating-point multiplication operation on the intermediate result and the quantization coefficient to obtain the operation result, and the quantization coefficient is used to reduce the numerical error generated when the elements in the weight vector are treated as floating-point numbers.

10. The method according to any one of claims 1 to 9, characterized in that: Before the data operation unit performs vector operation on the input weight vector and the activation vector, the data operation unit further includes: The data operation unit receives an asymmetric operation instruction corresponding to the weight vector and the activation vector.

11. A processor, characterized in that: The processor includes a data handling unit and a data computing unit, wherein: The data handling unit is used to obtain a quantized weight vector in the artificial intelligence AI model and an activation vector used to perform vector operations with the weight vector, wherein the elements in the weight vector are integers and the elements in the activation vector are floating-point numbers; The data handling unit is used to input the sign bit and the mantissa bit of each element in the weight vector as the sign bit and the mantissa bit of the floating point number with an exponent of 0 to the data operation unit, and input each element in the activation vector to the data operation unit; The data operation unit is used to perform vector operation on the input weight vector and the activation vector to obtain an operation result.

12. The processor according to claim 11, characterized in that The data operation unit includes a floating-point multiplication module and a floating-point addition module, and the vector operation is a vector inner product; The floating-point multiplication module is used to perform mantissa multiplication on the mantissa bits of each element in the weight vector and the mantissa bits of the corresponding elements in the activation vector to obtain a mantissa multiplication result; The floating-point addition module is used to perform a floating-point addition operation on the multiplication result of the exponent bit of each element in the activation vector and the mantissa to obtain the operation result.

13. The processor according to claim 12, characterized in that The data operation unit further includes an exponent alignment module, and the floating point addition module includes a mantissa accumulation submodule and a normalization submodule; The exponent alignment module is used to perform exponent alignment on the exponent bit of each element in the activation vector to obtain an exponent alignment result; The mantissa accumulation submodule is used to accumulate the mantissa multiplication results to obtain a mantissa addition result; The normalization submodule is used to perform normalization processing on the exponent alignment result and the mantissa addition result to obtain a calculation result.

14. The processor according to claim 13, characterized in that The floating point addition module includes a plurality of mantissa accumulation submodules and a plurality of normalization submodules.

15. The processor according to claim 11, characterized in that The data operation unit includes a floating-point multiplication module and a floating-point addition module, and the vector operation is a vector outer product; The data handling unit is used to input the exponent bit and the mantissa bit of each element in the accumulation matrix obtained in the vector outer product process into the floating point addition module; The floating-point multiplication module is used to compare the mantissa of each element in the weight vector with the mantissa of each element in the activation vector. The mantissas are multiplied to obtain the mantissa multiplication result; The floating-point addition module is used to input the exponent bit of each element in the activation vector, the mantissa multiplication result, and the exponent bit and mantissa bit of each element in the accumulation matrix into the floating-point addition module to obtain the operation result.

16. The processor according to claim 11, characterized in that The data handling unit is used for the data handling unit to use the mantissa bits of each element in the weight vector as the sign bit and mantissa bits of a floating point number with an exponent of 0, and to input 0 as the exponent bit of the floating point number with an exponent of 0 to the data calculation unit.

17. The processor according to any one of claims 11 to 16, characterized in that: The data handling unit is used to input the sign bit and the mantissa bit of each element in the weight vector into the data operation unit as the sign bit and the mantissa bit of a floating point number with an exponent of 0 and eight bits.

18. The processor according to claim 17, characterized in that In the case where the elements in the weight vector are less than eight bits, the method further includes: Filling 0 between the sign bit and the mantissa bit of the elements in the weight vector; or, The tail digits of the elements in the weight vector are padded with 0.

19. The processor according to any one of claims 11 to 18, characterized in that: The data operation unit is used to perform vector operation on the input weight vector and the vector to obtain an intermediate result; The data operation unit is used to perform a floating-point multiplication operation on the intermediate result and the quantization coefficient to obtain the operation result, and the quantization coefficient is used to reduce the numerical error generated when the elements in the weight vector are treated as floating-point numbers.

20. The processor according to any one of claims 11 to 19, characterized in that: The data operation unit is further used to receive asymmetric operation instructions corresponding to the weight vector and the activation vector.

21. A computing device, characterized in that The computing device comprises a memory and a processor as claimed in any one of claims 10 to 18 above.

Images