Processor, floating-point unit, and operation method

Abstract

The embodiments disclosed in the present application belong to the field of computing technology, and particularly relate to a processor, a floating-point operation unit and an operation method. The processor includes a data handling unit and a floating-point operation unit. In addition to the data handling module that realizes the data handling function, the data handling unit also includes a data decompression module. Among them, the data decompression module can decompress the first weight data that has been compressed in the AI ​​model to obtain the second weight data represented by a floating-point number. The data handling unit can transport the first activation data and the second weight data to the floating-point operation unit for floating-point operations. By adopting the present application, floating-point operations can be performed on the compressed first weight data and the uncompressed first activation data. In this way, when the AI ​​model is quantized and compressed, only the weight data can be compressed, which can reduce the impact of quantization compression on the accuracy of the AI ​​model.

Description

[0001] This application is a divisional application. The application number of the original application is 202311086354.8, and the original application date is August 24, 2023. The entire content of the original application is incorporated into this application by reference. Technical Field

[0002] The present application relates to the field of computing technology, and in particular to a processor, a floating-point operation unit, and an operation method. Background Art

[0003] Quantization and compression of artificial intelligence (AI) models refers to a method of quantizing the weight data and activation data in the AI ​​model from floating-point numbers to integers, thereby improving the running speed of the AI ​​model.

[0004] While quantization compression can increase the speed of AI models, it also reduces their accuracy. Because activation data has a wider range of values ​​and contains a certain proportion of outliers, the impact of activation data quantization on AI model accuracy is greater than that of weight data quantization. Summary of the Invention

[0005] The embodiments of the present application provide a processor, a floating-point operation unit, and an operation method, which can improve the computational efficiency of calculating activation data and weight data and reduce the impact of quantization compression on the accuracy of AI models. The corresponding technical solutions are as follows:

[0006] In a first aspect, a processor is provided, the processor including a data handling unit and a floating-point operation unit. The data handling unit implements a data handling function for the processor, including a data handling module and a data decompression module, wherein:

[0007] The data handling module is configured to obtain first weight data and first activation data of the artificial intelligence (AI) model, wherein the first weight data is obtained by compressing the original weight data in the AI ​​model. The data decompression module is configured to decompress the first weight data to obtain second weight data represented by floating-point numbers. The data handling module is further configured to input the first activation data and the second weight data into a floating-point arithmetic unit for floating-point operations.

[0008] In the solution shown in the present application, when the data handling module is carrying the first weight data and the first activation data to the floating-point operation unit, the data decompression module added in the data handling unit can decompress the compressed first weight data to obtain the second weight data represented by a floating-point number, and then the data handling module can input the second weight data and the uncompressed first activation data into the floating-point operation unit to realize floating-point operations of the weight data and the activation data.

[0009] In this way, on the one hand, only the weight data in the AI ​​model is compressed, which can avoid the impact of the quantization of the activation data on the accuracy of the AI ​​model. On the other hand, adding the data decompression module to the data handling unit avoids the data handling unit from transporting the first weight data to other computing units (such as vector computing cores) for decompression processing, which can improve the efficiency of decompressing the first weight data, and thus improve the efficiency of the processor in performing floating-point operations on the first activation data and the second weight data.

[0010] In one achievable manner, the compression process includes at least one of encoding compression and quantization compression. When the compression process includes encoding compression, the decompression process includes decoding. When the compression process includes quantization compression, the decompression process includes numerical conversion.

[0011] In the solution shown in this application, the compression process can be encoding and compressing the original weight data in the AI ​​model. For example, the original weight data of FP16 can be encoded into a block floating point format to obtain the first weight data, which can reduce the storage space occupied by the weight data in the AI ​​model. Accordingly, when the compression process is encoding compression, the corresponding decompression process is decoding processing, that is, decoding the encoded first weight data into second weight data represented by floating point numbers.

[0012] In the solution shown in this application, the compression process can also be to quantize and compress the original weight data in the AI ​​model. For example, the original weight data of FP16 can be quantized into the first weight data of INT8, which can reduce the storage space occupied by the weight data in the AI ​​model. Accordingly, when the compression process is quantization compression, the corresponding decompression process is numerical conversion, that is, converting the first weight data into second weight data represented by floating-point numbers.

[0013] In one implementation, the data handling module includes a data memory and a handling controller. The handling controller is configured to store the first weight data in the GM in the data memory, and then input the data memory into the data decompression module. After the data decompression module completes the decompression process, the second weight data is input into the floating-point arithmetic unit.

[0014] In the solution shown in the present application, the transport controller may input the first weight data stored in the data memory into the data decompression module before inputting the first weight data into the floating-point operation unit. After the data decoding module decompresses the first weight data into the second weight data, the transport controller may input the second weight data into the floating-point operation unit. In this way, the decompression processing of the first weight data can be implemented in the data transport unit, and there is no need to transport the first weight data to other operation units for decompression processing, which can improve the efficiency of decompression processing of the first weight data.

[0015] In one implementation, the data handling module includes a data memory and a handling controller. The handling controller is configured to input the first weight data in the GM into the data decompression module. After the data decompression module completes the decompression process, the second weight data is stored in the data memory, and then the second weight data stored in the data memory is input into the floating-point arithmetic unit.

[0016] In the solution shown in the present application, before the transport controller transports the first weight data stored in the GM to the data storage device, it can first input the first weight data into the data decompression module, and the data decoding module decompresses the first weight data into the second weight data. The transport controller can store the second weight data in the data storage device, and then input the second weight data in the data storage device into the floating-point operation unit. In this way, the decompression processing of the first weight data can be implemented in the data transport unit, and there is no need to transport the first weight data to other operation units for decompression processing, which can improve the efficiency of decompression processing of the first weight data.

[0017] In an implementable manner, the data transport unit is a direct memory access (DMA) or a tensor memory access (TMA).

[0018] In one implementable manner, the floating-point operation unit is a matrix operation core.

[0019] In a second aspect, a floating-point arithmetic unit is provided. The floating-point arithmetic unit includes a data cache module, a data decompression module, and a data calculation module, wherein: the data cache module is used to cache first weight data and first activation data of an artificial intelligence (AI) model, wherein the first weight data is obtained by compressing the original weight data in the AI ​​model. The data decompression module is used to decompress the first weight data to obtain second weight data represented by a floating-point number. The data calculation module is used to perform floating-point operations on the first activation data and the second weight data to obtain a calculation result.

[0020] In the solution described in this application, the floating-point operation unit can be a unit in the processor that performs floating-point operations on the weight data and activation data of the AI ​​model. A data decompression module is added to the floating-point operation unit. In this way, the compressed first weight data can be decompressed in the floating-point operation unit to obtain second weight data represented by floating-point numbers. The floating-point operation unit can then perform floating-point operations on the first activation data and the second weight data.

[0021] In this way, on the one hand, only the weight data in the AI ​​model is compressed, which can avoid the impact of the quantization of the activation data on the accuracy of the AI ​​model. On the other hand, adding the data decompression module to the floating-point operation unit avoids the need to move the first weight data to other operation units (such as vector operation cores) for decompression, which can improve the efficiency of decompressing the first weight data, and thus improve the efficiency of floating-point operations on the first activation data and the second weight data.

[0022] In one achievable manner, the compression process includes at least one of encoding compression and quantization compression. When the compression process includes encoding compression, the decompression process includes decoding. When the compression process includes quantization compression, the decompression process includes numerical conversion.

[0023] In the solution shown in this application, the compression process can be encoding and compressing the original weight data in the AI ​​model. For example, the original weight data of FP16 can be encoded into a block floating point format to obtain the first weight data, which can reduce the storage space occupied by the weight data in the AI ​​model. Accordingly, when the compression process is encoding compression, the corresponding decompression process is decoding processing, that is, decoding the encoded first weight data into second weight data represented by floating point numbers.

[0024] In the solution shown in this application, the compression process can also be to quantize and compress the original weight data in the AI ​​model. For example, the original weight data of FP16 can be quantized into the first weight data of INT8, which can reduce the storage space occupied by the weight data in the AI ​​model. Accordingly, when the compression process is quantization compression, the corresponding decompression process is numerical conversion, that is, converting the first weight data into second weight data represented by floating-point numbers.

[0025] In one implementation, the data cache module includes a core cache, an activation matrix register, and a weight matrix register. The core cache is configured to cache first weight data and first activation data, input the first weight data into the weight matrix register, and input the first activation data into the activation matrix register. The weight matrix register is configured to input the first weight data into the data decompression module. The activation matrix register is configured to input the first activation data into the data calculation module.

[0026] In the scheme shown in the present application, the first weight data and the first activation data can form a matrix to perform matrix operations on floating-point numbers. The data cache module can first cache the first weight data and the first activation data, and then the first weight data can be input into the weight matrix register, and the first activation data can be input into the activation matrix register. Before the first weight data in the weight matrix register is input into the data calculation module, the first weight data can be input into the data decompression module for decompression processing to obtain the second weight data. In this way, the data calculation module can perform matrix operations on the second weight data and the first activation data. In this way, the data decompression module is added to the floating-point operation unit to avoid transporting the first weight data to other operation units (such as vector operation cores) for decompression processing, which can improve the efficiency of decompressing the first weight data, thereby improving the efficiency of floating-point operations on the first activation data and the second weight data.

[0027] In one implementable manner, the floating-point operation unit includes multiple data decompression modules.

[0028] In the solution shown in the present application, since the first weight data is over-compressed, the data cache module can store more first weight data relative to the first activation data. By providing multiple data decompression modules to perform over-compression processing on the first weight data in the data cache module in parallel, the efficiency of decompressing the first weight data can be improved.

[0029] In one implementable manner, the floating-point operation unit is a matrix operation core.

[0030] In a third aspect, a calculation method is provided, which is executed by a processor, the processor including a data handling unit and a floating-point operation unit, the data handling unit implementing a data handling function for the processor, including a data handling module and a data decompression module. The calculation method includes:

[0031] The data handling module obtains first weight data and first activation data of the artificial intelligence (AI) model, where the first weight data is obtained by compressing the original weight data in the AI ​​model. The data decompression module decompresses the first weight data to obtain second weight data represented by floating-point numbers. The data handling module inputs the first activation data and the second weight data into the floating-point arithmetic unit for floating-point operations.

[0032] In one achievable manner, the compression process includes at least one of encoding compression and quantization compression. When the compression process includes encoding compression, the decompression process includes decoding. When the compression process includes quantization compression, the decompression process includes numerical conversion.

[0033] In one achievable manner, the compression process includes encoding the original weight data into first weight data represented by block floating point numbers, and the decoding process includes decoding the first weight data into second weight data represented by floating point numbers;

[0034] The quantization compression includes quantizing the original weight data into first weight data represented by integer numbers, and the numerical conversion includes converting the first weight data into second weight data represented by floating-point numbers.

[0035] In one achievable embodiment, the data handling module includes a data storage device and a handling controller. The data handling module obtains the first weight data and the first activation data of the artificial intelligence (AI) model by: the handling controller stores the first weight data in the GM into the data storage device, and then inputs the data storage device into the data decompression module.

[0036] The data transport module inputs the first activation data and the second weight data into the floating-point operation unit, including: the transport controller inputs the second weight data into the floating-point operation unit after the data decompression module completes the decompression processing.

[0037] In one feasible manner, the data transport module includes a data storage device and a transport controller; the above-mentioned data transport module obtains the first weight data and the first activation data of the artificial intelligence AI model, including: the transport controller inputs the first weight data in the GM into the data decompression module.

[0038] The above-mentioned data handling module inputs the first activation data and the second weight data into the floating-point operation unit, including: after the data decompression module completes the decompression processing, the handling controller stores the second weight data into the data storage device, and inputs the second weight data stored in the data storage device into the floating-point operation unit.

[0039] In an implementable manner, the data transport unit is a direct memory access (DMA) or a tensor memory access (TMA).

[0040] In one implementable manner, the floating-point operation unit is a matrix operation core.

[0041] In a fourth aspect, an operation method is provided, which is executed by a floating-point operation unit, and the floating-point operation unit includes a data cache module, a data decompression module, and a data calculation module. The operation method includes: the data cache module caches the first weight data and the first activation data of the artificial intelligence AI model, wherein the first weight data is obtained by compressing the original weight data in the AI ​​model. The data decompression module decompresses the first weight data to obtain the second weight data represented by a floating-point number. The data calculation module performs floating-point operations on the first activation data and the second weight data to obtain an operation result.

[0042] In one achievable manner, the compression process includes at least one of coding compression and quantization compression. When the compression process includes coding compression, the decompression process includes decoding processing; when the compression process includes quantization compression, the decompression process includes numerical conversion.

[0043] In one achievable manner, the compression process includes encoding original weight data into first weight data represented by block floating point numbers, and the decoding process includes decoding the first weight data into second weight data represented by floating point numbers.

[0044] The quantization compression includes quantizing the original weight data into first weight data represented by integer numbers, and the numerical conversion includes converting the first weight data into second weight data represented by floating-point numbers.

[0045] In one achievable embodiment, the data cache module includes a core cache, an activation matrix register, and a weight matrix register. The data cache module caches the first weight data and the first activation data of the artificial intelligence (AI) model, including: a core cache for caching the first weight data and the first activation data, inputting the first weight data into the weight matrix register, and inputting the first activation data into the activation matrix register. The weight matrix register inputs the first weight data into the data decompression module. The activation matrix register is used to input the first activation data into the data calculation module.

[0046] In one implementable manner, the floating-point operation unit includes a plurality of data decompression modules, and the floating-point operation unit is a matrix operation core.

[0047] In a fifth aspect, a computing device is provided, which includes a memory and a processor as described in the first aspect above, and the processor can be used to implement the calculation method described in the third aspect above.

[0048] In a sixth aspect, a computing device is provided, comprising a memory and a floating-point operation unit as described in the second aspect above, wherein the floating-point operation unit can be used to implement the operation method described in the fourth aspect above.

[0049] In the seventh aspect, a computer-readable storage medium is provided, which stores a computer program code. When the computer program code is executed by a computing device, the computing device executes the operation method described in the third aspect or the fourth aspect above.

[0050] In an eighth aspect, a computer program product comprising instructions is provided, which, when run on a computing device, enables the computing device to execute the operation method described in the third or fourth aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flowchart of the operation of weight data and activation data in related technologies;

[0052] Figure 2 is a structural diagram of a computing device provided in an embodiment of the present application;

[0053] Figure 3 This is a schematic diagram of a processor structure provided by an embodiment of the present application;

[0054] Figure 4 This is a flowchart of a calculation method provided in an embodiment of the present application;

[0055] Figure 5 This is a structural diagram of a data handling unit provided in an embodiment of the present application;

[0056] Figure 6 This is a structural diagram of a data handling unit provided in an embodiment of the present application;

[0057] Figure 7 is a schematic diagram of a method for executing a calculation on a computing device provided by an embodiment of the present application;

[0058] Figure 8 This is a schematic diagram of the structure of a floating-point arithmetic unit provided in an embodiment of the present application;

[0059] Figure 9 This is a flowchart of a calculation method provided in an embodiment of the present application;

[0060] Figure 10 This is a schematic diagram of a method for executing a calculation on a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0062] At the beginning of the design of the Artificial Intelligence (AI) model, in order to improve the accuracy of the AI ​​model, the weight data and activation data will be set to higher-precision floating-point numbers, such as FP32 and FP16. Floating-point numbers occupy more bits, such as FP32 floating-point numbers occupy 32 bits, and FP16 floating-point numbers occupy 16 bits. Therefore, floating-point numbers require more storage and computing resources when stored and calculated. Especially in some large models, such as large language models (LLM), the amount of data corresponding to the weight data is huge, and the storage of weight data alone requires hundreds of GB of storage space. Among them:

[0063] The AI ​​model can be any trained AI model, such as a large language model, an image recognition model, etc.

[0064] Weight data includes the weights of the AI ​​model after training. During the actual operation of the AI ​​model, weight data is generally represented in the form of vectors or matrices.

[0065] Activation data can be the output data of the AI ​​model's intermediate layers, related to the sample data input to the AI ​​model. During the actual operation of the AI ​​model, activation data can also be represented in the form of vectors or matrices, and often performs vector or matrix operations with weight data.

[0066] To improve the universality of AI models, quantization compression is used to reduce the storage space required to store the models and increase their speed. Quantization compression of AI models involves quantizing the weight and activation data in the AI ​​model from floating-point numbers to integers, thereby increasing the model's speed. For example, the original FP16 weight and activation data in the AI ​​model can be quantized to INT8. This reduces the storage of the weight data from 16 bits to 8 bits.

[0067] However, converting floating-point numbers to integers reduces their precision. The larger the floating-point range, the greater the loss in precision when converting them to integers. Therefore, quantization compression significantly impacts the accuracy of AI models.

[0068] Since in AI models, the numerical range of activation data is generally larger than that of weight data, and there is a certain proportion of outliers in the activation data, the error caused by the quantization of activation data is greater, and the impact on the accuracy of the AI ​​model is greater than the quantization of weight data. In addition, in most AI models, the amount of weight data is much larger than the amount of activation data. Therefore, quantizing the weight data can bring better quantization effects to the AI ​​model, and the impact on the accuracy of the AI ​​model is less than the quantization of the activation data. Therefore, in order to ensure the accuracy of the AI ​​model in related technologies, asymmetric quantization can be used, that is, only the weight data is quantized, and the activation data is not quantized.

[0069] Figure 1 This is a flowchart of the processor operating on quantized weight data and unquantized activation data when asymmetric quantization is used in related technologies. The processor includes a matrix operation core, a vector operation core, and a data handling unit. The data handling unit can be a direct memory access (DMA) or a tensor memory access (TMA). Figure 1As shown, the data handling unit is used to implement the data handling function for the processor. The operation process includes:

[0070] S1. The data handling unit can write activation data stored in the global memory (GM) to the local buffer of the matrix operation core. The activation data is represented by floating point numbers, such as FP16.

[0071] S2. The data handling unit can write the weight data stored in the GM to the local buffer of the vector operation core. The weight data can be the weight data represented by integers after quantizing the original weight data in the AI ​​model, such as the weight data represented by INT4.

[0072] S3. The vector calculator (vector calculation unit) in the vector operation core may convert the weight data in the first cache from integers to floating-point numbers, and write the converted floating-point numbers back to the local buffer of the vector operation core.

[0073] S4. The data transfer unit can transfer the weight data that has completed the numerical conversion in the local buffer of the vector operation core, that is, the weight data represented by floating-point numbers, to the local buffer of the vector operation core (the specific transfer process is not shown in the figure).

[0074] S5. The local buffer of the matrix operation core transfers the activation data and weight data represented by floating-point numbers to two matrix registers respectively.

[0075] S6. Input the activation data and weight data cached in the matrix register into the matrix calculator (matrix calculation unit), perform corresponding matrix operations, and write the obtained operation results into the result register.

[0076] S7. The data handling unit writes the operation result stored in the result register into GM to complete the operation.

[0077] Depend on Figure 1 As can be seen, operations on floating-point activation data and integer weight data require the vector cores to convert the integer weight data into floating-point weight data. Furthermore, data transfer units are required to transfer the floating-point weight data across cores, moving the floating-point weight data from the vector cores to the matrix cores. This consumes significant computing resources and bandwidth, resulting in low computational efficiency.

[0078] The computing method provided in the embodiment of the present application can avoid converting integer weight data into floating-point weight data through the vector operation core, and further can avoid cross-core data transfer through the data transfer unit, thereby reducing the occupation of computing resources and bandwidth resources and improving computing efficiency. The computing method provided in the embodiment of the present application is implemented in a computing device. Figure 2 This is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. Figure 2 As shown, the computing device 200 may include: a bus 202, a processor 204, a memory 206, and optionally the computing device 200 may further include a communication interface 208. The processor 204, the memory 206 and the communication interface 208 communicate with each other via the bus 202. The computing device 200 may be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 200. The computing device 200 may be a device for running a model, a terminal or a server. When the computing device 200 is a terminal, the computing device 200 includes but is not limited to a desktop computer, a mobile phone, a notebook, a tablet computer, etc. When the computing device 200 is a server, the computing device 200 may be a separate server, or a server cluster consisting of multiple servers, or a physical machine, or a virtual machine or container virtualized by virtualization technology.

[0079] The bus 202 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 The fact that only one line is used in the figure does not mean that there is only one bus or only one type of bus. Bus 202 may include a path for transmitting information between various components of computing device 200 (eg, memory 206, processor 204, communication interface 208).

[0080] The processor 204 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP). The processor 204 may further include a floating-point operation unit (FPU) and a data handling unit, and the floating-point operation unit may be a matrix operation core.

[0081] Memory 206 may include volatile memory, such as random access memory (RAM). Memory 206 may also include non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid state drive (SSD). All of the above memories are referred to as global memory.

[0082] The communication interface 208 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 200 and other devices or a communication network.

[0083] Figure 3 This is a schematic diagram of a processor structure provided by an embodiment of the present application. The processor may be the above-mentioned Figure 2 The processor 204 in the computing device, such as Figure 3 As shown, the processor includes a data handling unit and a floating-point operation unit, wherein the data handling unit includes a data handling module and a data decompression module. The processor can execute the operation method provided in the embodiment of the present application. Figure 4 This is a flowchart of another calculation method provided in the embodiment of the present application, see Figure 4 , the operation method executed by the processor includes:

[0084] Step 401: The data handling module obtains first weight data and first activation data of the AI ​​model, wherein the first weight data is obtained by compressing the original weight data in the AI ​​model.

[0085] During implementation, the data handling module can read the first weight data and first activation data of the AI ​​model from the GM, wherein the first weight data is obtained by compressing the original weight data in the AI ​​model, and the first activation data is a floating point number, which can be the activation data in the AI ​​model that has not been compressed.

[0086] In an embodiment of the present application, the compression process may include coding compression and quantization compression. Coding compression refers to encoding the original weight data into a specified coding format to reduce the storage space required to store the weight data of the AI ​​model. The coding format may be a block floating point number (Block Float Point) format, or it may be an 8-bit floating point number (8bitEfloat) coding format. Quantization compression refers to quantizing the original weight data from a floating point number to an integer number. For example, the original weight data of FP16 can be quantized into the first weight data of INT4. In an achievable manner, the first activation data may also be the weight data obtained by quantizing and compressing the original weight data and then coding and compressing it again. The compression process for the first weight data belongs to the prior art and can be performed before running the AI ​​model. It will not be described in detail in the embodiment of the present application.

[0087] Step 402: The data decompression module decompresses the first weight data to obtain second weight data represented by floating-point numbers.

[0088] During implementation, after the data handling module reads the first weight data and the first activation data of the AI ​​model from the GM, the data handling module can input the first weight data into the data decompression module for decompression processing to obtain the second weight data represented by floating-point numbers.

[0089] Corresponding to the compression process in step 401 including encoding compression and quantization compression, the decompression process in step 402 includes decoding processing and numerical conversion processing. When the compression process includes encoding compression, the decompression process includes decoding processing, that is, decoding the encoded first weight data into a floating point number. When the compression process includes quantization compression, the decompression process includes numerical conversion, that is, converting the quantized compressed first weight data into a floating point number.

[0090] In one example, when the compression process is encoding compression, the data decompression module is used to decode the first weight data, for example, decoding the first weight data encoded in block floating point format into floating point numbers in FP16 format to obtain second weight data. When the compression process is quantization compression, the data decompression module is used to perform numerical conversion on the first weight data, for example, converting the first weight data represented by INT4 into the second weight data represented by FP16.

[0091] In another example, the compression process is to first quantize and compress the original weight data, and then encode and compress the quantized and compressed weight data. For example, the compression process may be to first quantize the original weight data passed through FP16 into weight data represented by INT8, and then encode and compress the weight data represented by INT8. In this case, the data decompression module may include a data decoding submodule and a numerical conversion submodule. Among them, the data decoding submodule may first decode the first weight data obtained by the data handling module, and then the numerical conversion submodule may perform numerical conversion on the decoded weight data to obtain the second weight data represented by FP16.

[0092] Among them, the data decompression module is a logic circuit added to the data handling unit. Depending on the different decoding processes, technicians can add different logic circuits in the data handling unit to realize the decompression process of the first weight data. For example, adding a logic circuit to convert data from INT4 to FP16, or adding a logic circuit to decode data in block floating point format or 8bit Efloat encoding format into data represented by FP16, etc. In addition, during the execution of the embodiment of the present application, the software interface can be configured to indicate whether the data decoding module in the data handling unit needs to perform data decoding on the first weight data.

[0093] Step 403: The data handling module inputs the first activation data and the second weight data into the floating-point operation unit to perform floating-point operation.

[0094] In implementation, after the data decompression module decompresses the first weight data to obtain the second weight data, the data handling unit can store the second weight data and the first activation data into the registers in the floating-point operation unit, so that the floating-point operation unit performs floating-point operations on the second weight data and the first activation data.

[0095] A direct path may be provided between the data decompression module and the data handling module, so as to realize data transmission between the data decompression module and the data handling module. Figure 5 and Figure 6 Schematic diagrams of the structures of two data handling units provided in the embodiments of the present application. Figure 5 or Figure 6 As shown, the data handling module can be further divided into a handling controller and a data memory. The handling controller is used to control the handling of data between memory and / or registers, while the data memory is used to store the handled data. For example, the process of transferring data from memory to registers may include the handling controller reading the data from memory and writing the read data to the data memory. The handling controller then reads the data from the data memory and writes the read data to the register in the processor.

[0096] like Figure 5 As shown, the decompression processing of the first weight data inputted by the data handling module to the data decompression module in step 402 may include: the handling controller storing the first weight data stored in the GM to the data memory, and then inputting the data memory to the data decompression module. After the data decompression module completes the decompression processing, the handling controller then inputs the second weight data to the floating-point operation unit.

[0097] In implementation, after the transport controller reads the first weight data and the first activation data to be calculated from the GM, it can store the first weight data and the first activation data in a data memory, and then input the first weight data stored in the data memory into a data decompression module, which decompresses the first weight data to obtain second weight data represented by a floating-point number. After the data decompression module outputs the second weight data, the transport controller can store the second weight data and the first activation data in a register in a floating-point operation unit, so that the floating-point operation unit performs floating-point operations on the second weight data and the first activation data.

[0098] like Figure 6 As shown, the decompression processing of the first weight data inputted by the data transport module to the data decompression module in the above step 402 may also include: the transport controller first inputs the first weight data stored in the GM to the data decompression module, and after the data decompression module completes the decompression processing, the second weight data may be stored in the data memory, and then the transport controller may input the second weight data stored in the data memory into the floating-point operation unit.

[0099] In implementation, after the transport controller reads the first weight data and the first activation data to be calculated from the GM, it can input the first weight data into the data decompression module and store the first activation data into the data memory. After the data decompression module decompresses the first weight data to obtain second weight data represented by floating-point numbers, the transport controller can store the second weight data and the first activation data into registers in the floating-point operation unit, so that the floating-point operation unit performs floating-point operations on the second weight data and the first activation data.

[0100] exist Figure 5 and Figure 6 In the data handling unit shown, there is a direct path between the data storage device and the data decompression module, and the transmission rate of the first weight data between the data storage device and the data decompression module is much greater than the transmission rate of the first weight data between the matrix operation core and the vector operation core.

[0101] In an embodiment of the present application, a data decoding module is added to the data handling unit. In this way, the data handling unit can decompress the first weight data in the process of carrying the first weight data to obtain the second weight data represented by a floating point number. In this way, on the one hand, only the weight data in the AI ​​model is compressed, and the impact of quantization compression on the accuracy of the AI ​​model can be reduced while the AI ​​model is quantized and compressed. On the other hand, adding the data decompression module to the data handling unit avoids the data handling unit from carrying the first weight data to the vector operation core for decompression, which can improve the efficiency of decompression of the first weight data, and thus improve the efficiency of floating-point operations on the first activation data and the second weight data. On the other hand, compared with the related art, in addition to reducing the storage space of weight data through quantization compression in the embodiment of the present application, the storage space of weight data can also be reduced by encoding and compressing the weight data.

[0102] Figure 7 This is a schematic diagram of a method for performing operations on a computing device provided by an embodiment of the present application. Figure 7 The computing device shown includes a global memory and a processor, which may be the above-mentioned Figure 3 The processor shown in FIG. 1 may have a floating point operation unit that can be a matrix operation core. Figure 7 As shown, the matrix operation core includes at least a core buffer, a matrix register, a result buffer, and a matrix calculator. Among them, the core buffer is used to cache the first activation data and the second weight data written by the data handling unit, and the matrix register is used to store the first activation data and the second weight data in the form of matrices, respectively, as input matrices of the matrix calculator. The matrix calculator is capable of realizing matrix outer product or matrix inner product, and is used to perform floating-point matrix operations on the first activation data and the second weight data. The result buffer is used to cache the result data obtained after the floating-point matrix operation on the first activation data and the second weight data.

[0103] like Figure 7 As shown, the data transport unit obtains the first activation data and the first weight data from the GM, and decodes the first weight data into the second weight data, and then transports the first activation data and the second weight data to the matrix operation core buffer.

[0104] In the matrix operation core, the matrix operation core can store the first activation data in the form of a vector to a matrix register according to a pre-set tiling algorithm for performing matrix operations on the first activation data and the second weight data. The matrix register can be called an activation matrix register or a left matrix register, and the second weight data can be stored in the form of a vector to another matrix register. The other matrix register can be called a weight matrix register or a right matrix register.

[0105] The first activation data and second weight data stored in the activation matrix register and the weight matrix register can be input into the matrix calculator to perform floating-point matrix operations. For example, the matrix calculator can perform a matrix inner product operation on the first activation data and the second weight data. After the matrix calculator obtains the matrix operation result, it can store the matrix operation result in the result register. The data transfer unit can then transfer the matrix operation result in the result register to the GM.

[0106] It can be seen that the use of the calculation method provided in the embodiment of the present application can avoid the data transfer unit from transferring the first weight data to the vector operation core for decompression processing, avoid the cross-core transmission of the decompressed second weight data, and improve the efficiency of floating-point operations on the first activation data and the second weight data.

[0107] Figure 8 This is a structural diagram of a floating-point arithmetic unit provided in an embodiment of the present application. The floating-point arithmetic unit may be the above-mentioned Figure 2 The processor 204 in the computing device shown includes a floating point unit. Figure 8 As shown, the floating-point operation unit includes a data cache module, a data decompression module and a data calculation module, wherein the data cache module can be a register in the floating-point operation unit, which is used to cache the first activation data and the first weight data to be calculated in the AI ​​model. The data decompression module is used to decompress the first weight data to obtain the second weight data. The data calculation module is used to perform a floating-point matrix multiplication operation on the first activation data and the second weight data. The floating-point operation unit can implement another operation method provided in an embodiment of the present application.

[0108] Figure 9 This is a flowchart of a calculation method provided in an embodiment of the present application, see Figure 9 , the operation method performed by the floating-point operation unit includes:

[0109] Step 901: The data cache module caches the first weight data and first activation data of the AI ​​model, wherein the first weight data is obtained by compressing the original weight data in the AI ​​model.

[0110] In implementation, the data handling module can read the first weight data and first activation data of the AI ​​model from the GM, and then input the read first weight data and first activation data into the data cache module in the floating-point operation unit. The first weight data is obtained by compressing the original weight data in the AI ​​model, and the first activation data is a floating-point number, which can be the activation data in the AI ​​model that has not been compressed.

[0111] In an embodiment of the present application, the compression process may include encoding compression and quantization compression. Encoding compression refers to encoding the original weight data into a specified encoding format to reduce the storage space required to store the weight data of the AI ​​model. The encoding format may be a block floating point format, or may be an 8-bit Efloat encoding format. Quantization compression refers to quantizing the original weight data from a floating point number to an integer. For example, the original weight data of FP16 can be quantized into the first weight data of INT4. In one feasible manner, the first activation data may also be the weight data obtained by quantizing and compressing the original weight data and then encoding and compressing it again.

[0112] Step 902: The data decompression module decompresses the first weight data to obtain second weight data represented by floating-point numbers.

[0113] During implementation, the data handling module may input the cached first weight data into the data decompression module, and the data decompression module may decompress the first weight data to obtain the second weight data represented by floating-point numbers.

[0114] Corresponding to the compression process in step 901 including encoding compression and quantization compression, the decompression process in step 902 includes decoding processing and numerical conversion processing. When the compression process includes encoding compression, the decompression process includes decoding processing, that is, decoding the encoded first weight data into a floating point number. When the compression process includes quantization compression, the decompression process includes numerical conversion, that is, converting the quantized compressed first weight data into a floating point number.

[0115] Step 903: The data calculation module performs floating-point operations on the first activation data and the second weight data to obtain a calculation result.

[0116] During implementation, the data cache module performs floating-point operations on the first activation data in the data cache module and the second weight data obtained after decompression processing by the data decompression module to obtain an operation result.

[0117] In an embodiment of the present application, a data decompression module is added to the floating-point operation unit, so that the compressed first weight data can be directly decompressed in the floating-point operation unit to obtain the second weight data represented by a floating-point number, and then the floating-point operation unit can perform floating-point operations on the first activation data and the second weight data. In this way, on the one hand, only the weight data in the AI ​​model is compressed, which can avoid the impact of the quantization of the activation data on the accuracy of the AI ​​model. On the other hand, adding the data decompression module to the floating-point operation unit avoids the need to transport the first weight data to the vector operation core for decompression, which can improve the efficiency of decompressing the first weight data, and thus improve the efficiency of floating-point operations on the first activation data and the second weight data. On the other hand, compared with the related art, in addition to reducing the storage space of the weight data through quantization compression in the embodiment of the present application, the storage space of the weight data can also be reduced by encoding and compressing the weight data.

[0118] Figure 10 This is a schematic diagram of a method for performing operations on a computing device provided by an embodiment of the present application. Figure 10 The computing device shown includes a global memory and a processor, and the processor may include the above-mentioned Figure 8 The floating point operation unit shown in FIG. 1 may be a matrix operation core. Figure 10 As shown, the data cache module in the matrix operation core includes at least a core cache, an activation matrix register, a weight matrix register, a result register, and a data decompression module. Among them, the core cache is used to cache the first activation data and the second weight data written by the data handling unit, and the matrix register is used to store the first activation data and the second weight data in the form of matrices, respectively, as input matrices of the matrix calculator. The result cache is used to cache the result data obtained after the first activation data and the second weight data are subjected to the floating-point matrix operation. The data decompression module is used to decompress the compressed weight data.

[0119] like Figure 10 As shown, after the data handling unit reads the first activation data and the first weight data from the GM, the first activation data and the first weight data can be written into the core buffer in the matrix operation core. Then, the processor can store the first activation data in the matrix operation core in the form of a vector to a matrix register according to a preset tiling algorithm for performing matrix operations on the first activation data and the first weight data. The matrix register can be called an activation matrix register or a left matrix register, and the first weight data can be stored in the form of a vector to another matrix register. The other matrix register can be called a weight matrix register or a right matrix register.

[0120] The first activation data stored in the activation matrix register can be input into the data calculation module in the matrix operation core, that is, the matrix calculator. The first weight data stored in the weight matrix register can be input into the data decompression module located between the weight matrix register and the matrix calculator for decompression processing. The second weight data obtained by the decompression processing of the data decompression module can be input into the matrix calculator. Since there is a direct connection between the data decompression module, the weight matrix register and the matrix calculator, the transmission rate of the first weight data between the data decompression module, the weight matrix register and the matrix calculator is much greater than the transmission rate of the first weight data in the matrix operation core and the vector operation core.

[0121] Among them, the data decompression module is a logic circuit added between the weight matrix register and the matrix calculator. Depending on the decoding process, the technician can add different logic circuits in the data handling unit to realize the decompression process of the first weight data. For example, adding a logic circuit to convert the data from INT4 to FP16, or adding a logic circuit to decode the data in the block floating point format or 8bit Efloat encoding format into data represented by FP16, etc. In addition, during the execution of the embodiment of the present application, it can be indicated by configuring the software interface whether the data decoding module in the floating point operation unit needs to perform data decoding on the first weight data.

[0122] In addition, since the matrix calculator is generally designed symmetrically, that is, the data bit width corresponding to the second weight data and the first activation data are required to be the same, before the first weight data is input into the data decoding module, the first weight data can be padded with 0 so that the data bit width of the first weight data is the same as the first activation data. In this way, the first weight data can be decompressed into the second weight data having the same data bit width as the first activation data through the data decoding module.

[0123] The matrix calculator can perform floating-point matrix operations on the input first activation data and second weight data. For example, it can perform a matrix inner product operation on the first activation data and the second weight data. The matrix calculator can store the operation results in the result register. The data transfer unit can then transfer the operation results in the result register to the GM.

[0124] It can be seen that the use of the calculation method provided in the embodiment of the present application can avoid the data transfer unit from transferring the first weight data to the vector operation core for decompression processing, and avoid the cross-core transmission of the decompressed second weight data, which can improve the efficiency of floating-point operations on the first activation data and the second weight data.

[0125] In one example, since the first weight data stored in the core buffer and the weight matrix register is compressed weight data, the first weight data occupies less storage space, for example, the first weight data is quantized into 4-bit INT4 data. In this way, under the same storage space, the core buffer and the weight matrix register can store more first weight data. Therefore, a plurality of data decoding modules can be set in parallel between the weight matrix register and the matrix calculator. In implementation, the first weight data stored in the weight matrix register can be input into a plurality of data decoding modules for decoding processing, which can further improve the decoding efficiency of the first weight data, thereby improving the computing efficiency of the second weight data and the first activation data.

[0126] The embodiment of the present application provides a processor, which can be the above-mentioned Figure 3 The processor shown includes a data handling unit and a floating point operation unit, wherein the data handling unit is used to implement the data handling function in the processor, including a data handling module and a data decompression module. The processor can be used to implement the above Figure 4 The operation method shown, where:

[0127] The data handling module is used to obtain the first weight data and first activation data of the artificial intelligence AI model, wherein the first weight data is obtained by compressing the original weight data in the AI ​​model.

[0128] The data decompression module is used to decompress the first weight data to obtain second weight data represented by floating-point numbers.

[0129] The data handling module is used to input the first activation data and the second weight data into the floating-point operation unit to perform floating-point operation.

[0130] In one implementation, the compression process includes at least one of encoding compression and quantization compression; when the compression process includes encoding compression, the decompression process includes decoding processing; when the compression process includes quantization compression, the decompression process includes numerical conversion.

[0131] In one implementation, the compression processing includes encoding the original weight data into first weight data represented by block floating point numbers, and the decoding processing includes decoding the first weight data into second weight data represented by floating point numbers; the quantization compression includes quantizing the original weight data into first weight data represented by integer numbers, and the numerical conversion includes converting the first weight data into second weight data represented by floating point numbers.

[0132] In one implementation, the data transport module includes a data storage device and a transport controller; the transport controller is used to store the first weight data in the GM into the data storage device, and then input the data storage device into the data decompression module; the transport controller is used to input the second weight data into the floating-point operation unit after the data decompression module completes the decompression processing.

[0133] In one implementation, the data transport module includes a data storage device and a transport controller; the transport controller is used to input the first weight data in the GM into the data decompression module; the transport controller is also used to store the second weight data into the data storage device after the data decompression module completes the decompression processing; and the second weight data stored in the data storage device is input into the floating-point operation unit.

[0134] In one implementation, the data handling unit is a direct memory access (DMA) or a tensor memory access (TMA), and the floating-point operation unit is a matrix operation core.

[0135] The processor provided in the embodiment of the present application implements the processing of the first weight data and the first activation data operation and the above embodiment Figure 4 The contents of the corresponding operation methods belong to the same inventive concept, and their specific implementation processes are detailed in the above-mentioned method embodiments, which will not be repeated here. Using the processor provided in the embodiment of the present application, when the data handling unit is carrying the first weight data, the data decoding module in the data handling unit can decompress the first weight data to obtain the second weight data represented by a floating point number. In this way, on the one hand, only the weight data in the AI ​​model is compressed, which can reduce the impact of quantization compression on the accuracy of the AI ​​model while quantizing and compressing the AI ​​model. On the other hand, adding the data decompression module to the data handling unit avoids the data handling unit from carrying the first weight data to the vector operation core for decompression processing, which can improve the efficiency of decompression processing of the first weight data, and thereby improve the efficiency of floating-point operations on the first activation data and the second weight data.

[0136] The embodiment of the present application provides a floating point operation unit, which can be an operation core for performing floating point operations in a processor, such as the above-mentioned Figure 8 The floating point operation unit shown includes a data cache module, a data decompression module and a data calculation module. The floating point operation unit can be used to implement the above Figure 9 The operation method shown, where:

[0137] A data cache module is used to cache the first weight data and first activation data of the artificial intelligence AI model, wherein the first weight data is obtained by compressing the original weight data in the AI ​​model.

[0138] The data decompression module is used to decompress the first weight data to obtain second weight data represented by floating-point numbers.

[0139] The data calculation module is used to perform floating-point operations on the first activation data and the second weight data to obtain a calculation result.

[0140] In one possible implementation, the compression process includes at least one of encoding compression and quantization compression; when the compression process includes encoding compression, the decompression process includes decoding processing; when the compression process includes quantization compression, the decompression process includes numerical conversion.

[0141] In one implementation, the compression processing includes encoding the original weight data into first weight data represented by block floating point numbers, and the decoding processing includes decoding the first weight data into second weight data represented by floating point numbers; the quantization compression includes quantizing the original weight data into first weight data represented by integer numbers, and the numerical conversion includes converting the first weight data into second weight data represented by floating point numbers.

[0142] In one implementation, the data cache module includes a core cache, an activation matrix register and a weight matrix register; the core cache is used to cache first weight data and first activation data, input the first weight data into the weight matrix register, and input the first activation data into the activation matrix register; the weight matrix register is used to input the first weight data into the data decompression module; and the activation matrix register is used to input the first activation data into the data calculation module.

[0143] In one implementation, the floating-point operation unit is a matrix operation core.

[0144] The floating point operation unit provided in the embodiment of the present application realizes the processing of the first weight data and the first activation data operation and the above embodiment Figure 9 The contents of the corresponding operation methods belong to the same inventive concept, and their specific implementation processes are detailed in the above-mentioned method embodiments, which will not be repeated here. Using the floating-point operation unit provided in the embodiment of the present application, the first weight data can be decompressed by the data decoding module before being input into the data calculation module to obtain the second weight data represented by a floating-point number. In this way, on the one hand, only the weight data in the AI ​​model is compressed, which can avoid the influence of the quantization of the activation data on the accuracy of the AI ​​model. On the other hand, adding the data decompression module to the floating-point operation unit avoids the need to transport the first weight data to the vector operation core for decompression processing, which can improve the efficiency of the decompression processing of the first weight data, and thereby improve the efficiency of floating-point operations on the first activation data and the second weight data.

[0145] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the computing method provided in the present application.

[0146] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the operation method provided in the embodiment of the present application.

[0147] In this application, the words such as term "first", "second" are used to distinguish between identical items or similar items having substantially the same effects and functions, and it should be understood that there is no logical or temporal dependency between "first" and "second", nor is the quantity and execution order limited. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of various examples, the first weight data can be referred to as the second weight data, and similarly, the second weight data can be referred to as the first weight data. The first weight data and the second weight data can both be collectively referred to as weight data, and in some cases, can be separate and different weight data.

[0148] The term "at least one" in this application means one or more, and the term "plurality" in this application means two or more.

[0149] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A processor, characterized in that: The processor includes a data handling unit and a floating-point operation unit, and the data handling unit includes a data handling module and a data decompression module, wherein: The data handling module is configured to obtain first weight data and first activation data of an artificial intelligence (AI) model, wherein the first weight data is obtained by compressing the original weight data in the AI ​​model; The data decompression module is used to decompress the first weight data to obtain second weight data represented by floating-point numbers; The data handling module is configured to input the first activation data and the second weight data into the floating-point operation unit; The floating-point operation unit is used to perform floating-point operation on the first activation data and the second weight data to obtain an operation result.

2. The processor according to claim 1, wherein: The compression process includes at least one of encoding compression and quantization compression; When the compression process includes the encoding compression, the decompression process includes the decoding process; When the compression process includes the quantization compression, the decompression process includes numerical value conversion.

3. The processor according to claim 2, wherein: The compression process includes encoding the original weight data into first weight data represented by block floating point numbers, and the decoding process includes decoding the first weight data into second weight data represented by floating point numbers; The quantization compression includes quantizing the original weight data into first weight data represented by integer numbers, and the numerical conversion includes converting the first weight data into second weight data represented by floating-point numbers.

4. The processor according to any one of claims 1 to 3, characterized in that The data handling module includes a data storage device and a handling controller; The transport controller is configured to store the acquired first weight data in the data storage, and then input the data from the data storage into the data decompression module; The transport controller is configured to input the second weight data into the floating-point operation unit after the data decompression module completes the decompression process.

5. The processor according to any one of claims 1 to 3, characterized in that: The data handling module includes a data storage device and a handling controller; The transport controller is configured to input the acquired first weight data into the data decompression module; The transport controller is further configured to store the second weight data in the data memory after the data decompression module completes the decompression process; and input the second weight data stored in the data memory into the floating-point operation unit.

6. The processor according to any one of claims 1 to 3, characterized in that: The data transport unit is a direct memory access (DMA) or a tensor memory access (TMA).

7. The processor according to any one of claims 1 to 3, characterized in that: The floating-point operation unit is a matrix operation core.

8. A floating point arithmetic unit, characterized in that: The floating-point operation unit includes a data cache module, a data decompression module and a data calculation module. The data cache module includes a core buffer, an activation matrix register and a weight matrix register, wherein: The core buffer is used to cache first weight data and first activation data of an artificial intelligence (AI) model, wherein the first weight data is obtained by compressing the original weight data in the AI ​​model, the first weight data cached by the core buffer is used to be input into the weight matrix register, the first activation data cached by the core buffer is used to be input into the activation matrix register, the first weight data cached in the weight matrix register is used to be input into the data decompression module, and the first activation data cached by the activation matrix register is used to be input into the data calculation module; The data decompression module is used to decompress the first weight data to obtain second weight data represented by floating-point numbers; The data calculation module is used to perform floating-point operations on the first activation data and the second weight data to obtain a calculation result.

9. The floating-point arithmetic unit according to claim 8, wherein: The compression process includes at least one of encoding compression and quantization compression; When the compression process includes the encoding compression, the decompression process includes the decoding process; when the compression process includes the quantization compression, the decompression process includes numerical conversion.

10. The floating-point arithmetic unit according to claim 9, wherein: The compression process includes encoding the original weight data into first weight data represented by block floating point numbers, and the decoding process includes decoding the first weight data into second weight data represented by floating point numbers; The quantization compression includes quantizing the original weight data into first weight data represented by integer numbers, and the numerical conversion includes converting the first weight data into second weight data represented by floating-point numbers.

11. The floating-point arithmetic unit according to any one of claims 8 to 10, characterized in that: The floating-point operation unit includes multiple data decompression modules.

12. The floating-point arithmetic unit according to any one of claims 8 to 10, characterized in that: The floating-point operation unit is a matrix operation core.

13. A calculation method, characterized in that: Executed by a processor, the processor including a data handling unit and a floating-point operation unit, including a data handling module and a data decompression module, the method includes: The data handling module obtains first weight data and first activation data of the artificial intelligence (AI) model, wherein the first weight data is obtained by compressing the original weight data in the AI ​​model; The data decompression module decompresses the first weight data to obtain second weight data represented by floating-point numbers; The data handling module inputs the first activation data and the second weight data into the floating point operation unit; The floating-point operation unit is used to perform floating-point operation on the first activation data and the second weight data to obtain an operation result.

14. A calculation method, characterized in that: The method is executed by a floating-point operation unit, wherein the floating-point operation unit includes a data cache module, a data decompression module, and a data calculation module. The data cache module includes a core cache, an activation matrix register, and a weight matrix register. The method includes: The core buffer caches first weight data and first activation data of an artificial intelligence (AI) model, so as to input the first weight data into the weight matrix register and input the first weight data into the activation matrix register, wherein the first weight data is obtained by compressing the original weight data in the AI ​​model; The weight matrix register caches the first weight data to input the first weight data into the data decompression module; The activation matrix register buffers the first activation data to input the first activation data into the data calculation module; The data decompression module decompresses the first weight data to obtain second weight data represented by floating-point numbers; The data calculation module performs floating-point operations on the first activation data and the second weight data to obtain a calculation result.

15. A computing device, characterized in that The computing device includes a memory and a processor according to any one of claims 1 to 7.

16. A computing device, characterized in that The computing device includes a memory and a floating-point operation unit according to any one of claims 8 to 12.

Images