Data processing apparatus, chip, method, device, and medium

Abstract

To provide a data processing apparatus used for performing a matrix calculation between a plurality of first matrix elements and a plurality of second matrix elements.SOLUTION: A data processing apparatus receives multiple sets of quantized data corresponding to multiple first matrix elements. Each set of quantized data includes a quantization parameter and k first matrix elements having a first low-precision format. The data processing apparatus includes a data transmission unit that converts the format of the first matrix elements into a first high-precision format, and a systolic array of processing elements. The processing elements include an arithmetic logic unit that calculates a cumulative sum of products of each element of a first matrix in the first high-precision format and a corresponding second matrix, and a k-dimensional cumulative adder that calculates the products of the quantization parameter included in the quantized data of each matrix element set and the corresponding cumulative sum by using a configured floating-point multiplication unit, and cumulatively adds them.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, particularly to technical fields such as neural networks, deep learning, and artificial intelligence chips. Specifically, it relates to a data processing device, an integrated circuit chip, a data processing method, an electronic device, a computer-readable storage medium, and a computer program product.

Background Art

[0002] Artificial intelligence is a discipline that enables a computer to simulate certain human thinking processes and intelligent behaviors (such as learning, training of neural network models, thinking, planning, etc.). There are both hardware-level technologies and software-level technologies. The hardware technologies of artificial intelligence generally include technologies such as sensors, artificial intelligence dedicated chips, cloud computing, distributed storage, and big data processing. The artificial intelligent software technologies mainly include several directions such as natural language processing technology, computer vision technology, speech recognition technology, and machine learning / deep learning, big data processing technology, and knowledge graph technology.

[0003] The methods described in this part are not necessarily the methods previously assumed or adopted. Unless otherwise specified, none of the methods described in this part should be considered as prior art merely because they are included in this part. Similarly, unless otherwise specified, the problems mentioned in this part should not be considered as recognized in any prior art.

Summary of the Invention

[0004] The present disclosure provides a data processing device, an integrated circuit chip, a data processing method, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of the present disclosure, a data processing apparatus is provided for performing matrix calculations between a plurality of first matrix elements and a plurality of second matrix elements. The apparatus includes a data transmission unit configured to receive, from an external memory, a plurality of sets of quantized data corresponding to the plurality of first matrix elements, each set of quantized data including a quantization parameter and k first matrix elements having a first low-precision format, and perform format conversion on the k first matrix elements having the first low-precision format to obtain k first matrix elements having a first high-precision format, and a systolic array including processing elements, where each processing element is configured to sequentially calculate, for each set of quantized data, a cumulative sum of element-wise products of the k first matrix elements having the first high-precision format corresponding to the set of quantized data and corresponding k second matrix elements among the plurality of second matrix elements, and a k-dimensional accumulator including a floating-point multiplier, the k-dimensional accumulator being configured to calculate and cumulatively add the product of the quantization parameter included in each set of quantized data and the corresponding cumulative sum using the floating-point multiplier, thereby obtaining a matrix calculation result between the plurality of first matrix elements and the plurality of second matrix elements.

[0006] According to another aspect of the present disclosure, an integrated circuit chip including the above data processing apparatus is provided.

[0007] According to another aspect of the present disclosure, a data processing method is provided. This method is used to perform matrix calculations between a plurality of first matrix elements and a plurality of second matrix elements by using a data processing device. The data processing device includes a data transmission unit and a systolic array. The systolic array includes processing elements, and the processing elements include an arithmetic logic unit and a k-dimensional cumulative adder. Here, the k-dimensional cumulative adder includes a floating-point multiplier. The method includes receiving, via the data transmission unit, a plurality of sets of quantized data corresponding to the plurality of first matrix elements from an external memory, where each set of quantized data includes a quantization parameter and k first matrix elements having a first low-precision format; using the data transmission unit to perform format conversion on the k first matrix elements having the first low-precision format to obtain k first matrix elements having a first high-precision format; using the arithmetic logic unit to sequentially calculate, for each set of quantized data, the cumulative sum of the element products of the k first matrix elements having the first high-precision format corresponding to the set of quantized data and the k second matrix elements corresponding thereto; using the floating-point multiplier to calculate and cumulatively add the product of the quantization parameter included in each set of quantized data and the corresponding cumulative sum, thereby obtaining the matrix calculation result between the plurality of first matrix elements and the plurality of second matrix elements.

[0008] According to another aspect of the present disclosure, an electronic device is provided, including at least one processor and a memory communicatively connected to the at least one processor. Here, the memory stores instructions executable by the at least one processor, and when these instructions are executed by the at least one processor, the at least one processor can execute the method described above.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method described above is provided.

[0010] According to another aspect of the present disclosure, there is provided a computer program product including a computer program, which, when executed by a processor, implements the above-described method.

[0011] According to one or more embodiments of the present disclosure, the data processing apparatus according to the present disclosure uses a data transmission unit to convert the first matrix element having the first low-precision format after quantization into one having the first high-precision format, and directly performs the calculation of the product and cumulative sum of elements without performing inverse quantization. Further, by using the floating-point multiplication unit additionally provided in the k-dimensional accumulator, the product of the cumulative sum corresponding to each set of quantized data and the quantization parameter is calculated and cumulatively added. Thereby, before matrix calculation, it is not necessary to complete a large number of inverse quantization operations that take a long time and occupy a large amount of resources, reducing the consumption of computing resources and improving the efficiency of matrix calculation.

[0012] It should be understood that the content described in this part is not intended to identify the key points or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will be easily understood from the following description.

Brief Description of the Drawings

[0013] The drawings illustrate the embodiments by way of example and constitute a part of the specification. Together with the description in the text of the specification, they are used to explain the exemplary embodiments of the embodiments. The embodiments shown are only for illustrative purposes and do not limit the scope of the claims. In all the drawings, the same reference numerals refer to elements that are similar but not necessarily the same.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Mode for Carrying Out the Invention

[0014] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings. For ease of understanding, various details of the embodiments of the present disclosure are included therein, but they should be regarded as merely exemplary. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for clarity and brevity, descriptions of known functions and structures are omitted in the following description.

[0015] In the present disclosure, unless otherwise specified, terms such as "first", "second", etc. for explaining various elements are not intended to limit the positional relationship, timing relationship, or importance relationship of these elements. Such terms are used only to distinguish one element from another. In some examples, the first element and the second element may refer to the same example of the element, and in some cases, they may refer to different examples based on the context description.

[0016] The terms used in the description of various examples of the present disclosure are for the sole purpose of explaining a specific example and are not intended to be limiting. Unless otherwise clearly indicated in the context, if the number of elements is not particularly limited, the element may be one or more. Note that the term "and / or" used in the present disclosure covers any one and all possible combinations of the listed items.

[0017] In recent years, due to the rise of large language models (LLMs), parametric levels in the tens of billions to hundreds of billions have become the standard configuration of LLMs. Many companies pre-train large models on their own and fine-tune them according to various tasks and application scenarios. However, in the process of training and fine-tuning, the requirements for hardware resources are extremely high. Applying quantization technology to the model training and fine-tuning process can significantly reduce the hardware resources required for training large models.

[0018] Currently, in the process of training a model, a high-precision floating-point number format is usually used. However, high precision usually means a high bit width, which increases the burden on hardware resources. In the process of training and fine-tuning, quantization can quantize the high-precision floating-point number format into a specific low-precision format, reducing the requirements for hardware and alleviating the deployment pressure of large models.

[0019] In related technologies, before performing matrix calculations, it is necessary to dequantize the data in a specific low-precision format after quantization into a high-precision high-bit-width format. However, the dequantization process takes a lot of time and significantly occupies hardware resources.

[0020] To solve the above problems, the data processing device according to the present disclosure uses a data transmission unit to convert the first matrix elements having a first low-precision format after quantization into those having a first high-precision format, and directly performs the calculation of the product and cumulative sum of elements without performing dequantization. Furthermore, by using the floating-point multiplication unit additionally set in the k-dimensional accumulator, the product of the cumulative sum corresponding to each set of quantized data and the quantization parameter is calculated and cumulatively added, so that before matrix calculation, it is not necessary to complete a large number of dequantization operations that take a lot of time and occupy a lot of resources, reducing the consumption of computing resources and improving the efficiency of matrix calculation.

[0021] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0022] According to one aspect of the present disclosure, a data processing apparatus is provided. FIG. 1 is a block diagram showing the configuration of a data processing apparatus 100 according to an exemplary embodiment of the present disclosure. The data processing apparatus 100 is used to execute matrix calculations between a plurality of first matrix elements and a plurality of second matrix elements. As shown in FIG. 1, the data processing apparatus 100 receives a plurality of sets of quantized data corresponding to a plurality of first matrix elements from an external memory. Each set of quantized data includes a quantization parameter and k first matrix elements having a first low-precision format. The data transmission unit 110 is configured to perform format conversion on the k first matrix elements having the first low-precision format to obtain k first matrix elements having a first high-precision format, and includes a systolic array 120. The systolic array 120 includes a processing element 122. The processing element 122 is configured to sequentially calculate, for each set of quantized data, the cumulative sum of the element products of k first matrix elements having the first high-precision format corresponding to the set of quantized data and k corresponding second matrix elements among the plurality of second matrix elements. The k-dimensional accumulator 126 includes an arithmetic logic unit 124 and a floating-point multiplication unit 128. The k-dimensional accumulator 126 is configured to calculate and accumulate the product of the quantization parameter included in each set of quantized data and the corresponding cumulative sum using the floating-point multiplication unit 128, thereby obtaining the matrix calculation result between the plurality of first matrix elements and the plurality of second matrix elements.

[0023] In this way, the data processing apparatus according to the present disclosure uses a data transmission unit to convert the first matrix elements having the first low-precision format after quantization into those having the first high-precision format, and directly performs the calculation of the product and the cumulative sum of elements without performing inverse quantization. Further, by using the floating-point multiplication unit additionally provided in the k-dimensional accumulator, the product of the cumulative sum corresponding to each set of quantized data and the quantization parameter is calculated and cumulatively added. Thereby, before matrix calculation, it is not necessary to complete a large amount of inverse quantization operations that take a long time and occupy a large amount of resources, reducing the consumption of computing resources and improving the efficiency of matrix calculation.

[0024] A systolic array is a hardware architecture for efficient parallel computing and is widely applied in fields such as signal processing, image processing, and machine learning. Its basic principle is to perform data transmission and calculation at a certain pace (i.e., "systole") through a series of processing elements (PEs) to achieve efficient data processing and parallel computing. The systolic array is particularly suitable for linear algebra operations such as matrix multiplication. Each processing element is responsible for some multiplication and accumulation operations and completes the entire matrix multiplication process through the data flow.

[0025] In the data processing apparatus 100, the processing element 122 included in the systolic array 120 is used to execute matrix calculations (e.g., element multiplication and accumulation calculations) between a plurality of first matrix elements and a plurality of second matrix elements. As can be understood, the systolic array may include a plurality of processing elements, and each processing element is responsible for executing matrix calculations between a part (e.g., one row) of the first matrix elements in the first matrix and a part (e.g., one column) of the second matrix elements in the second matrix. Thereby, the plurality of processing elements can jointly complete the matrix calculation between the first matrix and the second matrix.

[0026] Hereinafter, each unit in the data processing apparatus will be described in detail.

[0027] The data transmission unit 110 is for receiving a plurality of sets of quantized data corresponding to a plurality of first matrix elements from an external memory. Each set of quantized data includes a quantization parameter and k first matrix elements having a first low-precision format. The plurality of sets of quantized data may be obtained by performing a quantization operation on the plurality of first matrix elements.

[0028] In some embodiments, the external memory can employ a High Bandwidth Memory (HBM).

[0029] In one exemplary embodiment, the plurality of first matrix elements have the following data in a first high-precision format (e.g., Bfloat16): [5, 3, 16, 2, -5, 8, 10, 3] If the preset k value is 4, the plurality of sets of quantization parameters can include the following.

[0030] The first set of quantized data: the quantization parameter = 16, and the four first matrix elements having a first low-precision format (e.g., NF4) are [11, 9, 15, 9]; The second set of quantized data: the quantization parameter = 10, and the four first matrix elements having a first low-precision format (e.g., NF4) are [2, 14, 15, 11].

[0031] According to some embodiments, as shown in FIG. 2, the quantization operation 200 can include step S201 of dividing the plurality of first matrix elements into a plurality of sets, each set including k first matrix elements; step S202 of determining the quantization parameter of each set based on the maximum value in each set; step S203 of normalizing the k first matrix elements in each set based on the quantization parameter of each set; and step S204 of quantizing the normalized first matrix elements into those having a first low-precision format.

[0032] In this way, by the above method, it is realized to group and quantize a plurality of first matrix elements in units of k. Here, k is a parameter related to the hardware structure of the processing elements in the systolic array, which will be described in detail below.

[0033] In one exemplary embodiment, the plurality of first matrix elements have the following data in the first high-precision format (e.g., Bfloat16): [5, 3, 16, 2, -5, 8, 10, 3] and the preset k value is 4. In step S201, the plurality of first matrix elements can be divided into two groups as [5, 3, 16, 2], [-5, 8, 10, 3] In step S202, the maximum values 16 and 10 of the two groups can be used as the quantization parameters of the two groups. In step S203, based on the quantization parameter of each group, the k first matrix elements in the group are normalized to [0.3125, 0.1875, 1, 0.125], [-0.5, 0.8, 1, 0.3] can be obtained. In step S204, the first matrix elements after normalization are quantized into those in the first low-precision format (e.g., NF4), [11, 9, 15, 9], [2, 14, 15, 11] can be obtained.

[0034] In some embodiments, it is possible to improve the accuracy by adopting non-linear quantization. However, the hardware cannot directly process the low-precision format data after non-linear quantization. Note that regardless of whether non-linear quantization or linear quantization is adopted, since the quantization parameters of multiple sets of quantization data are different, even if the low-precision format is converted to the high-precision format, the hardware still cannot directly perform matrix calculations. Therefore, in some embodiments of the related art, the data processing device needs to receive the data after inverse quantization, or a dedicated inverse quantization unit needs to be set in the data processing device, so that the unit in the data processing device used for matrix calculation can directly use the data after inverse quantization.

[0035] In one exemplary embodiment, the inverse quantization operation adopted in the related art is to convert two sets of four first matrix elements having a first low-precision format (e.g., NF4): [11, 9, 15, 9], [2, 14, 15, 11] to those having a first high-precision format (e.g., Bfloat16): [0.338, 0.161, 1.0, 0.161], [-0.525, 0.773, 1.0, 0.338] and further multiply the quantization parameters included in each set of quantization data by the four first matrix elements having the first high-precision format to obtain the result of inverse normalization: [5.4, 2.57, 16, 2.57, -5.25, 7.73, 10, 3.38].

[0036] Note that, for the data processing apparatus and data processing method according to the present disclosure, it is not necessary to completely execute the above inverse quantization operation using quantization parameters before matrix calculation. Instead, by using the data transmission unit 110, while receiving data, the format conversion is completed. Further, by using the arithmetic logic unit of the processing element in the systolic array, the cumulative sum of products of k elements in the data having the first high-precision format, which has not been multiplied by the quantization parameter, is directly calculated. And by using the floating-point multiplication unit additionally installed in the k-dimensional accumulator, the cumulative sum corresponding to each set of quantization data is multiplied by the quantization parameter included in the set of quantization data, and then the cumulative addition operation is executed. By such a method, before matrix calculation, it is not necessary to execute the inverse quantization operation on each quantization data, only format conversion is required, and after matrix calculation, only the product of the cumulative sum and the quantization parameter needs to be calculated. Therefore, compared with the related art, the calculation amount and the consumption of hardware resources are significantly reduced.

[0037] As described above, the data transmission unit 110 is not only used to receive multiple sets of quantization data, but also used to perform format conversion on k first matrix elements having the first low-precision format in each set of quantization data, thereby obtaining k first matrix elements having the first high-precision format.

[0038] According to some embodiments, the first low-precision format may be a 4-bit format (for example, NF4), and the first high-precision format may be a 16-bit format (for example, Bfloat16).

[0039] In one exemplary embodiment, the data transmission unit 110 is used to perform format conversion on two sets of four first matrix elements [11, 9, 15, 9], [2, 14, 15, 11] having the first low-precision format (for example, NF4), thereby obtaining four first matrix elements having the first high-precision format (for example, Bfloat16): [0.338, 0.161, 1.0, 0.161], obtain [-0.525, 0.773, 1.0, 0.338].

[0040] In some embodiments, a storage unit may be configured in the data processing device. The data transmission unit may store in the storage unit the quantization parameters included in each set of quantization data and the k first matrix elements having the first high-precision format after format conversion. Before executing matrix calculations, the processing elements in the systolic array can retrieve the corresponding matrix elements from the storage unit, and the k-dimensional accumulator can retrieve the quantization parameters from the storage unit.

[0041] The arithmetic logic unit 124 in the processing element 122 is configured to sequentially calculate, for each set of quantization data, the cumulative sum of the element products of the k first matrix elements having the first high-precision format corresponding to the set of quantization data and the corresponding k second matrix elements among the plurality of second matrix elements.

[0042] In one exemplary embodiment, if the plurality of first matrix elements are a0, a1, …, a31, the plurality of second matrix elements may be b0, b1, …, b31, and the matrix calculation between them is the cumulative sum of the element products between them:

[0043]

Equation

[0044] When k is 16, the plurality of sets of quantization data may include [a0, …, a15], [a16, …, a31] having the first low-precision format. The k second matrix elements corresponding to the k first matrix elements in the first set of quantization data may be [b0, …, b15], and the k second matrix elements corresponding to the k first matrix elements in the second set of quantization data may be It may be [b16,…,b31].

[0045] The arithmetic logic unit, for the first set of quantized data, [a0,…,a15] (having the first high-precision format) and the cumulative sum of products of elements of [b0,…,b15], that is,

[0046] [Number] calculate, and further, for the second set of quantized data, [a16,…,a31] (having the first high-precision format) and the cumulative sum of products of elements of [b16,…,b31], that is,

[0047] [Number] can be configured to calculate.

[0048] According to some embodiments, the arithmetic logic unit may include k multipliers, for k first matrix elements having the first high-precision format corresponding to one set of quantized data currently being processed, each multiplier being configured to calculate the product of an element of one of the first matrix elements and a corresponding second matrix element, and an adder tree configured to calculate the cumulative sum of products of the k elements output by the k multipliers.

[0049] By adopting the above structure, the cumulative sum of products of elements of k first matrix elements having the first high-precision format and k corresponding second matrix elements can be obtained quickly and accurately.

[0050] In some embodiments, in order to implement the above matrix calculation, the arithmetic logic unit may adopt other structures.

[0051] Since each set of quantized data only includes the quantized data corresponding to some of the data used for matrix calculation, in order to obtain the result of matrix calculation, it is necessary to use a k-dimensional cumulative adder to cumulatively add the cumulative sums obtained by the arithmetic logic unit for different sets of quantized data. Also, when calculating the product and cumulative sum of the above elements, since the object operated on by the arithmetic logic unit is data that has been format-converted but not inverse-quantized, the obtained result cannot be used as it is. By setting a floating-point multiplier in the k-dimensional cumulative adder and multiplying the cumulative sum of the products of the elements output by the arithmetic logic unit by the corresponding quantization parameter and then cumulatively adding, the above problem can be solved. Also, compared to all other inverse quantization schemes (for example, multiplying the quantization parameter by all matrix elements after quantization, or multiplying the quantization parameter by other intermediate calculation results), multiplying the quantization parameter by the cumulative sum of the products of k elements can reduce the computational complexity as much as possible and reduce the time consumption due to inverse quantization.

[0052] In some embodiments, the first matrix element can characterize the weight, and the second matrix element can characterize the variable. The k second matrix elements used when the arithmetic logic unit performs matrix calculation may not be quantized, thereby avoiding an excessive impact on the accuracy of the calculation.

[0053] In some embodiments, the second matrix elements can perform quantization and format conversion in a manner similar to the first matrix elements. That is, the data transmission unit receives multiple sets of second quantization data corresponding to multiple second matrix elements from an external memory (hereinafter, the quantization data corresponding to multiple first matrix elements is referred to as multiple sets of first quantization data, and the quantization parameters included therein are referred to as first quantization parameters), and each set of second quantization data includes a second quantization parameter and k second matrix elements having a second low-precision format. Further, format conversion is performed on the k second matrix elements having the second low-precision format, so as to obtain k second matrix elements having a second high-precision format. Accordingly, the arithmetic logic unit is configured to sequentially calculate the cumulative sum of the element products of k first matrix elements having a first high-precision format and the corresponding k second matrix elements among the multiple second matrix elements for one set of first quantization data and the corresponding one set of second quantization data. The k-dimensional cumulative adder is configured to calculate and cumulatively add the product of the corresponding first quantization parameter, the second quantization parameter, and the above cumulative sum using a floating-point multiplication unit, thereby obtaining the matrix calculation result between the multiple first matrix elements and the multiple second matrix elements.

[0054] In the quantization process, in order to further reduce the data volume, the quantization parameters may be quantized. In order to distinguish between the quantization parameters for quantizing the matrix elements and the quantization parameters for quantizing the quantization parameters, in the present disclosure, the former is referred to as the in-group quantization parameter, and the latter is referred to as the global quantization parameter.

[0055] According to some embodiments, the quantization parameter can include an intra-group quantization parameter. In step S202, determining the quantization parameter of the group based on the maximum value in each group can include quantizing the maximum value of each group based on the global quantization parameter into a value with a second low-precision format in order to obtain the intra-group quantization parameter of the group. The k-dimensional cumulative adder can be configured to calculate and cumulatively add the product of the intra-group quantization parameter included in the quantized data of each group and the corresponding cumulative sum using a floating-point multiplication unit.

[0056] In this way, by the above method, the data volume can be further reduced, and the burden on the hardware resources can be reduced.

[0057] The data transmission unit may directly receive the intra-group quantization parameter after inverse quantization, or the inverse quantization operation on the intra-group quantization parameter may be completed by a specific module in the data processing device.

[0058] According to some embodiments, the data transmission unit can be configured to receive the global quantization parameter from an external memory, perform a format conversion on the intra-group quantization parameter, and thereby obtain the intra-group quantization parameter in a second high-precision format. The k-dimensional cumulative adder can be configured to calculate and cumulatively add the product of the intra-group quantization parameter, the corresponding cumulative sum, and the global quantization parameter included in the quantization parameter of each group using a floating-point multiplication unit.

[0059] In this way, by the above method, the operation of inverse quantizing the intra-group quantization parameter can be further omitted, thereby further improving the calculation efficiency and reducing the consumption of hardware resources.

[0060] According to some embodiments, the second low-precision format may be an 8-bit format, and the second high-precision format may be a 16-bit format. Since the intra-group quantization parameter affects the precision of each matrix element and there is only one intra-group quantization parameter for each group, it can be set and stored with relatively high precision. Since the intra-group quantization parameter is essentially the maximum value among the matrix elements of each group, the corresponding high-precision format can be unified with the high-precision format used for the matrix elements.

[0061] According to another aspect of the present disclosure, an integrated circuit chip is provided. As shown in FIG. 3, the integrated circuit chip 300 may include a data processing device 310. The data processing device 310 can be realized, for example, by the above-mentioned data processing device 100.

[0062] According to another aspect of the present disclosure, a data processing method is provided. The method is used to perform matrix calculations between a plurality of first matrix elements and a plurality of second matrix elements by using a data processing device. The data processing device includes a data transmission unit and a systolic array. The systolic array includes processing elements, and the processing elements include an arithmetic logic unit and a k-dimensional accumulator, where the k-dimensional accumulator includes a floating-point multiplier.

[0063] As shown in FIG. 4, data processing method 400 includes step S401 of receiving, via a data transmission unit, multiple sets of quantized data corresponding to multiple first matrix elements from an external memory, where each set of quantized data includes a quantization parameter and k first matrix elements having a first low-precision format; step S402 of performing format conversion on the k first matrix elements having the first low-precision format by using the data transmission unit to obtain k first matrix elements having a first high-precision format; step S403 of sequentially calculating, by using an arithmetic logic unit, for each set of quantized data, the cumulative sum of the element products of the k first matrix elements having the first high-precision format corresponding to the set of quantized data and k second matrix elements; and step S404 of calculating and cumulatively adding the product of the quantization parameter included in each set of quantized data and the corresponding cumulative sum by using a floating-point multiplication unit, thereby obtaining a matrix calculation result between the multiple first matrix elements and the multiple second matrix elements.

[0064] As can be understood, the operations and effects of steps S401 to S404 in data processing method 400 can refer to the above descriptions of the data transmission unit 110 in data processing apparatus 100, the arithmetic logic unit 124 and the k-dimensional cumulative adder 126 included in the processing element 122 in systolic array 120, and the description here is omitted.

[0065] According to some embodiments, the multiple sets of quantized data may be obtained by performing a quantization operation on the multiple first matrix elements. The quantization operation may include dividing the multiple first matrix elements into multiple sets, with each set including k first matrix elements; determining the quantization parameter of each set based on the maximum value in each set; normalizing the k first matrix elements in each set based on the quantization parameter of each set; and quantizing the normalized first matrix elements into those having a first low-precision format.

[0066] According to some embodiments, the first low-precision format may be a 4-bit format, and the first high-precision format may be a 16-bit format.

[0067] According to some embodiments, the arithmetic logic unit may include k multipliers, each multiplier configured to calculate the element product of one of the k first matrix elements having the first high-precision format corresponding to one set of quantization data being currently processed and a corresponding second matrix element, and an adder tree configured to calculate the cumulative sum of the k element products output by the k multipliers.

[0068] According to some embodiments, the quantization parameter may include an intra-group quantization parameter. Determining the quantization parameter of the group based on the maximum value in each group may include quantizing the maximum value of each group based on the global quantization parameter to a value having a second low-precision format to obtain the intra-group quantization parameter of the group.

[0069] According to some embodiments, the data processing method 400 may further include receiving, via a data transmission unit, a global quantization parameter from an external memory, and using the data transmission unit to perform a format conversion on the intra-group quantization parameter to obtain an intra-group quantization parameter in a second high-precision format (not shown). In step S404, using a floating-point multiplication unit to calculate and cumulatively add the product of the quantization parameter included in the quantization data of each group and the corresponding cumulative sum, thereby obtaining a matrix calculation result between a plurality of first matrix elements and a plurality of second matrix elements, may include using the floating-point multiplication unit to calculate and cumulatively add the product of the intra-group quantization parameter, the corresponding cumulative sum, and the global quantization parameter included in the quantization parameter of each group.

[0070] According to some embodiments, the second low-precision format may be an 8-bit format, and the second high-precision format may be a 16-bit format.

[0071] According to embodiments of the present disclosure, there are further provided an electronic device, a readable storage medium, and a computer program product.

[0072] Next, referring to FIG. 5, a block diagram of an electronic device 500 serving as a server or a client of the present disclosure will be described, which is an example of a hardware device applicable to each aspect of the present disclosure. The electronic device represents various forms of digital electronic computer devices, such as laptop computers, desktop computers, tablets, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may further represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components, their connection relationships, and their functions shown in this specification are merely exemplary and are not intended to limit the implementation of the present disclosure described and / or claimed in this specification.

[0073] As shown in FIG. 5, the electronic device 500 may include a computing unit 501, which can execute various appropriate operations and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. Also, various programs and data necessary for the operation of the electronic device 500 may be stored in the RAM 503. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0074] In the electronic device 500, a plurality of components including an input unit 506, an output unit 507, a memory unit 508, and a communication unit 509 are connected to an I / O interface 505. The input unit 506 may be any type of device capable of inputting information into the electronic device 500. The input unit 506 can receive the input numerical or character information and generate key signal inputs related to user settings and / or function controls of the electronic device. It may include, but is not limited to, a mouse, a keyboard, a touch screen, a track board, a track ball, an operation lever, a microphone, and / or a remote control. The output unit 507 may be any type of device capable of presenting information. It may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The memory unit 508 includes, but is not limited to, a magnetic disk and an optical disk. The communication unit 509 enables the electronic device 500 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks. It may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth (registered trademark) device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0075] The computing unit 501 may be various general-purpose and / or dedicated processing components having processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that execute machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 executes each of the methods, processes, and / or operations described above. For example, in some embodiments, these methods, processes, and / or operations can be realized as a computer software program tangibly included in a machine-readable medium such as the storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed into the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the methods, processes, and / or operations described above can be executed. Alternatively, in other embodiments, the computing unit 501 is configured to execute these methods, processes, and / or operations in any other suitable way (e.g., by firmware).

[0076] The various embodiments of the systems and techniques described above in this specification may be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include the following. Implemented in one or more computer programs, the one or more computer programs may be executed and / or interpreted in a programmable system including at least one programmable processor, the programmable processor may be a dedicated or general-purpose programmable processor, and may receive data and instructions from a memory system, at least one input device, and at least one output device, and may transmit the data and instructions to the memory system, the at least one input device, and the at least one output device.

[0077] The program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, a dedicated computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations defined in the flowchart and / or block diagram are implemented. The program code may be executed entirely by a machine, partially by a machine, partially by a machine as an independent software package and partially by a remote machine, or entirely by a remote machine or server.

[0078] In the context of this disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be either a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium include, but are not limited to, electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing.

[0079] To provide for interaction with a user, the systems and techniques described herein may be implemented on a computer, which includes a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and a pointing device (e.g., a mouse or trackball) by which the user may provide input to the computer. Other kinds of devices may be further provided for interacting with the user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input received from the user may be in any form (including voice input, voice entry, or tactile entry).

[0080] The systems and techniques described herein may be implemented in a computing system that includes a background member (e.g., a data server), a computing system that includes a middleware member (e.g., an application server), a computing system that includes a front-end member (e.g., a user computer having a graphical user interface or a web browser, and a user can interact with embodiments of the systems and techniques described herein through the graphical user interface or the web browser), or a computing system that includes any combination of such background members, middleware members, or front-end members. The members of the system may be interconnected by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0081] A computer system may include a client and a server. The client and the server are generally far apart from each other and usually interact via a communication network. A computer program having a client-server relationship with each other is run on the corresponding computer to generate a client-server relationship. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

[0082] It should be understood that the steps may be reordered, added, or deleted again using various forms of flows shown above. For example, each step described in the present disclosure may be executed in parallel, sequentially, or in a different order, and is not limited herein as long as the technical solutions disclosed in the present disclosure can achieve the desired results.

[0083] Embodiments or examples of the present disclosure have been described with reference to the drawings. However, the above methods, systems, and apparatuses are merely exemplary embodiments or examples, and it should be understood that the scope of the present disclosure is not limited by these embodiments or examples, but is limited only by the scope of the claims after authorization and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalent elements. Note that each step may be executed in an order different from the order described in the present disclosure. Furthermore, various elements in the embodiments or examples may be combined in various ways. In short, with the evolution of technology, many of the elements described herein may be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A data processing apparatus used for performing matrix calculations between a plurality of first matrix elements and a plurality of second matrix elements, the apparatus comprising: receiving a plurality of sets of quantized data corresponding to the plurality of first matrix elements from an external memory, each set of quantized data including a quantization parameter and k first matrix elements having a first low-precision format; a data transmission unit configured to perform format conversion on the k first matrix elements having the first low-precision format to obtain k first matrix elements having a first high-precision format; a systolic array including processing elements; wherein the processing element is configured to sequentially calculate, for each set of quantized data, a cumulative sum of element-wise products of k first matrix elements having the first high-precision format corresponding to the set of quantized data and corresponding k second matrix elements among the plurality of second matrix elements; a k-dimensional accumulator including a floating-point multiplier, the k-dimensional accumulator being configured to calculate and accumulate the product of the quantization parameter included in each set of quantized data and the corresponding cumulative sum using the floating-point multiplier, thereby obtaining a matrix calculation result between the plurality of first matrix elements and the plurality of second matrix elements. A data processing apparatus comprising:

2. The plurality of sets of quantized data are obtained by performing a quantization operation on the plurality of first matrix elements, and the quantization operation includes: dividing the plurality of first matrix elements into a plurality of sets, each set including k first matrix elements; determining the quantization parameter of each set based on the maximum value in each set; normalizing the k first matrix elements in each set based on the quantization parameter of each set; quantizing the normalized first matrix elements into those having the first low-precision format. The apparatus according to claim 1.

3. The quantization parameter includes an intra-group quantization parameter, and determining the quantization parameter of each set based on the maximum value in each set includes: quantizing the maximum value of each set into one having a second low-precision format based on a global quantization parameter to obtain the intra-group quantization parameter of each set. The apparatus according to claim 2.

4. The data transmission unit Receiving the global quantization parameter from the external memory, configured to perform format conversion on the intra-group quantization parameter, thereby obtaining an intra-group quantization parameter in a second high-precision format; The k-dimensional cumulative adder is configured to calculate and cumulatively add the product of a triple of the intra-group quantization parameter, the corresponding cumulative sum, and the global quantization parameter included in the quantization parameter of each group by using the floating-point multiplier, The device according to claim 3.

5. The arithmetic logic unit, For k first matrix elements having a first high-precision format corresponding to one set of quantization data being currently processed, each multiplier is configured to calculate the product of an element of one of the first matrix elements and a corresponding second matrix element. k multipliers, An adder tree configured to calculate a cumulative sum of products of k elements output by the k multipliers, the apparatus according to claim 1.

6. The first low-precision format is a 4-bit format, and the first high-precision format is a 16-bit format, the apparatus according to claim 1.

7. The second low-precision format is an 8-bit format, and the second high-precision format is a 16-bit format, the apparatus according to claim 4.

8. An integrated circuit chip, An integrated circuit chip including the data processing device according to any one of claims 1 to 7.

9. A data processing method, used to perform matrix calculation between a plurality of first matrix elements and a plurality of second matrix elements by using a data processing device, the data processing device includes a data transmission unit and a systolic array, and the systolic array includes a processing element, and the processing element includes an arithmetic logic unit and a k-dimensional cumulative adder, wherein the k-dimensional cumulative adder includes a floating-point multiplier, and the method includes: Receiving, via the data transmission unit, a plurality of sets of quantization data corresponding to the plurality of first matrix elements from an external memory, wherein each set of quantization data includes a quantization parameter and k first matrix elements having a first low-precision format; Using the data transmission unit, format conversion is performed on the k first matrix elements having the first low-precision format to obtain k first matrix elements having the first high-precision format; Using the arithmetic logic unit, for each set of quantization data in turn, calculate the cumulative sum of the element products of k first matrix elements having the first high-precision format corresponding to the set of quantization data and k second matrix elements corresponding thereto; Using the floating-point multiplication unit, calculate and cumulatively add the product of the quantization parameter included in each set of quantization data and the corresponding cumulative sum, thereby obtaining the matrix calculation result between the plurality of first matrix elements and the plurality of second matrix elements. A data processing method including.

10. The plurality of sets of quantization data are obtained by performing a quantization operation on the plurality of first matrix elements, and the quantization operation includes: Dividing the plurality of first matrix elements into a plurality of sets, each set including k first matrix elements; Determining the quantization parameter of each set based on the maximum value in each set; Normalizing the k first matrix elements in each set based on the quantization parameter of each set; Quantizing the first matrix element after normalization into the one having the first low-precision format. The method according to claim 9.

11. The quantization parameter includes an intra-group quantization parameter, and determining the quantization parameter of each set based on the maximum value in each set includes: Obtaining the intra-group quantization parameter of the set, including quantizing the maximum value of each set into the one having the second low-precision format based on the global quantization parameter. The method according to claim 10.

12. Receiving the global quantization parameter from the external memory via the data transmission unit; Further including using the data transmission unit to perform format conversion on the intra-group quantization parameter, thereby obtaining the intra-group quantization parameter in the second high-precision format. Here, by using the floating-point multiplication unit, calculating the product of the quantization parameter included in each set of quantization data and the corresponding cumulative sum of additions, and then performing cumulative addition, the matrix calculation result between the plurality of first matrix elements and the plurality of second matrix elements can be obtained. The method according to claim 11, wherein by using the floating-point multiplication unit, the product of the in-group quantization parameter, the corresponding cumulative sum of additions, and the global quantization parameter included in each set of quantization parameters is calculated and then cumulatively added.

13. The arithmetic logic unit For k first matrix elements having a first high-precision format corresponding to one set of quantization data currently being processed, each multiplier is configured to calculate the product of one of the first matrix elements and the corresponding second matrix element, and k multipliers; The method according to claim 9, further comprising an adder tree configured to calculate a cumulative sum of the products of the k elements output by the k multipliers.

14. The method according to claim 9, wherein the first low-precision format is a 4-bit format and the first high-precision format is a 16-bit format.

15. The method according to claim 12, wherein the second low-precision format is an 8-bit format and the second high-precision format is a 16-bit format.

16. An electronic device, comprising: At least one processor; and A memory communicatively connected to the at least one processor, wherein The memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor can be caused to execute the method according to any one of claims 9 to 15.

17. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to any one of claims 9 to 15.

18. A computer program product including a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 9 to 15 is realized.

Images