Data processing method, computing apparatus and computing chip

Abstract

The present application provides a data processing method, a computing apparatus and a computing chip. The method provided in the present application comprises: a tensor processing unit performs a matrix multiplication operation on an input first matrix and second matrix so as to obtain an operation result, and extracts from the operation result the maximum value in a specified dimension; a first computing unit computes the difference value between the current flow-in element and the maximum value, wherein elements in the operation result successively flow into the first computing unit; a second computing unit performs an exponential operation on the difference value to obtain an exponential operation result corresponding to the element; a third computing unit performs a cumulative operation on all of the exponential operation results corresponding to all of the elements in the operation result, so as to obtain a cumulative sum; a vector computing module computes the reciprocal of the cumulative sum, and uses the reciprocal as a normalization factor of an activation operation; and a second pipeline module performs a vector multiplication operation on the exponential operation result corresponding to each element and the normalization factor, so as to obtain a softmax value corresponding to each element. By means of pipeline design, the data processing method, the computing apparatus and the computing chip provided in the present application can improve the computing efficiency of softmax computation.

Description

A data processing method, a computing device, and a computing chip

[0001] This application claims priority to Chinese Patent Application No. 202411224449.6, filed on September 2, 2024, entitled "A Data Processing Method, Computing Device and Computing Chip", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, a computing device, and a computing chip. Background Technology

[0003] The Transformer model is an important architecture in deep learning, widely used in fields such as natural language processing and computer vision.

[0004] In fields such as natural language processing and computer vision, the running speed of large models is a concern. Among them, the computational efficiency of Transformer is one of the main factors affecting the running speed of large models. As a crucial part of the Transformer architecture, the computational efficiency of softmax has a significant impact on the performance of the entire large model. Improving the computational efficiency of softmax is particularly important for optimizing the running speed of large models.

[0005] Summary of the Invention

[0006] The data processing method, computing device, and computing chip provided in this application are used to improve the computational efficiency of softmax and optimize the running speed of large models.

[0007] Specifically, this application is implemented through the following technical solution:

[0008] This application provides a computing device, comprising a tensor processor and a vector processor, wherein the vector processor includes a first pipeline module, a vector computation module, and a second pipeline module; the first pipeline module includes cascaded first computation unit, second computation unit, and third computation unit; wherein...

[0009] The tensor processor is used to perform matrix multiplication on the input first matrix and second matrix, obtain the operation result, and extract the maximum value in a specified dimension from the operation result;

[0010] The first calculation unit is used to calculate the difference between the currently flowing element and the maximum value; wherein, each element in the calculation result flows into the first calculation unit in sequence;

[0011] The second calculation unit is used to perform an exponential operation on the difference to obtain the exponential operation result corresponding to the element;

[0012] The third calculation unit is used to perform a cumulative calculation on all exponents corresponding to all elements in the calculation result to obtain a cumulative sum.

[0013] The vector calculation module is used to calculate the reciprocal of the accumulated sum and use the reciprocal as a normalization factor for the activation operation;

[0014] The second pipeline module is used to perform vector multiplication on the exponent operation result corresponding to each element in the operation result and the normalization factor to obtain the softmax value corresponding to each element.

[0015] A second aspect of this application provides a data processing method, wherein the data processing method utilizes the computing device described in any one of the first aspects of this application to perform data processing, the method comprising:

[0016] The tensor processor performs matrix multiplication on the input first and second matrices to obtain the result, and extracts the maximum value in the specified dimension from the result.

[0017] The first calculation unit calculates the difference between the currently flowing element and the maximum value; wherein, each element in the calculation result flows into the first calculation unit in sequence;

[0018] The second calculation unit performs an exponential operation on the difference to obtain the exponential operation result corresponding to the element.

[0019] The third calculation unit performs a cumulative sum operation on all exponent operation results corresponding to all elements in the calculation result to obtain the cumulative sum.

[0020] The vector calculation module calculates the reciprocal of the accumulated sum and uses the reciprocal as a normalization factor for the activation operation;

[0021] The second pipeline module performs vector multiplication on the exponential operation result corresponding to each element and the normalization factor to obtain the softmax value corresponding to each element.

[0022] A third aspect of this application provides a computing chip, which includes any of the computing devices provided in the first aspect of this application.

[0023] The data processing method, computing device, and computing chip provided in this application, by setting up a tensor processor and a vector processor, and setting up a first pipeline module, a vector calculation module, and a second pipeline module in the vector processor, and further setting up a cascaded first calculation unit, a second calculation unit, and a third calculation unit in the first pipeline module, can simultaneously obtain the maximum value by performing matrix multiplication operations on the input first matrix and second matrix using the tensor processor. Furthermore, through the two-pipeline design, each element in the calculation result flows sequentially into the first pipeline module. Based on the first calculation unit in the first pipeline module, the difference between the currently flowing element and the maximum value can be calculated, and then the second pipeline module performs an exponential operation on the difference to obtain... The exponential operation result corresponding to the element is obtained, and then the third calculation unit performs a cumulative operation on all exponential operation results corresponding to all elements in the operation result to obtain a cumulative sum. Further, after calculating the cumulative sum, the softmax value of each element can be calculated through the vector calculation module and the second pipeline module. In this way, firstly, the calculation task is decomposed into multiple independent parts for parallel processing, which can improve the calculation speed; secondly, the cascaded calculation units in the first pipeline module can enable each calculation unit to start processing new input data while the previous calculation unit completes its task, which can reduce waiting time, save calculation time, improve the calculation efficiency of softmax, and thus improve the overall running efficiency of the large model. Attached Figure Description

[0024] Figure 1 is a schematic diagram of a computing device according to a first embodiment of the present application;

[0025] Figure 2 is a schematic diagram of a second embodiment of the computing device provided in this application;

[0026] Figure 3 is a schematic diagram of a third embodiment of the computing device provided in this application;

[0027] Figure 4 is a schematic diagram of Embodiment 4 of the computing device provided in this application;

[0028] Figure 5 is a flowchart of Embodiment 1 of the data processing method provided in this application;

[0029] Figure 6 is a flowchart of Embodiment 2 of the data processing method provided in this application;

[0030] Figure 7 is a flowchart of Embodiment 3 of the data processing method provided in this application. Detailed Implementation

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0032] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0033] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0034] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0035] Figure 1 is a schematic diagram of a computing device according to a first embodiment of this application. Referring to Figure 1, the computing device provided in this embodiment may include a tensor processor 110 and a vector processor 120. The vector processor 120 includes a first pipeline module 121, a vector computing module 122, and a second pipeline module 123. The first pipeline module 121 includes a cascaded first computing unit U1, a second computing unit U2, and a third computing unit U3.

[0036] The tensor processor 110 is used to perform matrix multiplication on the input first matrix and second matrix to obtain the operation result, and extract the maximum value in a specified dimension from the operation result.

[0037] The first calculation unit U1 is used to calculate the difference between the currently flowing element and the maximum value; wherein, each element in the calculation result flows into the first calculation unit U1 in sequence;

[0038] The second calculation unit U2 is used to perform an exponential operation on the difference to obtain the exponential operation result corresponding to the element;

[0039] The third calculation unit U3 is used to perform a cumulative calculation on all exponents corresponding to all elements in the calculation result to obtain a cumulative sum.

[0040] The vector calculation module 122 is used to calculate the reciprocal of the accumulated sum and use the reciprocal as a normalization factor for the activation operation;

[0041] The second pipeline module 123 is used to perform vector multiplication on the exponent operation result corresponding to each element in the operation result and the normalization factor to obtain the softmax value corresponding to each element.

[0042] Specifically, referring to Figure 1, the tensor processor 110 is used to perform matrix multiplication on the input first matrix and second matrix to obtain the operation result, and extract the maximum value in the specified dimension from the operation result.

[0043] It should be noted that the first and second matrices are set according to actual needs, and are not limited in this embodiment. For example, in one possible implementation, for a multi-head attention mechanism, it is necessary to calculate softmax(Q*K). T )*V, where the first matrix can be Q and the second matrix can be K. T .

[0044] Specifically, the dimensions of the first matrix, the second matrix, and the specified dimension in the calculation result are all set according to actual needs, and are not limited in this embodiment. For example, in one possible implementation, the specified dimension of the calculation result can be the dimension of the row vector or the dimension of the column vector.

[0045] Specifically, it can be understood that matrix multiplication can be represented by the following formula: C = A * B

[0046] Where C is the result of the operation;

[0047] A is the first matrix;

[0048] B is the second matrix.

[0049] In a specific implementation, for example, in one possible approach, the first matrix is ​​an m*n matrix and the second matrix is ​​an n*p matrix. In this case, after performing matrix multiplication, the result is an m*p matrix. Furthermore, when the specified dimension is the dimension corresponding to the row vector, the dimension of the row vector can be extracted from the m*p matrix, that is, the elements of each row are compared in parallel to find the maximum value max(xi).

[0050] Further, referring to Figure 1, the first calculation unit U1 is used to calculate the difference between the currently flowing element and the maximum value; wherein, each element in the calculation result flows into the first calculation unit U1 in sequence.

[0051] Specifically, referring to the example above, the result of the operation is essentially the previous matrix, and the elements in the result refer to the individual values ​​in that matrix. For example, referring to the example above, when the result is an m*p matrix, the result includes m*p elements. These m*p elements flow into the first calculation unit U1 in sequence. After processing the first element, the first calculation unit U1 processes the second element, and so on.

[0052] Furthermore, the first calculation unit U1 can calculate the difference between the currently flowing element and the maximum value using the following formula: mi = xi - max(xi)

[0053] Where mi is the difference between the currently flowing element xi and the maximum value;

[0054] xi represents the element currently flowing in;

[0055] max(xi) is the maximum value.

[0056] It should be noted that, as described above, the elements in the calculation result flow into the first calculation unit U1 in sequence. That is, the elements flow into the first calculation unit U1 in the order they appear. After the previous element flows into the first calculation unit U1 and completes the calculation, it enters the second calculation unit U2, and then the next element flows into the first calculation unit U1 for calculation.

[0057] Referring to the example above, in one embodiment, and as described earlier, for ease of explanation, the result of the operation is denoted as matrix C. This matrix is ​​an m*p matrix containing m*p elements, each denoted by Cij, where i ranges from 1 to m and j ranges from 1 to p. First, element C11 enters the first calculation unit U1, which performs a difference calculation (i.e., calculates the difference between it and the maximum value). After the calculation, the difference enters the second calculation unit U2 for exponentiation. While the second calculation unit U2 is calculating the exponentiation result corresponding to element C11, element C12 enters the first calculation unit U1 for difference calculation. The first calculation unit U1 simultaneously calculates the difference between element C12 and the maximum value, and so on, until element Cmp finally enters the first calculation unit U1 for difference calculation.

[0058] Further, please refer to Figure 1. The second calculation unit U2 is used to perform an exponential operation on the difference to obtain the exponential operation result corresponding to the element.

[0059] Specifically, the implementation principle of exponentiation is as follows: pi = e mi

[0060] Where pi is the result of the exponentiation operation corresponding to the i-th element;

[0061] mi is the difference between the i-th element and the maximum value.

[0062] It should be noted that, referring to Figure 1 and the preceding description, in the first pipeline module 121, after the first calculation unit U1 calculates the difference between a certain element and its maximum value, this difference flows sequentially into the second calculation unit U2. The second calculation unit U2 performs an exponential operation on the currently flowing difference. Based on the above description, it can be understood that after the first calculation unit U1 completes the calculation of the difference m11 between element C11 and its maximum value, m11 flows into the second calculation unit U2. At this time, element C12 flows into the first calculation unit U1 for difference calculation, and m11 flows into the second calculation unit U2 for exponential operation.

[0063] Furthermore, referring to Figure 1, the third calculation unit U3 is used to perform a cumulative calculation on all exponents corresponding to all elements in the calculation result to obtain a cumulative sum.

[0064] Specifically, the results of all exponent operations corresponding to all elements in the calculation result are summed according to the following formula:

[0065] Where n is the cumulative sum;

[0066] xi is an element;

[0067] max(xi) is the maximum value.

[0068] In other words, the accumulation operation is as follows:

[0069] As described above, pi is the result of the exponentiation operation corresponding to the i-th element, and the value of i ranges from 1 to m*p.

[0070] It should be noted that in the first pipeline module 121, the exponentiation results output by the second calculation unit U2 flow sequentially into the third calculation unit U3. The third calculation unit U3 calculates a partial sum based on the incoming exponentiation results until the exponentiation results of all elements flow into the third calculation unit U3, obtaining the final sum. Referring to the above description, when the exponentiation result p11 of element C11 flows into the third calculation unit U3, element C12 flows into the second calculation unit U2 for exponentiation, and element C13 flows into the first calculation unit U1 for difference calculation.

[0071] Furthermore, when each element flows into the third calculation unit U3, the third calculation unit U3 will perform an accumulation operation on the incoming elements to obtain a partial sum. If more elements continue to enter the third calculation unit U3, the accumulation operation will continue until all elements have flowed into the third calculation unit U3, at which point the accumulation operation will stop, and the sum of the exponential operation results of all elements flowing into the third calculation unit U3 will be obtained.

[0072] The computing device provided in this embodiment, by setting up a first pipeline module and cascading a first computing unit, a second computing unit, and a third computing unit in the first pipeline module, can sequentially flow each element in the calculation result into the first computing unit, the second computing unit, and the third computing unit to form a pipelined parallel computing, which significantly improves computing efficiency.

[0073] Further, referring to Figure 1, the vector calculation module 122 is used to calculate the reciprocal of the sum and use the reciprocal as a normalization factor for the activation operation.

[0074] Specifically, the reciprocal of the sum can be calculated using the following formula: q = 1 / n

[0075] Where q is the reciprocal of the sum;

[0076] n is the cumulative sum.

[0077] Further, please refer to Figure 1. The second pipeline module 123 is used to perform vector multiplication on the exponent operation result corresponding to each element in the operation result and the normalization factor to obtain the softmax value corresponding to each element.

[0078] Specifically, a vector multiplication operation is performed on the exponent result corresponding to each element in the calculation result, as well as the normalization factor, using the following formula: si=pi*q

[0079] Where si is the softmax value corresponding to the i-th element;

[0080] pi is the result of the exponentiation of the i-th element;

[0081] q is the normalization factor, which is the reciprocal of the sum.

[0082] The computing device provided in this embodiment, by setting up a tensor processor and a vector processor, and setting up a first pipeline module, a vector computation module and a second pipeline module in the vector processor, and further setting up a cascaded first computation unit, a second computation unit and a third computation unit in the first pipeline module, can obtain the maximum value while performing matrix multiplication on the input first matrix and second matrix using the tensor processor. Furthermore, through the two pipeline design, each element in the computation result flows sequentially into the first pipeline module. Based on the first computation unit in the first pipeline, the difference between the currently flowing element and the maximum value can be calculated. Then, the second pipeline module performs an exponential operation on the difference to obtain the value corresponding to the element. The exponential operation results are then processed by the third calculation unit, which accumulates all the exponential operation results corresponding to all elements in the result to obtain a sum. Further, after obtaining the sum, the softmax value of each element is calculated using the vector calculation module and the second pipeline module. This approach achieves two advantages: firstly, decomposing the computational task into multiple independent parts for parallel processing improves computational speed; secondly, the cascaded calculation units in the first pipeline module allow each unit to begin processing new input data while the previous unit completes its task, reducing waiting time, saving computation time, and improving the computational efficiency of softmax, thereby enhancing the overall operating efficiency of the large model.

[0083] Optionally, in one possible implementation, when the first matrix and the second matrix are data in fixed-point form, the tensor processor 110 and the first computing unit U1 are specifically used to perform operations based on the data in fixed-point form;

[0084] The second calculation unit U2 is specifically used to look up the floating-point exponent value corresponding to the difference from a preset first correspondence table for the difference, and use the found exponent value as the exponent operation result corresponding to the element; wherein, the first correspondence table is used to record the mapping relationship between fixed-point data and the floating-point exponent value corresponding to the fixed-point data;

[0085] The third computing unit U3, the vector computing module 122, and the second pipeline module 123 are specifically used to perform calculations based on floating-point data so that the output softmax value is floating-point data.

[0086] The computing device provided in this embodiment can output a softmax value in floating-point form based on a first matrix and a second matrix in fixed-point form. Its internal working principle is described below:

[0087] Specifically, fixed-point data refers to numerical values ​​represented using a fixed number of decimal and integer digits. Specifically, when the first and second matrices are in fixed-point format, the computation result output by the tensor processor 110, as well as the maximum value extracted from the computation result in a specified dimension, are also in fixed-point format. Correspondingly, the difference output by the first computation unit U1 is also in fixed-point format.

[0088] Furthermore, when the difference output by the first calculation unit U1 is data in fixed-point form, the second calculation unit U2 calculates the exponential operation result based on the fixed-point to floating-point lookup method.

[0089] It should be noted that the first mapping table is used to record the mapping relationship between fixed-point data and the corresponding floating-point exponent value. In essence, the first mapping table is a lookup table used to map fixed-point data to floating-point exponent values; furthermore, this lookup table is predefined and contains all possible fixed-point data and their corresponding floating-point exponent values.

[0090] In practice, the second calculation unit U2 will look up the corresponding relationship containing the current inflow difference from the preset first correspondence table, and then determine the floating-point exponent value recorded in the found correspondence as the exponent operation result corresponding to the difference.

[0091] It is understood that the exponential operation result output by the second calculation unit U2 is data in floating-point form. Therefore, the third calculation unit U3, the vector calculation module 122 and the second pipeline module 123 are specifically used to perform calculations based on floating-point data to output a softmax value in floating-point form.

[0092] Please refer to the previous description for the process of data processing by the third computing unit, vector computing module, and second pipeline module; it will not be repeated here.

[0093] In the apparatus provided in this embodiment, when the input first and second matrices are fixed-point data, the tensor processor and the first computing unit directly perform operations based on the fixed-point data. Fixed-point operations are generally more efficient than floating-point operations. Using fixed-point data for preliminary operations can reduce computational complexity, increase computational speed, and improve computational efficiency. Furthermore, when calculating the exponential operation result, the second computing unit calculates the exponential operation result based on a pre-constructed first correspondence table that records the mapping relationship between fixed-point data and the corresponding floating-point exponential value. In this way, on the one hand, fixed-point data can be converted into floating-point data to meet practical needs; on the other hand, when calculating the exponential operation result, the table lookup method can be used to further save computational load and improve computational efficiency.

[0094] Optionally, in one possible implementation, when the first matrix and the second matrix are data in floating-point form, the tensor processor 110 and the first computing unit U1 are specifically used to perform operations based on the floating-point data;

[0095] The second calculation unit U2 is specifically used to look up the target function corresponding to the difference from a preset second correspondence table for the difference; wherein, the second correspondence table records the mapping relationship between floating-point data and the linear function corresponding to the floating-point data, and the function value of the linear function corresponding to the floating-point data is equal to the exponent value of the floating-point data;

[0096] The second calculation unit U2 is further configured to substitute the difference as an independent variable into the objective function to obtain the dependent variable value, and to determine the dependent variable value as the exponential operation result corresponding to the element;

[0097] The third computing unit U3, the vector computing module 122, and the second pipeline module 123 are specifically used to perform calculations based on floating-point data so that the output softmax value is floating-point data.

[0098] The computing device provided in this embodiment can output a softmax value in floating-point form based on a first matrix and a second matrix in floating-point form. Its internal working principle is described below:

[0099] Specifically, floating-point data typically consists of a sign bit, an exponent bit, and a mantissa (fractional part). Floating-point data can represent very large or very small numerical ranges, and its precision changes dynamically within the numerical range.

[0100] Specifically, when the first and second matrices are in floating-point format, the calculation results output by the tensor processor 110 and the maximum value extracted from the calculation results in a specified dimension are also in floating-point format. Correspondingly, the difference output by the first computation unit U1 is also in floating-point format.

[0101] Furthermore, when the difference output by the first calculation unit U1 is data in fixed-point form, the second calculation unit U2 calculates the exponentiation result based on a floating-point lookup table (ax+b).

[0102] In specific implementation, the second calculation unit U2 is specifically used to look up the target function corresponding to the difference from a preset second correspondence table for the difference, and substitute the difference as the independent variable into the target function to obtain the dependent variable value, and then determine the dependent variable value as the exponential operation result corresponding to the element; wherein, the second correspondence table records the mapping relationship between floating-point data and the linear function corresponding to the floating-point data, and the function value of the linear function corresponding to the floating-point data is equal to the exponential value of the floating-point data.

[0103] Specifically, the second mapping table records the mapping relationship between floating-point data and the corresponding linear function. The function value of the linear function corresponding to the floating-point data is equal to the exponent value of the floating-point data. That is, each floating-point data has a corresponding linear function, and the function value of the linear function under the floating-point data represents the exponent value of the floating-point data. In practical implementation, piecewise linear functions can be used to fit the exponential function to obtain the linear function corresponding to each floating-point data.

[0104] In specific implementation, the second calculation unit U2 finds the objective function in the preset second correspondence table and substitutes the difference into the objective function to calculate the dependent variable value, which represents the exponent value of the floating-point data. This process uses the linear function in the second correspondence table to calculate the exponent operation result, transforming the complex exponent operation into a simple linear function operation.

[0105] Based on the above introduction, for example, in one possible implementation, the difference is 0.2. Looking up the objective function corresponding to the difference 0.2 in the second correspondence table, we find f(x) = 3x + 2.1. Substituting the difference 0.2 into the objective function, we get f(0.2) = 3 * 0.2 + 2.1, resulting in an exponential calculation of 2.7. This method simplifies complex exponential operations into simple linear function calculations, improving computational efficiency.

[0106] Furthermore, it can be understood that the exponential operation result output by the second calculation unit U2 is data in floating-point form. Therefore, the third calculation unit U3, the vector calculation module 122, and the second pipeline module 123 are specifically used to perform calculations based on floating-point data to output a softmax value in floating-point form.

[0107] In the apparatus provided in this embodiment, when the input first matrix and second matrix are floating-point data, the tensor processor and the first computing unit are specifically used to perform operations based on the floating-point data. Furthermore, when the second computing unit calculates the exponential operation result, it calculates the exponential operation result based on a preset second correspondence table used to record the mapping relationship between floating-point data and the linear function corresponding to the floating-point data. In this way, when calculating the exponential operation result, the function value of the linear function corresponding to the floating-point data is equivalent to the exponential value of the floating-point data, simplifying the complex exponential operation into a simple linear function calculation, which can save the amount of computation and improve the computational efficiency.

[0108] Figure 2 is a schematic diagram of a second embodiment of the computing device provided in this application. Referring to Figure 2, the computing device provided in this embodiment further includes a memory 130; wherein,

[0109] The memory 130 is used to store the calculation results of each calculation.

[0110] Specifically, the calculation results of each calculation are stored in memory 130 for subsequent data use. Memory 130 can be DM, DDR, etc., but this embodiment does not limit its use.

[0111] Figure 3 is a schematic diagram of a third embodiment of the computing device provided in this application. Referring to Figure 3, the computing device provided in this embodiment, based on the above embodiments, includes a memory 130 comprising a first storage module 131 corresponding to the tensor processor 110 and a second storage module 132 corresponding to the vector processor 120; wherein,

[0112] The first storage module 131 is used to store the calculation result and the maximum value;

[0113] The second storage module 132 is used to store the exponential operation results corresponding to all elements in the operation result, as well as the cumulative sum.

[0114] The device provided in this embodiment divides the memory into a first storage module that stores the operation result and the maximum value, and a second storage module that stores the exponential operation result corresponding to all elements in the operation result, as well as the cumulative sum. In this way, on the one hand, the storage requirements of the tensor processor and the vector processor are separated, reducing data conflicts, realizing parallel processing, optimizing memory usage, and improving overall processing efficiency; on the other hand, storage resources can be better utilized, latency and errors can be reduced, and the overall performance of the computing device can be improved.

[0115] Figure 4 is a schematic diagram of a fourth embodiment of the computing device provided in this application. Referring to Figure 4, the second storage module 132 provided in this embodiment includes a storage unit C1 and a cache unit H1; wherein,

[0116] The cache unit H1 is used to cache the exponent operation results corresponding to all elements in the operation result, as well as the cumulative sum;

[0117] The storage unit C1 is used to store the result of the exponential operation and the summation; wherein the data stored in the storage unit C1 is in fixed-point format.

[0118] Specifically, as can be understood, referring to Figure 4, after the second calculation unit U2 calculates the exponentiation result of each element, it stores the exponentiation result in the cache unit H1. After the third calculation unit U3 calculates the sum of all elements, it stores the sum in the cache unit H1.

[0119] Furthermore, the cache unit H1 converts the exponentiation result and the sum into fixed-point data and stores it in the storage unit C1 (in a specific implementation, the floating-point data is converted into int8 data and stored in the storage unit). The cache unit H1 is also used to output the exponentiation result and the sum to the vector calculation module 122 and the second pipeline module 123 for subsequent processing.

[0120] The apparatus provided in this embodiment, by further subdividing the second storage module into a cache unit for caching the exponential operation results corresponding to all elements in the operation result, and a storage unit for storing the exponential operation results in fixed-point form and the accumulated sum, can avoid conflicts that occur when multiple processors access the same storage module at the same time, thereby reducing the latency and errors caused by access conflicts.

[0121] Corresponding to the aforementioned embodiment of a computing device, this application also provides an embodiment of a data processing method.

[0122] Figure 5 is a flowchart of a first embodiment of the data processing method provided in this application. Referring to Figure 5, the method provided in this embodiment utilizes the computing device described in any one of the first aspects of this application to perform data processing. The method includes:

[0123] S501, the tensor processor performs matrix multiplication on the input first matrix and second matrix to obtain the operation result, and extracts the maximum value in the specified dimension from the operation result.

[0124] S502, the first calculation unit calculates the difference between the currently flowing element and the maximum value; wherein, each element in the calculation result flows into the first calculation unit in sequence.

[0125] S503, the second calculation unit performs an exponential operation on the difference to obtain the exponential operation result corresponding to the element.

[0126] S504. The third calculation unit performs a cumulative sum operation on all exponent operation results corresponding to all elements in the calculation result to obtain the cumulative sum.

[0127] S505, The vector calculation module calculates the reciprocal of the accumulated sum and uses the reciprocal as the normalization factor for the activation operation.

[0128] S506, the second pipeline module performs vector multiplication on the exponential operation result corresponding to each element and the normalization factor to obtain the softmax value corresponding to each element.

[0129] The method in this embodiment is implemented based on any of the computing devices provided in the first aspect of this application. The specific implementation principle and process are similar and will not be repeated here.

[0130] Figure 6 is a flowchart of Embodiment 2 of the data processing method provided in this application. Referring to Figure 6, based on the above embodiment, when the first matrix and the second matrix are fixed-point data, the method includes:

[0131] S601, The tensor processor and the first computing unit perform operations based on fixed-point data.

[0132] S602. The second calculation unit searches for the exponent value corresponding to the difference from a preset first correspondence table for the difference, and uses the found exponent value as the exponent operation result corresponding to the element; wherein, the first correspondence table is used to record the mapping relationship between fixed-point data and the floating-point form exponent value corresponding to the fixed-point data.

[0133] S603, the third calculation unit, the vector calculation module, and the second pipeline module perform calculations based on floating-point data so that the output softmax value is floating-point data.

[0134] For details on the specific implementation process and principles of each step, please refer to the descriptions in the previous embodiments, which will not be repeated here.

[0135] Figure 7 is a flowchart of Embodiment 3 of the data processing method provided in this application. Referring to Figure 7, based on the above embodiment, when the first matrix and the second matrix are floating-point data, the method includes:

[0136] S701, The tensor processor and the first computing unit perform operations based on floating-point data.

[0137] S702. The second calculation unit searches for the target function corresponding to the difference from a preset second correspondence table for the difference; wherein, the second correspondence table records the mapping relationship between floating-point data and the linear function corresponding to the floating-point data, and the function value of the linear function corresponding to the floating-point data is equal to the exponent value of the floating-point data.

[0138] S703. The second calculation unit substitutes the difference as an independent variable into the objective function to obtain the dependent variable value, and determines the dependent variable value as the result of the exponential operation corresponding to the element.

[0139] S704, the third computing unit, the vector computing module, and the second pipeline module perform calculations based on floating-point data so that the output softmax value is floating-point data.

[0140] For the specific implementation process and principles of each step, please refer to the description in the previous embodiments, which will not be repeated here.

[0141] Furthermore, this application also provides a computing chip, which includes any of the computing devices provided in the first aspect of this application.

[0142] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A computing device, characterized in that, The computing device includes a tensor processor and a vector processor. The vector processor includes a first pipeline module, a vector computation module, and a second pipeline module. The first pipeline module includes a cascaded first computation unit, a second computation unit, and a third computation unit. The tensor processor is used to perform matrix multiplication on the input first matrix and second matrix, obtain the operation result, and extract the maximum value in a specified dimension from the operation result. The first calculation unit is used to calculate the difference between the currently flowing element and the maximum value; wherein, each element in the calculation result flows into the first calculation unit in sequence; The second calculation unit is used to perform an exponential operation on the difference to obtain the exponential operation result corresponding to the element; The third calculation unit is used to perform a cumulative calculation on all exponents corresponding to all elements in the calculation result to obtain a cumulative sum. The vector calculation module is used to calculate the reciprocal of the accumulated sum and use the reciprocal as a normalization factor for the activation operation; The second pipeline module is used to perform vector multiplication on the exponent operation result corresponding to each element in the operation result and the normalization factor to obtain the softmax value corresponding to each element.

2. The computing device according to claim 1, characterized in that, When the first matrix and the second matrix are data in fixed-point form, the tensor processor and the first computing unit are specifically used to perform operations based on the data in fixed-point form. The second calculation unit is specifically used to look up the floating-point exponent value corresponding to the difference from a preset first correspondence table for the difference, and use the found exponent value as the result of the exponent operation corresponding to the element; wherein, the first correspondence table is used to record the mapping relationship between fixed-point data and the floating-point exponent value corresponding to the fixed-point data; The third computing unit, the vector computing module, and the second pipeline module are specifically used to perform calculations based on floating-point data so that the output softmax value is floating-point data.

3. The computing device according to claim 1, characterized in that, When the first matrix and the second matrix are data in floating-point form, the tensor processor and the first computing unit are specifically used to perform operations based on the floating-point data. The second calculation unit is specifically used to look up the target function corresponding to the difference from a preset second correspondence table for the difference; wherein, the second correspondence table records the mapping relationship between floating-point data and the linear function corresponding to the floating-point data, and the function value of the linear function corresponding to the floating-point data is equal to the exponent value of the floating-point data; The second calculation unit is further configured to substitute the difference as an independent variable into the objective function to obtain the dependent variable value, and to determine the dependent variable value as the exponential operation result corresponding to the element; The third computing unit, the vector computing module, and the second pipeline module are specifically used to perform calculations based on floating-point data so that the output softmax value is floating-point data.

4. The computing device according to claim 1, characterized in that, The computing device further includes a memory; wherein... The memory is used to store the calculation results of each calculation.

5. The computing device according to claim 4, characterized in that, The memory includes a first storage module corresponding to the tensor processor and a second storage module corresponding to the vector processor; wherein... The first storage module is used to store the calculation result and the maximum value; The second storage module is used to store the exponential operation results corresponding to all elements in the operation result, as well as the cumulative sum.

6. The computing device according to claim 5, characterized in that, The second storage module includes storage units and cache units; wherein, The cache unit is used to cache the exponentiation results corresponding to all elements in the calculation result, as well as the cumulative sum; The storage unit is used to store the result of the exponential operation and the summation; wherein the data stored in the storage unit is in fixed-point format.

7. A data processing method, characterized in that, The data processing method utilizes the computing device according to any one of claims 1-6 to perform data processing, the method comprising: The tensor processor performs matrix multiplication on the input first and second matrices to obtain the result, and extracts the maximum value in the specified dimension from the result. The first calculation unit calculates the difference between the currently flowing element and the maximum value; wherein, each element in the calculation result flows into the first calculation unit in sequence; The second calculation unit performs an exponential operation on the difference to obtain the exponential result corresponding to the element. fruit; The third calculation unit performs a cumulative sum operation on all exponent operation results corresponding to all elements in the calculation result to obtain the cumulative sum. The vector calculation module calculates the reciprocal of the accumulated sum and uses the reciprocal as a normalization factor for the activation operation; The second pipeline module performs vector multiplication on the exponential operation result corresponding to each element and the normalization factor to obtain the softmax value corresponding to each element.

8. The method according to claim 7, characterized in that, When the first matrix and the second matrix are data in fixed-point form, the tensor processor and the first computing unit perform operations based on the data in fixed-point form. The second calculation unit, for the difference, looks up the exponent value corresponding to the difference from a preset first correspondence table, and uses the found exponent value as the result of the exponent operation corresponding to the element; wherein, the first correspondence table is used to record the mapping relationship between fixed-point data and the floating-point form exponent value corresponding to the fixed-point data; The third computing unit, the vector computing module, and the second pipeline module perform calculations based on floating-point data so that the output softmax value is in floating-point format.

9. The method according to claim 7, characterized in that, When the first matrix and the second matrix are data in floating-point form, the tensor processor and the first computing unit perform operations based on the floating-point data. The second calculation unit searches for the target function corresponding to the difference from a preset second correspondence table; wherein, the second correspondence table records the mapping relationship between floating-point data and the linear function corresponding to the floating-point data, and the function value of the linear function corresponding to the floating-point data is equal to the exponent value of the floating-point data; The second calculation unit substitutes the difference as an independent variable into the objective function to obtain the dependent variable value, and determines the dependent variable value as the result of the exponential operation corresponding to the element; The third computing unit, the vector computing module, and the second pipeline module perform calculations based on floating-point data so that the output softmax value is in floating-point format.

10. A computing chip, characterized in that, The computing chip includes the computing device according to any one of claims 1-6.

Images