Operator execution method, device, computer equipment and storage medium

Abstract

The present application discloses an operator execution method, apparatus, computer equipment, and storage medium, belonging to the field of artificial intelligence technology. In this method, after obtaining the output result of the first precision of the first operation in the fusion operator, the output result is stored in a TLR array according to a preset data arrangement method. Based on the preset data arrangement method, the data in the TLR array is converted from the first precision to the second precision, the second precision being the input precision corresponding to the second operation in the fusion operator. After determining that the converted data meets the data arrangement requirements corresponding to the second operation, the data is processed by each thread to obtain the calculation result of the fusion operator, and the processing is determined according to the category of the second operation. In this way, the data arrangement method of the output result of the previous operation in the fusion operator is changed at the time of output, and the data arrangement requirements corresponding to the next operation can be met after the precision conversion, without having to exchange data between threads. The hardware overhead is relatively small, and thus the performance of the fusion operator can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an operator execution method, apparatus, computer equipment, and storage medium. Background Art

[0002] Generally, a fusion operator includes multiple tensor operations and scalar operations. When the data precision between adjacent operations in the fusion operator is different, it is necessary to implement precision conversion in the fusion operator. After the precision conversion, the data arrangement of the output result of the previous operator will change. The changed arrangement does not meet the data arrangement requirements corresponding to the next operator, and subsequent operations cannot be performed directly. In order not to affect subsequent operations, related technologies use multiple instructions for data exchange between threads to rearrange the data. However, the hardware overhead of data exchange between threads is relatively large, which will seriously affect the performance of the fusion operator. Summary of the Invention

[0003] Embodiments of the present application provide an operator execution method, apparatus, computer device, and storage medium to reduce the hardware overhead of data rearrangement in a fusion operator and improve the performance of the fusion operator.

[0004] In a first aspect, an embodiment of the present application provides an operator execution method, wherein a fusion operator includes a first operation and a second operation, wherein the output of the first operation corresponds to a first precision, and the input of the second operation corresponds to a second precision, and the method includes:

[0005] After obtaining the output result of the first operation, storing the output result in a thread local register (TLR) array according to a preset data arrangement when converting from the first precision to the second precision, where each row of TLRs in the TLR array corresponds to one thread;

[0006] Based on the preset data arrangement, converting the data in the TLR array from the first precision to the second precision;

[0007] When it is determined that the data in the TLR array after the precision conversion meets the data arrangement requirements corresponding to the second operation, the data in the TLR array is processed by each thread to obtain the calculation result of the fusion operator, wherein the processing is determined according to the category of the second operation.

[0008] In some embodiments, the output result includes Q rows of data, the TLR array includes M rows of TLRs, and the i-th row of TLRs in the TLR array corresponds to the i-th thread, where Q and M are integers greater than zero, and 0≤i<Q-1;

[0009] The preset data arrangement method satisfies: the data of the i-th row and the i+Q / 2-th row in the output result are arranged in the i-th row. k to (i+1) In the TLR corresponding to k threads, the data in row i and row i+Q / 2 are arranged in different TLR columns, and the data in rows 0 to Q / 2-1 are arranged in the same manner, and the data in rows Q / 2 to Q-1 are arranged in the same manner. k is the number of threads corresponding to each row of data in the output result;

[0010] When the first precision is lower than the second precision, the preset data arrangement method also satisfies: each row of data in the output result is arranged continuously by TLR column, and is sequentially arranged in the k TLRs corresponding to each TLR column until half of the storage of each TLR is empty, and then each TLR is sequentially arranged to be full.

[0011] In some embodiments, when the first precision is lower than the second precision, before converting the data in the TLR array from the first precision to the second precision based on the preset data arrangement, the method further includes:

[0012] expanding a TLR column in the TLR array according to a space required to store the output result of the second precision;

[0013] For each TLR column with data in the TLR array, half of the data in the TLR column is moved to an idle TLR column according to the rule of moving data between TLRs corresponding to the same thread, and the half of the data is the data occupying half of the storage space at the head of each TLR, or the half of the data is the data occupying half of the storage space at the tail of each TLR.

[0014] In some embodiments, when the first precision is higher than the second precision, the preset data arrangement method further satisfies: each row of data in the output result is arranged continuously by TLR group, and is arranged row by row in ascending order of row numbers in the k rows and 2 columns of TLRs corresponding to each TLR group, and one TLR is arranged after one TLR is fully arranged in the two TLRs in each row before another TLR is arranged.

[0015] In some embodiments, when the first precision is higher than the second precision, the method further includes:

[0016] If it is determined that the data in the TLR array does not meet the data arrangement requirement after the precision conversion, exchanging the data in the TLR array according to the data arrangement requirement and a rule for exchanging data in TLRs corresponding to the same thread; and

[0017] After determining that the data in the TLR array after the exchange meets the data arrangement requirement, the data in the TLR array is processed by each thread to obtain the calculation result of the fusion operator.

[0018] In some embodiments, after converting from the first precision to the second precision, each TLR has free storage space, and the data arrangement requirement is that each row of data in the output result is arranged continuously in TLR columns and no free storage space is left between TLR rows;

[0019] According to the data arrangement requirement and according to a rule for exchanging data in TLRs corresponding to the same thread, the data in the TLR array is exchanged, including:

[0020] Swapping the data in the TLR corresponding to each thread according to a rule that the free storage space of each TLR is full and consecutive data in the same row of the output result are arranged in each TLR;

[0021] According to the rule of not leaving free storage space between TLR rows, the data in the TLR corresponding to each thread is exchanged so that the data in the TLR array meets the data arrangement requirement.

[0022] In some embodiments, processing the data in the TLR array by each thread to obtain the calculation result of the fusion operator includes:

[0023] When the type of the second operation is a scalar operation, calculating the data in the TLR array through each thread to obtain a calculation result of the fusion operator;

[0024] When the category of the second operation is a tensor operation, the data in the TLR array is written into the tensor core through each thread, and the tensor core is used to perform a tensor operation on the written data to obtain a calculation result of the fusion operator.

[0025] In some embodiments, the first operation is a tensor operation and the second operation is a scalar operation, or the first operation is a tensor operation and the second operation is a tensor operation, or the first operation is a scalar operation and the second operation is a tensor operation, or the first operation is a scalar operation and the second operation is a scalar operation.

[0026] In a second aspect, an embodiment of the present application provides an operator execution device, wherein a fusion operator includes a first operation and a second operation, wherein the output of the first operation corresponds to a first precision, and the input of the second operation corresponds to a second precision, and the device includes:

[0027] an output module, configured to, after obtaining an output result of the first operation, store the output result in a thread local register (TLR) array according to a preset data arrangement when converting from the first precision to the second precision, where each row of TLRs in the TLR array corresponds to one thread;

[0028] a conversion module, configured to convert the data in the TLR array from the first precision to the second precision based on the preset data arrangement;

[0029] A processing module is used to process the data in the TLR array through each thread to obtain the calculation result of the fusion operator when it is determined that the data in the TLR array after the precision conversion meets the data arrangement requirements corresponding to the second operation, wherein the processing is determined according to the category of the second operation.

[0030] In a third aspect, an embodiment of the present application provides a computer device comprising a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, wherein the artificial intelligence chip implements any of the above-mentioned operator execution methods when executing the computer program.

[0031] In a fourth aspect, an embodiment of the present application provides a storage medium. When a computer program in the storage medium is executed by a processor of a computer device, the computer device can execute any of the above-mentioned operator execution methods.

[0032] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements any of the above-mentioned operator execution methods when executed by a processor.

[0033] In an embodiment of the present application, a fusion operator includes a first operation and a second operation, wherein the output of the first operation corresponds to a first precision, and the input of the second operation corresponds to a second precision. After obtaining the output result of the first operation, the output result is stored in a TLR array according to a preset data arrangement when converting from the first precision to the second precision. Each row of TLRs in the TLR array corresponds to a thread. Based on the preset data arrangement, the data in the TLR array is converted from the first precision to the second precision. When it is determined that the data in the TLR array after the precision conversion meets the data arrangement requirements corresponding to the second operation, the data in the TLR array is processed by each thread to obtain the calculation result of the fusion operator, wherein the processing is determined according to the category of the second operation. In this way, by changing the data arrangement of the output result of the previous operation in the fusion operator at the time of output, the data arrangement requirements corresponding to the next operation in the fusion operator can be met after the precision conversion is performed based on the preset data arrangement, and subsequent operations can be directly performed without exchanging data between threads. Therefore, the hardware overhead of data rearrangement can be saved, thereby improving the performance of the fusion operator. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0035] Figure 1 A schematic diagram of the structure of an artificial intelligence chip provided in an embodiment of the present application;

[0036] Figure 2 A schematic diagram of the output result of an operator provided in an embodiment of the present application;

[0037] Figure 3 A schematic diagram of an arrangement of output results in a TLR array in the related art;

[0038] Figure 4 A schematic diagram of data in a TLR array in the related art;

[0039] Figure 5 A schematic diagram of data in another TLR array in the related art;

[0040] Figure 6 It is a data schematic diagram of another TLR array in the related art;

[0041] Figure 7 It is a data schematic diagram of another TLR array in the related art;

[0042] Figure 8 A schematic diagram of the execution process of an operator execution method provided in an embodiment of the present application;

[0043] Figure 9 A schematic diagram of the arrangement of output results in a TLR array provided in an embodiment of the present application;

[0044] Figure 10 A schematic diagram of another arrangement of output results in a TLR array provided in an embodiment of the present application;

[0045] Figure 11 A method for Figure 9 Schematic diagram of the data after precision conversion;

[0046] Figure 12 A method for Figure 10 Schematic diagram of the data after precision conversion;

[0047] Figure 13 A schematic diagram of the execution process of another operator execution method provided in an embodiment of the present application;

[0048] Figure 14a and Figure 14b A schematic diagram of the arrangement of two output results in the TLR array provided in an embodiment of the present application;

[0049] Figure 15a 、 Figure 15b and Figure 15c Schematic diagram of the arrangement of the three output results in the TLR array provided in the embodiment of the present application;

[0050] Figure 16 A method for Figure 14a Schematic diagram after data exchange;

[0051] Figure 17 A schematic diagram of data exchange of 16 types of data provided in an embodiment of the present application;

[0052] Figure 18 A method for Figure 15a Schematic diagram after data exchange;

[0053] Figure 19 A schematic diagram of data exchange of 18 types of data provided in an embodiment of the present application;

[0054] Figure 20 A schematic diagram of the structure of an operator execution device provided in an embodiment of the present application;

[0055] Figure 21 A schematic diagram of the hardware structure of a computer device for implementing an operator execution method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to reduce the hardware overhead of data rearrangement in the fusion operator and improve the performance of the fusion operator, the embodiments of the present application provide an operator execution method, apparatus, computer equipment and storage medium.

[0057] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application can be combined with each other if there is no conflict.

[0058] To facilitate understanding of this application, the technical terms involved in this application are:

[0059] 1. Fusion operator.

[0060] A fusion operator typically includes multiple tensor and scalar operations. Tensor operations include matrix multiplication and convolution, while scalar operations include addition and subtraction. Compared to tensor operations, scalar operations have lower computational complexity. To improve the overall computational performance of the fusion operator, two types of hardware with different computational capabilities are typically used: tensor cores and scalar cores to handle tensor and scalar operations, respectively. Specifically, the tensor cores process tensor operations and output the results with appropriate precision, such as 32-bit floating point (FP32), to the scalar core's TLR. The scalar core then performs scalar operations on the data in the TLR.

[0061] Taking a fusion operator consisting of a matrix multiplication and a subsequent activation function (ReLU) as an example, the tensor core is responsible for quickly completing the matrix multiplication operation and outputting the operation result in FP32 to the TLR of the scalar core. Then, the scalar core applies ReLU to each data in the TLR to obtain the final calculation result of the fusion operator.

[0062] In this way, the calculation speed of the fusion operator can be guaranteed while ensuring the necessary numerical accuracy, thereby improving the overall calculation performance of the fusion operator.

[0063] 2. Data arrangement method.

[0064] The data arrangement method is used to describe how the output results of an operation such as matrix multiplication are arranged in the TLR array. Each row of TLRs in the TLR array corresponds to a thread, and each TLR is usually used to store FP32 data. The following is also introduced as an example of each TLR storing FP32 data. However, it should be noted that each TLR can also store data of other precisions such as FP8 or FP16, and as technology develops, each TLR may also store FP64 or FP128 data in the future.

[0065] 3. Precision conversion.

[0066] In artificial intelligence chips, the precision of the data processed is mostly floating-point, such as FP8, FP16, FP32, FP64, FP128, etc. In addition, precision conversion usually occurs between adjacent precisions, such as precision conversion between FP8 and FP16, between FP16 and FP32, between FP32 and FP64, or between FP64 and FP128.

[0067] 4. Data arrangement requirements.

[0068] Data layout requirements refer to the layout requirements when data is output.

[0069] In fusion operators, data output generally occurs in two scenarios: from the TLR array to the tensor core, and from the TLR array to shared memory. The former corresponds to the case where the next operator is a tensor operation, such as matrix multiplication, while the latter corresponds to the case where the next operation is a scalar operation. Therefore, the data layout requirements can be considered the same for the next operator. To simplify development, technicians set the data layout requirements for both scenarios to be the same.

[0070] As an example, the data arrangement requirement is that when data is output, each row of data is stored continuously in its corresponding k rows of TLRs by TLR columns and no free storage space is left between TLR rows.

[0071] When using artificial intelligence chips to implement large model calculations, in order to save memory bandwidth and reduce the delay in data exchange between main memory and caches at all levels, developers will fuse multiple small operators together to form a fused operator.

[0072] Generally, a fusion operator includes multiple tensor and scalar operations. When the data precision of adjacent operations in a fusion operator differs, precision conversion is required within the fusion operator. This is because the data layout of the output results of the previous operator changes after precision conversion. This changed layout does not meet the data layout requirements of the next operator, making subsequent operations such as output to shared memory or output to Tensor Cores impossible.

[0073] To avoid affecting subsequent operations, related techniques use instructions for exchanging data between multiple threads to rearrange data. The following describes related techniques using an example where a fused operator includes matrix multiplication and addition, with the output precision of the matrix multiplication being FP32 and the input precision of the addition being FP16.

[0074] See also Figure 1 , Figure 1 A schematic structural diagram of an artificial intelligence chip 100 provided in an embodiment of the present application includes a tensor core, a scalar core, and a load store cache (LSC). The tensor core and the scalar core can communicate with each other through an interface, and the scalar core and the load store cache can also communicate with each other through an interface.

[0075] In related technologies, the tensor core performs matrix multiplication operations to obtain FP32 output results. Assume that the output result is as follows Figure 2 The 16 rows and 16 columns of data shown are Figure 2 R represents the row, C represents the column, and RiCi represents the data in the i-th row and j-th column of the output result. Both i and j are numbered from 0 to 15.

[0076] Afterwards, the tensor core can store the output results in the TLR array of the scalar core. The i-th row TLR in the TLR array corresponds to the i-th thread, and each TLR can store one FP32 data. Assume that the TLR array contains 32 rows and 8 columns of TLR, that is, there are 32 threads, each thread corresponds to a row of TLR, and each row of TLR is numbered from 0 to 7. TLRi represents the i-th column TLR. Then, Figure 2 The data shown in the TLR array is arranged as follows Figure 3 As shown, part of the data is displayed with a gray background to facilitate the distinction of the same-same data in the output results.

[0077] Afterwards, the scalar core can Figure 3 The data shown is converted from FP32 to FP16, at which point half of the storage space is freed up in each TLR.

[0078] The scalar core can then use the shuffle instruction (an instruction used to exchange data between threads) to Figure 3 For example, the data in the TLR corresponding to thread 1 is exchanged with the TLR corresponding to thread 0, the data in the TLR corresponding to thread 3 is exchanged with the TLR corresponding to thread 2, and the data in the TLR corresponding to thread 31 is exchanged with the TLR corresponding to thread 30. After the exchange, the data in the TLR array is as follows: Figure 4 shown.

[0079] Then, the scalar core can continue to use the shuffle instruction to Figure 4 For example, the data in the TLR corresponding to thread 2 is exchanged with the TLR corresponding to thread 1, the data in the TLR corresponding to thread 6 is exchanged with the TLR corresponding to thread 5, and the data in the TLR corresponding to thread 30 is exchanged with the TLR corresponding to thread 29. The data in the TLR array after the exchange is as follows: Figure 5 shown.

[0080] The scalar core can then continue to use the shuffle instruction to Figure 5 For example, the data in the TLRi corresponding to thread 0 is exchanged with the TLRi corresponding to thread 2, the data in the TLRi corresponding to thread 1 is exchanged with the TLRi corresponding to thread 3, etc. The data in the TLRi corresponding to thread 28 is exchanged with the TLRi corresponding to thread 30, and the data in the TLRi corresponding to thread 29 is exchanged with the TLRi corresponding to thread 31, where i is an odd number (i.e., 1, 3, 5, 7). The data in the TLR array after the exchange is as follows: Figure 6 shown.

[0081] Then, the scalar core can exchange the data in the TLR corresponding to the same thread. After the exchange, the data in the TLR array is as follows: Figure 7 As shown, Figure 7 The data shown meets the data arrangement requirements corresponding to the addition operation.

[0082] Finally, the Arithmetic Logic Unit (ALU) in the scalar core uses each thread to Figure 7 The data shown are added to obtain the calculation result of the fusion operator.

[0083] From the above process, we can see that after the precision conversion, three inter-thread data exchanges are performed through three shuffle instructions. Figure 1 In practical applications, each time data is exchanged between threads using the shuffle instruction, the data must be stored from the TLR array to the shared memory of the load-store cache. The data is then rearranged within the load-store cache buffer and stored back to the TLR array via shared memory. This process of three thread data exchanges must be repeated three times, which is not only time-consuming but also occupies TLR resources for a long time, extending the total runtime of the fusion operator and severely affecting the performance of the fusion operator.

[0084] In order to solve the above problems, the inventors came up with the idea of ​​designing a special data arrangement method (i.e., a preset data arrangement method). When the output result of the previous operation in the fusion operator is output using the preset data arrangement method, the precision conversion based on the preset data arrangement method can directly meet the data arrangement requirements corresponding to the next operation in the fusion operator, and subsequent operations can be performed directly without exchanging data between threads. Therefore, the hardware overhead of data rearrangement can be saved, thereby improving the performance of the fusion operator.

[0085] The execution subject of the operator execution method provided in this application can be an artificial intelligence chip 100, such as a graphics processing unit (GPU), a general-purpose computing on graphics processing units (GPGPU), a domain specific architecture (DSA), etc. The structure of the artificial intelligence chip 100 can be found in Figure 1 , which will not be described in detail here. Furthermore, AI chip 100 includes both tensor cores and scalar cores, so the steps in this method can be partially executed by the tensor cores and partially executed by the scalar cores. The following describes the method in which the steps are partially executed by the tensor cores and partially executed by the scalar cores.

[0086] See also Figure 8 , Figure 8 A schematic diagram of the execution process of an operator execution method provided in an embodiment of the present application includes the following steps.

[0087] In step 801 , the tensor core executes the first operation in the fusion operator and obtains an output result of the first precision.

[0088] The first operation may be a tensor operation such as a matrix multiplication operation, or a scalar operation such as an addition operation or a subtraction operation, and the output result of the first operation is usually in a matrix form, that is, it contains multiple rows and columns of data.

[0089] In step 802 , the tensor core stores the output result in a TLR array according to a preset data arrangement when converting from the first precision to the second precision, where each row of TLRs in the TLR array corresponds to one thread.

[0090] The second precision is the input precision of the second operation in the fusion operator, the second operation is the next operation of the first operation, and the second operation can be a tensor operation or a scalar operation.

[0091] Assume that the output result of the first operation includes Q rows and P columns of data, the TLR array includes M rows and N columns of TLRs, and the i-th row of TLRs in the TLR array corresponds to the i-th thread, where Q, P, M, and N are integers greater than zero, and 0≤i<Q-1.

[0092] In practical applications, Q can be 8, 16, or 32, and M can be twice Q. The following explanation uses Q as an example. When Q is 16, M is 32, and P can be 16, 32, or 64. N is determined by M, the storage space of a single TLR, and the storage space required for the output result of the first operation.

[0093] Then, the preset data arrangement method satisfies: the data of row i and row i+Q / 2 in the output result are arranged in row i. k to (i+1) In the TLR corresponding to k threads, the data in row i and row i+Q / 2 are arranged in different TLR columns, and the data in rows 0 to Q / 2-1 are arranged in the same way, and the data in rows Q / 2 to Q-1 are arranged in the same way. k is the number of threads corresponding to each row of data in the output result, such as k=4.

[0094] Furthermore, when the first precision is lower than the second precision, the preset data arrangement method may further satisfy the following conditions: each row of data in the output result is arranged continuously by TLR column, and is sequentially arranged in the k TLRs corresponding to each TLR column until each TLR is half empty, and then sequentially arranged to fill each TLR. When the data is arranged continuously by TLR column, the arrangement order between the TLR columns can be arbitrary. That is, when each row of data is arranged in multiple TLR columns, the arrangement order of the multiple TLR columns can be arbitrary.

[0095] The following describes the preset data arrangement method when the first precision is lower than the second precision with reference to several examples.

[0096] Example 1: The fused operator includes matrix multiplication operation 1 and matrix multiplication operation 2 in sequence. The output precision of matrix multiplication operation 1 is FP16, and the input precision of matrix multiplication operation 2 is FP32.

[0097] Assume that the output of matrix multiplication operation 1 is as follows Figure 2 As shown, then, Figure 2 The preset data arrangement of the data in the TLR array can be as follows Figure 9 As shown, TLRi represents the TLR in the i-th column, and i ranges from 0 to 3. Taking row 0 of the output as an example, the data in row 0 is sorted into TLR0 and TLR2 corresponding to threads 0 to 3. The order of arrangement between TLR columns is: TLR0 first, then TLR2 (this order applies to rows 0 to 7). Therefore, row 0 is first sorted into TLR0 until each TLR is half empty, then into each TLR. Then, row 0 is sorted into TLR2 until each TLR is half empty, then into each TLR. Taking row 8 of the output as an example, the data in row 8 is sorted into TLR1 and TLR3 corresponding to threads 0 to 3. The order of arrangement between TLR columns is: TLR1 first, then TLR3 (this order applies to rows 8 to 15). Then, the 8th row of data is first arranged sequentially in TLR1 until half of the storage space of each TLR is empty, and then each TLR is filled sequentially. Then, it is arranged sequentially in TLR3 until half of the storage space of each TLR is empty, and then each TLR is filled sequentially.

[0098] It should be noted that Figure 9 Just for example, there can be other data arrangement methods, such as Figure 9 The data in the TLR column is exchanged at least once. Figure 9 For example, the data in TLR0 and TLR2 are exchanged, which is equivalent to changing the order of TLR columns of data from row 0 to row 7. Figure 9For example, the data in TLR1 and TLR3 are exchanged, which is equivalent to changing the order of TLR columns of data from row 8 to row 15. Figure 9 Taking the exchange of data in TLR0 and TLR1 as an example, it is equivalent to changing the TLR column arrangement order of the data in rows 0 to 7 and the TLR column arrangement order of the data in rows 8 to 15 at the same time.

[0099] Example 2: The fused operator includes a subtraction operation and a matrix multiplication operation. The output precision of the subtraction operation is FP8, and the input precision of the matrix multiplication operation is FP16.

[0100] Assume that the output of the scalar operation is as follows Figure 2 As shown, then, Figure 2 The preset data arrangement of the data in the TLR array can be as follows Figure 10 As shown, TLRi represents the TLR of the i-th column, and the value of i ranges from 0 to 1. In order to clearly show Figure 10 The data in the TLRs corresponding to threads 16 to 31 are plotted in another column. In fact, the TLRs corresponding to threads 16 to 31 are connected below the TLRs corresponding to threads 0 to 15. Taking row 0 of the output as an example, the data in row 0 is placed in TLR0 corresponding to threads 0 to 3. After being sequentially arranged in TLR0 until half of each TLR is empty, each TLR is then sequentially arranged to fill up. Taking row 8 of the output as an example, the data in row 8 is placed in TLR1 corresponding to threads 0 to 3. After being sequentially arranged in TLR1 until half of each TLR is empty, each TLR is then sequentially arranged to fill up.

[0101] It should be noted that Figure 10 The data shown is only an example, and there may be other data arrangements, such as Figure 10 The data in the TLR is exchanged between columns.

[0102] In step 803 , the scalar core converts the data in the TLR array from the first precision to the second precision based on a preset data arrangement.

[0103] Because more storage space is required after increasing precision, the TLR columns in the TLR array can be expanded based on the space required to store the output results of the second precision. Then, for each TLR column in the TLR array that has data, half of the data in this TLR column is moved to an idle TLR column according to the rules for moving data between TLRs corresponding to the same thread. This half of data can be the data occupying half of the storage space at the head of each TLR in this TLR column, or it can be the data occupying half of the storage space at the tail of each TLR in this TLR column. That is, for each TLR column that has data, the first half of the data in the i-th TLR in the TLR column can be moved to the i-th TLR in an idle TLR column, or the second half of the data in the i-th TLR in the TLR column can be moved to the i-th TLR in an idle TLR column. Moving half of the data can be accomplished within the TLR array without resorting to the load-store cache, resulting in relatively low instruction execution complexity.

[0104] The following introduces precision conversion with the help of the above examples.

[0105] For example 1: In order to Figure 9 The data shown is converted from FP16 to FP32, which requires expanding 4 columns of TLR. Then, the second half of the data of each TLR in the original 4 columns of TLR can be moved to an idle TLR column, and the following is obtained: Figure 11 The data shown is then converted to the required precision.

[0106] For example 2: In order to Figure 10 The data shown is converted from FP8 to FP16, which requires expanding 2 columns of TLR. Then, the second half of the data of each TLR in the original 2 columns of TLR can be moved to an idle TLR column. Figure 12 The diagram shows the arrangement of the data after the shift. Then, the data in the TLR array can be converted in precision.

[0107] In step 804, when the scalar core determines that the data in the TLR array after precision conversion meets the data arrangement requirements corresponding to the second operation, the scalar core processes the data in the TLR array through each thread to obtain the calculation result of the fusion operator. The processing is determined according to the category of the second operation.

[0108] The data arrangement requirement is that each row of data in the output result is stored continuously in TLR columns and no free storage space is left between TLR rows. In addition, when the second operation type is a scalar operation, the data in the TLR array can be calculated by each thread to obtain the calculation result of the fusion operator (corresponding to Figure 8804a); When the second operation type is a tensor operation, the data in the TLR array can be written into the tensor core through each thread, and the tensor core performs a tensor operation on the written data to obtain the calculation result of the fusion operator (corresponding to Figure 8 804b).

[0109] For example 1, Figure 11 The data layout requirements are met, so each thread can directly process the data in the TLR array to obtain the calculation results of the fusion operator. Specifically, the scalar core can store the data in the TLR array to the tensor core, which then performs the matrix multiplication operation 2 to obtain the calculation results of the fusion operator.

[0110] Similarly, for Example 2, Figure 12 The data layout requirements are met, so each thread can directly process the data in the TLR array to obtain the calculation results of the fusion operator. Specifically, the scalar core can store the data in the TLR array to the tensor core, which then performs the matrix multiplication operation to obtain the calculation results of the fusion operator.

[0111] In an embodiment of the present application, when the precision of different operations in a fused operator differs, a preset data arrangement is used to output the output data of the previous operator. Combined with an instruction that exchanges data within the TLR corresponding to the same thread (referring to an instruction that moves half of the data), this allows for fast and efficient conversion and data rearrangement from low-precision data to high-precision data. Because data exchange between TLRs corresponding to the same thread incurs significantly less hardware overhead than inter-thread TLR exchange, this reduces the hardware overhead of conversion and data rearrangement, thereby improving the performance of the fused operator. Furthermore, the reduced number of instructions and execution complexity can simplify the programming process and improve instruction execution efficiency.

[0112] See also Figure 13 , Figure 13 A schematic diagram of the execution process of another operator execution method provided in an embodiment of the present application includes the following steps.

[0113] In step 1301 , the tensor core executes the first operation in the fused operator and obtains an output result of the first precision.

[0114] The first operation may be a tensor operation such as a matrix multiplication operation, or a scalar operation such as an addition operation or a subtraction operation, and the output result of the first operation is usually in a matrix form, that is, it contains multiple rows and columns of data.

[0115] In step 1302 , the tensor core stores the output result in a TLR array according to a preset data arrangement when converting from the first precision to the second precision, where each row of TLRs in the TLR array corresponds to one thread.

[0116] The second precision is the input precision of the second operation in the fusion operator, the second operation is the next operation of the first operation, and the second operation can be a tensor operation or a scalar operation.

[0117] Assume that the output result of the first operation includes Q rows and P columns of data, the TLR array includes M rows and N columns of TLRs, and the i-th row of TLRs in the TLR array corresponds to the i-th thread, where Q, P, M, and N are integers greater than zero, and 0≤i<Q-1.

[0118] In practical applications, Q can be 8, 16, or 32, and M can be twice Q. The following explanation uses Q as an example. When Q is 16, M is 32, and P can be 16, 32, or 64. N is determined by M, the storage space of a single TLR, and the storage space required for the output result of the first operation.

[0119] Then, the preset data arrangement method satisfies: the data of row i and row i+Q / 2 in the output result are arranged in row i. k to (i+1) In the TLR corresponding to k threads, the data in row i and row i+Q / 2 are arranged in different TLR columns, and the data in rows 0 to Q / 2-1 are arranged in the same way, and the data in rows Q / 2 to Q-1 are arranged in the same way. k is the number of threads corresponding to each row of data in the output result, such as k=4.

[0120] Furthermore, when the first precision is higher than the second precision, the preset data arrangement method can also satisfy the following requirements: each row of data in the output result is arranged continuously by TLR group, and the k rows and two columns of TLRs corresponding to each TLR group are arranged row by row in ascending order of row number, with one TLR arranged after each row of two TLRs has been fully populated before the next TLR is arranged. Each TLR group includes two columns of TLRs, and the two columns of TLRs can be adjacent or non-adjacent in the TLR array. When arranged continuously by TLR group, the arrangement order between groups can be arbitrary. That is, when there are multiple TLR groups, the arrangement order between the multiple TLR groups can be arbitrary.

[0121] The following describes the preset data arrangement method when the first precision is higher than the second precision with reference to several examples.

[0122] Example 3: The fused operator includes matrix multiplication and subtraction operations in sequence. The output precision of the matrix multiplication operation is FP32, and the input precision of the subtraction operation is FP16.

[0123] Assume that the output of the matrix multiplication operation is as follows Figure 2 As shown, then, Figure 2The preset data arrangement of the data in the TLR array can be as follows Figure 14a In order to clearly display the data in the same row in the output results, Figure 14a Some data are marked in gray.

[0124] Taking the data in row 0 of the output result as an example, the data in row 0 is arranged in TLR0, TLR1, TLR4, and TLR5 corresponding to threads 0 to 3. Moreover, TLR0 and TLR1 are TLR group 0, TLR4 and TLR5 are TLR group 1, and the order of arrangement between groups is: TLR group 0 first, then TLR group 1 (the data in rows 0 to 7 follow this order of arrangement between groups). Then, the data in row 0 is first arranged row by row in the 4 rows and 2 columns of TLRs corresponding to TLR group 0, in ascending order of row numbers, and after one TLR is arranged in each row of 2 TLRs, another TLR is arranged. Then, the data in row 0 is arranged row by row in the 4 rows and 2 columns of TLRs corresponding to TLR group 1, in ascending order of row numbers, and after one TLR is arranged in each row of 2 TLRs, another TLR is arranged.

[0125] Taking the 8th row of data in the output result as an example, the 8th row of data is arranged in TLR2, TLR3, TLR6, and TLR7 corresponding to threads 0 to 3. In addition, TLR2 and TLR3 are TLR group 2, TLR6 and TLR7 are TLR group 3, and the order of arrangement between groups is: TLR group 2 first, then TLR group 3 (the data from rows 8 to 15 follow this order of arrangement between groups). Then, the 8th row of data is first arranged row by row in the 4 rows and 2 columns of TLRs corresponding to TLR group 2, in ascending order of row numbers, and after one TLR is arranged in each row of 2 TLRs, another TLR is arranged. Then, in the 4 rows and 2 columns of TLRs corresponding to TLR group 3, it is arranged row by row in ascending order of row numbers, and after one TLR is arranged in each row of 2 TLRs, another TLR is arranged.

[0126] It should be noted that Figure 14a The following is just an example. In fact, there are other ways to arrange data, such as Figure 14a The data in the TLR column is exchanged at least once. Figure 14a Take the data exchange between TLR1 and TLR5 as an example, and the result after exchange is as follows Figure 14b The data shown.

[0127] Example 4: The fused operator includes scalar operation 1 and scalar operation 2, and the output precision of scalar operation 1 is FP16, and the input precision of scalar operation 2 is FP8.

[0128] Assume that the output of the matrix multiplication operation is as follows Figure 2 As shown, then, Figure 2The preset data arrangement of the data in the TLR array can be as follows Figure 15a As shown. Taking the 0th row of data in the output result as an example, the 0th row of data is arranged in TLR0 and TLR1 corresponding to threads 0 to 3, and TLR0 and TLR1 are a TLR group. Then, the 0th row of data is arranged row by row in the k rows and 2 columns TLR corresponding to this TLR group in ascending order of row numbers, and after one TLR is fully arranged in the two TLRs in each row, the other TLR is arranged. Taking the 8th row of data in the output result as an example, the 8th row of data is arranged in TLR2 and TLR3 corresponding to threads 0 to 3, and TLR2 and TLR3 are a TLR group. Then, the 8th row of data is arranged row by row in this TLR group in ascending order of row numbers, and after one TLR is fully arranged in the two TLRs in each row, the other TLR is arranged.

[0129] It should be noted that Figure 15a The following is just an example. In fact, there are other ways to arrange data, such as Figure 15a The data in the TLR column is exchanged at least once. Figure 15a Taking the data exchange between TLR1 and TLR3 as an example, we can get the following result: Figure 15b The data shown. Figure 15a Taking the data exchange between TLR1 and TLR3 as an example, we can get the following result: Figure 15b The data shown is based on Figure 15a Taking the data exchange among TLR1, TLR2 and TLR3 as an example, we can get the following result: Figure 15c The data shown.

[0130] In step 1303 , the scalar core converts the data in the TLR array from the first precision to the second precision based on a preset data arrangement.

[0131] Since the storage space required will become less after the precision is reduced, each TLR in the TLR array will have free storage space, generally half of the free storage space.

[0132] In step 1304, when it is determined that the data in the TLR array after the precision conversion does not meet the data arrangement requirements corresponding to the second operation, the data in the TLR array is exchanged according to the data arrangement requirements and the rules for exchanging data in the TLR corresponding to the same thread.

[0133] The data arrangement requirement is that each row of data is stored contiguously by column in the TLRs corresponding to k threads, with no free storage space left between TLR rows. Therefore, the data in the TLRs corresponding to each thread can be swapped according to the rule that the free storage space in each TLR is fully occupied and the consecutive data in the same row of the output result is arranged in each TLR. Furthermore, the data in the TLRs corresponding to each thread can be swapped according to the rule that no free storage space is left between rows of the same TLR, so that the data in the TLR array meets the data arrangement requirement.

[0134] It should be noted that exchanging data in the TLRs corresponding to the same thread can be completed in the TLR array without the need for load-store cache. In other words, the two aforementioned exchanges of data in the TLRs corresponding to each thread can be performed directly in the TLR array, and the execution complexity of the instruction is relatively low.

[0135] The data exchange process is introduced below with reference to the above examples.

[0136] For example 3, Figure 14a For example, Figure 14a If the data arrangement requirements are not met, the data in the TLR corresponding to each thread can be swapped according to the rule that the free storage space of each TLR is full and the consecutive data in the same row of the output result are arranged in each TLR. For example, for the i-th thread, the data in each odd-numbered column TLR in the i-th row TLR corresponding to it is swapped to the adjacent previous even-numbered column TLR. The data in the TLR array after the swap is as follows Figure 16 shown.

[0137] Then, according to the rule of not leaving free storage space between TLR rows, Figure 16 The data in the TLR corresponding to the same thread are exchanged. For example, the data in TLR2 corresponding to each thread is exchanged to TLR1, the data in TLR4 is exchanged to TLR2, and the data in TLR6 is exchanged to TLR3. The data in the TLR array after the exchange is as follows Figure 17 shown.

[0138] It should be noted that Figure 14b It also does not meet the data arrangement requirements. Figure 14b The steps for exchanging the data shown are similar and will not be repeated here.

[0139] For example 4, Figure 15a For example, Figure 15aIf the data arrangement requirements are not met, the data in the TLR corresponding to each thread can be swapped according to the rule that the free storage space of each TLR is full and the consecutive data in the same row of the output result is arranged in each TLR. For example, the data in TLR1 of each thread is swapped to TLR0, and the data in TLR3 is swapped to TLR2. The data in the TLR array after the swap is as follows Figure 18 As shown (in order to clearly show Figure 18 The data in the TLRs corresponding to threads 16 to 31 are drawn in another column. In fact, the TLRs corresponding to threads 16 to 31 are connected below the TLRs corresponding to threads 0 to 15).

[0140] Then, according to the rule of not leaving free storage space between TLR rows, Figure 18 The data in the TLR corresponding to the same thread are exchanged, for example, the data in TLR2 corresponding to each thread is exchanged to TLR1. The data in the TLR array after the exchange is as follows Figure 19 As shown (in order to clearly show Figure 19 The data in the TLRs corresponding to threads 16 to 31 are drawn in another column. In fact, the TLRs corresponding to threads 16 to 31 are connected below the TLRs corresponding to threads 0 to 15).

[0141] It should be noted that Figure 15b 、 Figure 15c It also does not meet the data arrangement requirements. Figure 15b 、 Figure 15c The steps for exchanging the data shown are similar and will not be repeated here.

[0142] In step 1305 , the data in the TLR array is processed by each thread to obtain a calculation result of the fusion operator, and the processing is determined according to the type of the second operation.

[0143] Among them, when the type of the second operation is a scalar operation, the data in the TLR array can be calculated by each thread to obtain the calculation result of the fusion operator (corresponding to Figure 13 1305a); When the second operation type is a tensor operation, the data in the TLR array can be written into the tensor core through each thread, and the tensor core performs a tensor operation on the written data to obtain the calculation result of the fusion operator (corresponding to Figure 13 1305b).

[0144] For example 3, Figure 17 The data arrangement requirements have been met, so each thread can directly process the data in the TLR array to obtain the calculation result of the fusion operator. Specifically, the scalar core can use each thread to perform subtraction operations on the data in the TLR array to obtain the calculation result of the fusion operator.

[0145] Similarly, for Example 4, Figure 19 The data arrangement requirements have been met, so the data in the TLR array can be directly processed by each thread to obtain the calculation result of the fusion operator. Specifically, the scalar core can use each thread to perform scalar operation 2 on the data in the TLR array to obtain the calculation result of the fusion operator.

[0146] In an embodiment of the present application, when the precisions of different operations in a fused operator differ, a preset data arrangement is used to output the output data of the previous operator. This, combined with two instructions for exchanging data within the TLRs corresponding to the same thread (instructions for exchanging data twice within each thread's TLR), allows for fast and efficient conversion and data rearrangement from high-precision data to low-precision data. Because exchanging data between TLRs within the same thread requires significantly less hardware overhead than exchanging data between TLRs across threads, data rearrangement hardware overhead can be reduced, thereby improving the performance of the fused operator. Furthermore, the reduced number of instructions and execution complexity can simplify the programming process and improve instruction execution efficiency.

[0147] Based on the same technical concept, an embodiment of the present application also provides an operator execution device. The principle of solving the problem by the operator execution device is similar to that of the above-mentioned operator execution method. Therefore, the implementation of the operator execution device can refer to the implementation of the operator execution method, and the repeated parts will not be repeated.

[0148] See also Figure 20 , Figure 20 A schematic diagram of the structure of an operator execution device provided in an embodiment of the present application includes:

[0149] an output module 201 configured to, after obtaining an output result of the first operation, store the output result in a thread local register (TLR) array according to a preset data arrangement when converting from the first precision to the second precision, where each row of TLRs in the TLR array corresponds to one thread;

[0150] a conversion module 202, configured to convert the data in the TLR array from the first precision to the second precision based on the preset data arrangement;

[0151] The processing module 203 is used to process the data in the TLR array through each thread to obtain the calculation result of the fusion operator when it is determined that the data in the TLR array after the precision conversion meets the data arrangement requirements corresponding to the second operation, wherein the processing is determined according to the category of the second operation.

[0152] In some embodiments, the output result includes Q rows of data, the TLR array includes M rows of TLRs, and the i-th row of TLRs in the TLR array corresponds to the i-th thread, where Q and M are integers greater than zero, and 0≤i<Q-1;

[0153] The preset data arrangement method satisfies: the data of the i-th row and the i+Q / 2-th row in the output result are arranged in the i-th row. k to (i+1) In the TLR corresponding to k threads, the data in row i and row i+Q / 2 are arranged in different TLR columns, and the data in rows 0 to Q / 2-1 are arranged in the same manner, and the data in rows Q / 2 to Q-1 are arranged in the same manner. k is the number of threads corresponding to each row of data in the output result;

[0154] When the first precision is lower than the second precision, the preset data arrangement method also satisfies: each row of data in the output result is arranged continuously by TLR column, and is sequentially arranged in the k TLRs corresponding to each TLR column until half of the storage of each TLR is empty, and then each TLR is sequentially arranged to be full.

[0155] In some embodiments, when the first precision is lower than the second precision, the conversion module 202 is further configured to:

[0156] Before converting the data in the TLR array from the first precision to the second precision based on the preset data arrangement, expanding TLR columns in the TLR array according to a space required to store the output result of the second precision;

[0157] For each TLR column with data in the TLR array, half of the data in the TLR column is moved to an idle TLR column according to the rule of moving data between TLRs corresponding to the same thread, and the half of the data is the data occupying half of the storage space at the head of each TLR, or the half of the data is the data occupying half of the storage space at the tail of each TLR.

[0158] In some embodiments, when the first precision is higher than the second precision, the preset data arrangement method further satisfies: each row of data in the output result is arranged continuously by TLR group, and is arranged row by row in ascending order of row numbers in the k rows and 2 columns of TLRs corresponding to each TLR group, and one TLR is arranged after one TLR is fully arranged in the two TLRs in each row before another TLR is arranged.

[0159] In some embodiments, when the first precision is higher than the second precision, the method further includes:

[0160] The exchange module 204 is configured to exchange the data in the TLR array according to the data arrangement requirement and a rule for exchanging data in TLRs corresponding to the same thread, if it is determined that the data in the TLR array does not meet the data arrangement requirement after the precision conversion.

[0161] The processing module 203 is further configured to, after determining that the data in the TLR array after the exchange meets the data arrangement requirement, process the data in the TLR array through each thread to obtain a calculation result of the fusion operator.

[0162] In some embodiments, after converting from the first precision to the second precision, each TLR has free storage space, and the data arrangement requirement is that each row of data in the output result is arranged continuously by TLR column and no free storage space is left between TLR rows; the switching module 204 is specifically configured to:

[0163] Swapping the data in the TLR corresponding to each thread according to a rule that the free storage space of each TLR is full and consecutive data in the same row of the output result are arranged in each TLR;

[0164] According to the rule of not leaving free storage space between TLR rows, the data in the TLR corresponding to each thread is exchanged so that the data in the TLR array meets the data arrangement requirement.

[0165] In some embodiments, the processing module 203 is specifically configured to:

[0166] When the type of the second operation is a scalar operation, calculating the data in the TLR array through each thread to obtain a calculation result of the fusion operator;

[0167] When the category of the second operation is a tensor operation, the data in the TLR array is written into the tensor core through each thread, and the tensor core is used to perform a tensor operation on the written data to obtain a calculation result of the fusion operator.

[0168] In some embodiments, the first operation is a tensor operation and the second operation is a scalar operation, or the first operation is a tensor operation and the second operation is a tensor operation, or the first operation is a scalar operation and the second operation is a tensor operation, or the first operation is a scalar operation and the second operation is a scalar operation.

[0169] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, the functional modules in the embodiments of the present application may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The coupling between the modules can be achieved through some interfaces, which are usually electrical communication interfaces, but it is not ruled out that they may be mechanical interfaces or other forms of interfaces. Therefore, the modules described as separate components may or may not be physically separated, and may be located in one place or distributed to different locations of the same or different devices. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0170] After introducing the operator execution method and apparatus according to an exemplary embodiment of the present application, a computer device according to another exemplary embodiment of the present application is introduced next.

[0171] Based on the same technical concept, the embodiment of the present application provides a computer device, such as Figure 21 As shown, it includes at least one artificial intelligence chip 100 and a memory 200 connected to the at least one artificial intelligence chip 100. The specific connection medium between the artificial intelligence chip 100 and the memory 200 is not limited in the embodiment of the present application. Figure 21 For example, the artificial intelligence chip 100 and the memory 200 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0172] In an embodiment of the present application, the memory 200 stores instructions that can be executed by at least one artificial intelligence chip 100. At least one artificial intelligence chip 100 can execute the steps of the above-mentioned operator execution method by executing the instructions stored in the memory 200.

[0173] Among them, the artificial intelligence chip 100 is the control center of the computer device. It can use various interfaces and lines to connect various parts of the computer device, and realize operator execution by running or executing instructions stored in the memory 200 and calling data stored in the memory 200. Optionally, the artificial intelligence chip 100 may include one or more processing units. The artificial intelligence chip 100 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the artificial intelligence chip 100. In some embodiments, the artificial intelligence chip 100 and the memory 200 can be implemented on the same chip. In some embodiments, they can also be implemented separately on independent chips.

[0174] The artificial intelligence chip 100 can be a general-purpose processor, such as a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented and executed by a hardware processor, or by a combination of hardware and software modules in the processor.

[0175] The memory 200 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs, and modules. The memory 200 may include at least one type of storage medium, such as flash memory, a hard disk, a multimedia card, a card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), a magnetic storage device, a magnetic disk, an optical disk, and the like. The memory 200 is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer device, but is not limited thereto. The memory 200 in the embodiments of the present application may also be a circuit or any other device capable of performing a storage function, used to store program instructions and / or data.

[0176] In an exemplary embodiment, a storage medium is also provided. When a computer program stored in the storage medium is executed by a processor of a computer device, the computer device is capable of performing any of the above-described operator execution methods. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0177] In an exemplary embodiment, a computer program product is further provided. When the computer program product is executed by a computer device, the computer device can implement any exemplary method provided in this application.

[0178] Furthermore, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0179] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0180] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application also includes these modifications and variations.

Claims

1. An operator execution method, characterized in that: The fusion operator includes a first operation and a second operation, where the second operation is a next operation of the first operation, an output of the first operation corresponds to a first precision, and an input of the second operation corresponds to a second precision. The method includes: After obtaining the output result of the first operation, the artificial intelligence chip stores the output result in a thread local register TLR array according to a preset data arrangement method when converting from the first precision to the second precision, where each row of TLRs in the TLR array corresponds to one thread; Based on the preset data arrangement, converting the data in the TLR array from the first precision to the second precision; When it is determined that the data in the TLR array after the precision conversion meets the data arrangement requirements corresponding to the second operation, the data in the TLR array is processed by each thread to obtain the calculation result of the fusion operator, wherein the processing is determined according to the category of the second operation.

2. The method according to claim 1, wherein The output result includes Q rows of data, the TLR array includes M rows of TLRs, the i-th row of TLRs in the TLR array corresponds to the i-th thread, wherein Q and M are integers greater than zero, and 0≤i<Q-1; The preset data arrangement method satisfies: the data of the i-th row and the i+Q / 2-th row in the output result are arranged in the i-th row. k to (i+1) In the TLR corresponding to k threads, the data in row i and row i+Q / 2 are arranged in different TLR columns, and the data in rows 0 to Q / 2-1 are arranged in the same manner, and the data in rows Q / 2 to Q-1 are arranged in the same manner. k is the number of threads corresponding to each row of data in the output result; When the first precision is lower than the second precision, the preset data arrangement method also satisfies: each row of data in the output result is arranged continuously by TLR column, and is sequentially arranged in the k TLRs corresponding to each TLR column until half of the storage of each TLR is empty, and then each TLR is sequentially arranged to be full.

3. The method according to claim 1 or 2, wherein: When the first precision is lower than the second precision, before converting the data in the TLR array from the first precision to the second precision based on the preset data arrangement, the method further includes: expanding a TLR column in the TLR array according to a space required to store the output result of the second precision; For each TLR column with data in the TLR array, half of the data in the TLR column is moved to an idle TLR column according to the rule of moving data between TLRs corresponding to the same thread, and the half of the data is the data occupying half of the storage space at the head of each TLR, or the half of the data is the data occupying half of the storage space at the tail of each TLR.

4. The method according to claim 2, wherein When the first precision is higher than the second precision, the preset data arrangement method also satisfies: each row of data in the output result is arranged continuously by TLR group, and is arranged row by row in ascending order of row numbers in the k rows and 2 columns of TLRs corresponding to each TLR group, and one TLR is arranged after one TLR is fully arranged in the two TLRs in each row before another TLR is arranged.

5. The method according to claim 1 or 4, wherein: When the first precision is higher than the second precision, the method further includes: If it is determined that the data in the TLR array does not meet the data arrangement requirement after the precision conversion, exchanging the data in the TLR array according to the data arrangement requirement and a rule for exchanging data in TLRs corresponding to the same thread; and After determining that the data in the TLR array after the exchange meets the data arrangement requirement, the data in the TLR array is processed by each thread to obtain the calculation result of the fusion operator.

6. The method according to claim 5, wherein After converting from the first precision to the second precision, each TLR has free storage space, and the data arrangement requirement is that each row of data in the output result is arranged continuously in TLR columns and no free storage space is left between TLR rows; According to the data arrangement requirement and according to a rule for exchanging data in TLRs corresponding to the same thread, the data in the TLR array is exchanged, including: Swapping the data in the TLR corresponding to each thread according to a rule that the free storage space of each TLR is full and consecutive data in the same row of the output result are arranged in each TLR; According to the rule of not leaving free storage space between TLR rows, the data in the TLR corresponding to each thread is exchanged so that the data in the TLR array meets the data arrangement requirement.

7. The method according to claim 1, wherein Processing the data in the TLR array through each thread to obtain the calculation result of the fusion operator includes: When the type of the second operation is a scalar operation, calculating the data in the TLR array through each thread to obtain a calculation result of the fusion operator; When the category of the second operation is a tensor operation, the data in the TLR array is written into the tensor core through each thread, and the tensor core is used to perform a tensor operation on the written data to obtain a calculation result of the fusion operator.

8. The method according to claim 1, wherein The first operation is a tensor operation, the second operation is a scalar operation, or, The first operation is a tensor operation, the second operation is a tensor operation, or, The first operation is a scalar operation, the second operation is a tensor operation, or, The first operation is a scalar operation, and the second operation is a scalar operation.

9. An operator execution device, characterized in that: A fusion operator includes a first operation and a second operation, where the second operation is the next operation of the first operation, an output of the first operation corresponds to a first precision, and an input of the second operation corresponds to a second precision. The device is applied to an artificial intelligence chip and includes: an output module, configured to, after obtaining an output result of the first operation, store the output result in a thread local register (TLR) array according to a preset data arrangement when converting from the first precision to the second precision, where each row of TLRs in the TLR array corresponds to one thread; a conversion module, configured to convert the data in the TLR array from the first precision to the second precision based on the preset data arrangement; A processing module is used to process the data in the TLR array through each thread to obtain the calculation result of the fusion operator when it is determined that the data in the TLR array after the precision conversion meets the data arrangement requirements corresponding to the second operation, wherein the processing is determined according to the category of the second operation.

10. A computer device comprising a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, characterized in that: When the artificial intelligence chip executes the computer program, the steps of the method as claimed in any one of claims 1 to 8 are implemented.

11. A storage medium, characterized in that: When the computer program in the storage medium is executed by a processor of a computer device, the computer device can perform the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 8 when executed by a processor.

Images