Maximum value searching device, dot product operation device and tensor operation system

Abstract

The invention provides a maximum value searching device which is used for searching a maximum index in a plurality of integer source operands, the device comprises a first maximum value searching unit, and the first maximum value searching unit comprises a first one-hot encoder which is used for obtaining the plurality of source operands and generating one-hot codes corresponding to the source operands; the first bitwise or module is used for carrying out bitwise or operation on the one-hot codes of the plurality of source operands and obtaining an intermediate result, and the bitwise or module is consistent with the coding bit width of the one-hot encoder; the first leading zero search module is used for searching the intermediate result to obtain a leading zero search result, the format of the leading zero search result is a binary data format, and the data bit width of the leading zero search result is consistent with the bit width corresponding to the source operands; and the first bitwise negation module is used for carrying out bitwise negation on the leading zero search result to obtain the maximum value in the plurality of source operands. According to the invention, the time sequence and area requirements can be better met.

Description

Technical Field

[0001] The present invention relates to the field of computers, in particular to the field of computer data processing, and more particularly to a maximum exponent device, a dot product operation device and chip, and a computing device. Background Art

[0002] In the process of rapid development and application of artificial intelligence, we have to mention the hardware devices used for convolutional neural network (CNN) calculations. As we all know, the data corresponding to convolutional neural networks are mainly floating-point matrices. In the process of neural network training, the main thing involved is the multiplication and addition of floating-point matrices. At this time, we have to mention the tensor operation unit. The so-called tensor operation unit (Tensor Core) is a unit used for floating-point matrix multiplication and addition operations.

[0003] Among them, when the tensor operation unit performs the dot product (DP, Dot Product) operation of the row and column vectors in the floating-point matrix, before performing floating-point addition to obtain the final matrix element operation result, it is necessary to align the exponent sum corresponding to the mantissa product of the input row and column vectors to obtain a unified exponent sum. Before the exponent sum corresponding to each mantissa product is shifted and aligned, it is necessary to know the specific displacement amount, that is, the difference between the exponent sum corresponding to the mantissa product and the maximum exponent sum. This means that before the shift is performed, the maximum exponent in the mantissa product needs to be found. And because the floating-point matrix of the convolutional neural network is relatively large, each time the tensor unit is used for calculation, it is impossible to directly calculate the matrix completely, and only multiple batches of calculations can be performed. Each calculation obtains a partial result of the matrix. Therefore, when searching for the maximum exponent sum, each dot product unit not only needs to search for the maximum value of the exponent sum corresponding to the mantissa product of the row and column vector currently being calculated, but also needs to compare these exponent sums with the exponent sums of the intermediate results at the same position that have been calculated before in the floating-point matrix. Taking the current typical tensor computing GPU as an example, each DP needs to find the maximum exponent of 8+1 FP16 / BF16 (data dimension is 8). The maximum exponent logic (EMAX module, Exponent Max) and the logic of finding the shift value usually need to be completed within one clock cycle. Therefore, the EMAX logic is the key path of the entire tensor computing unit timing. The existing EMAX mainly adopts the general maximum exponent logic of the direct comparison method, which compares the exponents corresponding to all the mantissa products through comparison logic to obtain the maximum exponent. The higher the data dimension, the more complex the comparison logic.

[0004] However, due to the large number of DP instances (each set of row and column vectors corresponds to a dot product unit DP) (a tensor operation unit has 32 DPs), EMAX generates a large area overhead, which is about 30% of the entire DP area and about 15% of the entire Tensor Core area. Therefore, how to efficiently find the maximum exponent with a smaller area has become the key to the design of the tensor operation unit. Since the existing EMAX uses comparison logic, as the dimension of floating-point data increases, the EMAX module will only become more and more complex and cannot meet the timing and area requirements well. A better EMAX solution is urgently needed. Summary of the invention

[0005] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a new maximum value search device, dot product operation device and tensor operation system.

[0006] According to a first aspect of the present invention, a maximum value search device is provided, which is used to search for a maximum value among multiple integer source operands, and the device includes a first maximum value search unit, wherein the first maximum value search unit includes: a first one-hot encoder, used to obtain multiple source operands and generate a one-hot encoding corresponding to each source operand; a first bitwise OR module, used to perform a bitwise OR operation on the one-hot encodings of the multiple source operands and obtain an intermediate result, wherein the bitwise OR module is consistent with the encoding bit width of the one-hot encoder; a first leading zero search module, used to search the intermediate result to obtain a leading zero search result, wherein the format of the leading zero search result is a binary data format and the data bit width is consistent with the bit width corresponding to the multiple source operands; a first bitwise inversion module, used to perform a bitwise inversion on the leading zero search result to obtain the maximum value among the multiple source operands.

[0007] Preferably, the first one-hot encoder is 2 m bit-wide one-hot encoder; the first bitwise OR module includes 2 m A bit-wide OR gate unit, the input dimension of each bit of the OR gate unit is consistent with the dimension of the multiple source operands; wherein m represents the exponent bit width of the multiple floating-point data.

[0008] Preferably, the device further comprises: a comparison module, a second maximum value search unit, and a splicing module, wherein the second maximum value search unit comprises a second unique-hot encoder, a second bitwise OR module, a second leading zero search module, and a second bitwise inversion module; wherein the first maximum value search unit is configured to obtain a high-order source operand and calculate a high-order maximum value based on the high-order source operand, wherein the high-order source operand is a preset high-order bit among the multiple source operands; the comparison module is used to obtain a low-order mask based on the high-order maximum value, wherein the low-order mask comprises preset low-order bits among the multiple source operands after comparison processing, wherein the comparison processing refers to setting the low-order bits of the source operands whose high-order bits among the multiple source operands are inconsistent with the high-order maximum value to zero; the second maximum value search unit is used to obtain the preset low-order bits of the multiple source operands, perform an AND operation on the low-order mask and obtain a low-order source operand, and calculate a low-order maximum value based on the low-order source operand; the splicing module is used to splice the high-order maximum value and the low-order maximum value to obtain the maximum value among the multiple source operands.

[0009] Preferably, the first one-hot encoder is a 2h-bit width one-hot encoder; the first bitwise OR module includes a 2h-bit width OR gate unit; the second one-hot encoder is a 2 l bit-width one-hot encoder; the second bitwise OR module includes 2 l Bit width or gate unit; wherein h represents a preset high-order bit, l represents a preset low-order bit, and h+l is the exponent width of the multiple floating-point data.

[0010] Preferably, the device also includes: a general maximum value search unit and a selection module; wherein the first maximum value search unit is configured to calculate a first maximum value, which is the maximum value of some dimensions among the multiple source operands; the general maximum index unit is used to calculate a second maximum value using a direct comparison method, which is the maximum value of the remaining dimensions among the multiple source operands; and the selection module is used to select a final maximum value from the first maximum value and the second maximum value.

[0011] Preferably, the first maximum value search unit is used to calculate the maximum value of the high 16-dimensional data of the multiple source operands; the general maximum value search unit is used to calculate the maximum value of the low 16-dimensional data of the multiple source operations.

[0012] According to a second aspect of the present invention, there is provided a method for searching for the maximum exponent sum in a floating-point dot product operation, the method comprising: S1, obtaining multiple exponent sums corresponding to multiple mantissa products of floating-point numbers; S2, using the multiple exponent sums as source operands, and using the apparatus as described in the first aspect of the present invention to search for the maximum exponent sum among the multiple exponent sums.

[0013] According to a third aspect of the present invention, there is provided a dot product operation device for performing dot product operation of floating-point row and column vectors, the device comprising: a multiplier array for performing multiplication operation on the floating-point row and column vectors to obtain a plurality of mantissa products and their corresponding exponent sums corresponding to the row and column vectors; a maximum exponent module for calculating the difference between the exponent sum corresponding to each mantissa product and the maximum exponent sum, wherein the maximum exponent module comprises exponent sum logic, maximum exponent logic, and exponent difference logic, the exponent sum logic being used to obtain the exponent sum corresponding to the plurality of mantissa products, the maximum exponent logic being configured as the device as described in the first aspect of the present invention, and being used to search for the maximum exponent sum among the exponent sums corresponding to the plurality of mantissa products, the exponent difference logic being used to calculate the difference between the exponent sum corresponding to each mantissa product and the maximum exponent sum; an exponent alignment shift module for performing alignment shift operation on the exponent sums corresponding to the plurality of mantissa products based on the calculation result of the maximum exponent module; and a product accumulation module for performing accumulation operation on the mantissa products and their corresponding exponent sums after the alignment shift operation by the exponent alignment shift module to obtain the dot product of the floating-point row and column vectors.

[0014] According to a fourth aspect of the present invention, there is provided a tensor operation system, comprising: a scheduling unit and a tensor operation unit, wherein the scheduling unit is used to send matrix multiplication and addition instructions to the tensor operation unit; the tensor operation unit is used to perform matrix multiplication and addition operations based on the multiplication and addition instructions, wherein the tensor operation unit comprises a control module and a computing module, the control module is used to obtain and cache data to be operated upon receiving the matrix multiplication and addition instructions, and the computing module comprises a dot product operation array composed of a plurality of devices according to the third aspect of the present invention, and is used to perform dot product operations on the data to be operated cached by the control module after decoding and sparse matrix selection.

[0015] Compared with the prior art, the advantages of the present invention are: the present invention uses one-hot encoding for maximum value search, and the timing performance at each frequency is better than the baseline direct comparison method. As the frequency increases, the one-hot encoding method is less sensitive to timing. Compared with the optimal baseline solution in the same scenario, the area of ​​the present invention can be reduced by about 20.9% at 1.5GHz and about 32.8% at 1.6GHz, and the timing violation will not occur until the timing reaches 1.9GHz. In addition, the present invention can better adapt to matrix operations in different floating-point formats, can be flexibly promoted, and the search level can be adjusted according to timing and area requirements. It can also be applied to other application scenarios with maximum / minimum comparison of multiple source operands, and the potential PPA problem (PPA: Power Performance Area, used to measure the comprehensive performance of the chip in three directions: power consumption, performance, and area) is optimized to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0017] Figure 1 A schematic diagram of the basic structure of a tensor operation unit according to an embodiment of the present invention;

[0018] Figure 2 A schematic diagram of a maximum logic structure of implementing FP8 floating point dot product operations using a direct comparison method according to an embodiment of the present invention;

[0019] Figure 3 2 is a schematic diagram of the structure of a maximum value search device according to an embodiment of the present invention;

[0020] Figure 4 A schematic diagram of the working principle of a maximum value search device according to an embodiment of the present invention;

[0021] Figure 5 A schematic diagram of the structure and working principle of a two-stage maximum value search device according to an embodiment of the present invention;

[0022] Figure 6 A schematic diagram of the structure and working principle of a maximum value search device of a multiplexed general maximum value search module according to an embodiment of the present invention;

[0023] Figure 7 Schematic diagram of experimental results according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail by specific embodiments below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0025] As described in the background technology of the present invention, the maximum exponential logic of the direct comparison method used in the prior art occupies a large area. Faced with the emergence of new data formats, it is increasingly unable to meet the timing and area requirements. Therefore, a new maximum value search solution that meets the timing and reduces the area overhead is urgently needed.

[0026] In order to better understand the present invention, the basic structure of the tensor operation unit is first introduced. The tensor operation unit is used in data center chips for artificial intelligence and high-performance computing to enable highly parallel processing of massive data, while providing flexible mixed operation precision and accelerating the throughput of artificial intelligence network operations. Figure 1As shown, the tensor operation unit includes a control part and a calculation part, wherein the calculation part includes a dot product operation array composed of multiple dot product units. The process of operating a floating-point matrix can be summarized as follows: the scheduling unit sends a matrix multiplication and addition instruction to the tensor operation unit; the control part of the tensor operation unit sends a read request to the register and receives the returned data to be operated to the internal data cache; the cached data is sent to the dot product operation unit after decoding and sparse matrix selection; each dot product operation unit generates a partial result of the result matrix, and continues to select other data in the cache to calculate a new partial result (each element in the final matrix operation result is actually the accumulation of multiple partial results corresponding to the element position. Each dot product unit must accumulate the partial result before the same element position after each calculation until the result matrix required by the instruction is fully calculated). In the dot product operation, if the dimension of the row / column vector that can be processed by a dot product operation is n, the computing power of the dot product unit is called DP-n. Taking the typical tensor operation GPU chip mentioned in the background technology as an example, each dot product unit needs to process the dot product of the row vector (1x8) and the column vector (8x1) to produce a (1x1) operation result. The dot product operation power of the GPU is DP-8. In each clock cycle, the tensor operation unit of the GPU can be composed of 32 dot product units with 4 rows and 8 columns to produce a 4x8 matrix multiplication and addition result. It should be noted that when performing floating-point dot product operations, before performing floating-point addition to obtain the final matrix element operation result, it is necessary to align the exponent sum corresponding to the mantissa product of the input row and column vectors to obtain a unified exponent sum. Before shifting the exponent sum corresponding to each mantissa product, it is necessary to know the specific displacement, that is, the difference between the exponent sum corresponding to each mantissa product and the maximum exponent sum. The structure of the dot product unit still refers to Figure 1 As shown, each dot product unit includes a multiplier array, a maximum exponent module, an exponent alignment displacement module, and a multiplication accumulation module, wherein the multiplier array is used to perform multiplication operations on floating-point row and column vectors to obtain the mantissa products of the row and column vectors, the maximum exponent module is used to find the maximum exponent sum in the exponent sums corresponding to the mantissa products and calculate the difference between the exponent sum corresponding to each mantissa product and the maximum exponent sum, the exponent alignment displacement module is used to perform displacement alignment operations on the exponent sum corresponding to each mantissa product according to the difference between the exponent sum corresponding to each mantissa product calculated by the maximum exponent module and the maximum exponent sum; the multiplication accumulation module is used to perform accumulation operations on the mantissa products after the displacement operations to obtain the dot products of the floating-point row and column vectors. Since the specific process of the dot product operation of the floating-point row and column vectors and the basic principle of the dot product unit are technologies known to those skilled in the art, they will not be described in detail here, and the maximum exponent module will be introduced in detail below.

[0027] The maximum exponent module is the most critical part of the dot product unit, which involves exponent sum logic, maximum exponent logic, and exponent difference logic. Since the maximum exponent logic and exponent difference logic in the maximum exponent module usually need to be completed within one clock cycle, the maximum exponent logic is the critical path of the entire tensor operation unit in timing, and the design complexity is different for different floating-point types. Still taking the current typical tensor operation GPU chip mentioned in the background technology as an example, each dot product operation unit needs to find the maximum exponent sum of 8+1 FP16 / BF16, and for other data formats, the corresponding data dimensions are different, and it may be necessary to find 16+1 maximum exponent sums, 32+1 maximum exponent sums or even more. Among them, the more typical one is the FP8 floating-point data format, which is introduced into AI applications based on the Transformer model to increase the computing power of Attention matrix calculations. For floating-point matrices in FP8 data format, each dot product unit of the tensor operation unit needs to process the inner product of the 32-dimensional FP8 vector, and EMAX needs to select the maximum value from 33 indices, which further increases the difficulty of EMAX. The existing general solution is the direct comparison method, that is, to obtain the maximum value after grouping multi-level comparison. Taking the three-choice logic as an example, EMAX (the input is represented by s0, s1, and s2, and the three-choice logic needs to compare s0 vs s1, s0 vs s2, and s1 vs s2 respectively to obtain the maximum value among s0, s1, and s2) contains 3 2-input comparators and 1 3mux1 logic. Similarly, 4-choice-1 consists of 6 2-input comparators and 1 4mux1. Extending to n-choice-1, a total of n(n-1) / 2 2-input comparators and 1 n-to-1 selector are required.

[0028] Due to timing constraints, at most two levels of comparison can be performed. Assume that multiple m-select-1 logic solutions are used to complete the maximum value comparison of FP8, such as Figure 2 As shown, the first layer needs 10 3-choose-1 and 1 2-choose-1, and the second layer needs 1 12-choose-1. Since the principle of direct comparison method is known to those skilled in the art, it is also easy to understand that the maximum value logic can be realized by multiple 3-choose-1 schemes, so it will not be explained here. Figure 2 Provide an expanded description.

[0029] Due to timing constraints, under the general direct comparison method, other possible combination schemes are traversed, and the area constraints are considered under the combination that meets the timing. The optimal combination among these possible combination schemes is shown in Table 1:

[0030] Table 1

[0031] First floor First level selection quantity First floor last Second floor Total area cost Choose 1 from 4 7 Choose 1 from 5 8 out of 1 80 2-to-1 comparators

[0032] As can be seen from Table 1, for the dot product unit of the FP8 floating-point data format, the optimal solution among all the maximum value logic solutions is seven 4mux1s plus one 5mux1 in the first stage and one 8mux1 in the second stage. After 1.5G frequency synthesis, the area overhead of the tensor operation unit containing 32 dot product units corresponding to the FP8 floating-point data format is relatively high (about 30%), which is a very large area overhead for the dot product unit, so further optimization is needed.

[0033] In view of this, the present invention proposes a maximum value search scheme based on one-hot encoding, which can achieve more efficient finding of the maximum value when there are more sources involved in the comparison, and reduce the timing and surface-based overhead. Among them, in order to better understand the present invention, the concept of one-hot encoding is first introduced. One-hot encoding is also called one-bit effective encoding. Its method is to use an N-bit state register to encode N states, each state has its own independent register bit, and at any time, only one of them is valid. For example, when a string of binary values ​​with a width of 3 bits is one-hot encoded, the one-hot encodings corresponding to 000, 001, 010, 011, 100, 101, 110, and 111 are: 00000001 (the number of trailing 0s is 0), 00000010 (the number of trailing 0s is 1), 00000100 (the number of trailing 0s is 2), 00001000 (the number of trailing 0s is 3), 00010000 (the number of trailing 0s is 4), 00100000 (the number of trailing 0s is 5), 01000000 (the number of trailing 0s is 6), and 10000000 (the number of trailing 0s is 7). From the characteristics of one-hot encoding, one-hot encoding can be simply understood as using the number of trailing 0s in the one-hot encoding to represent the value of the encoded value. The cost of one-hot encoding is determined by the maximum encoded binary value. For example, in the above example, the maximum encoded binary value is 111, and the one-hot encoding cost is 8 bits, that is, 2 3 Since one-hot encoding is a technique known to those skilled in the art, it will not be described in detail here.

[0034] Based on the characteristics of one-hot encoding, according to one embodiment of the present invention, Figure 3As shown, a maximum value search device of the present invention includes: a one-hot encoder, a bitwise OR module, a leading zero search module, and a bitwise inversion module, wherein the one-hot encoder is used to obtain multiple source operands and generate a one-hot encoding corresponding to each source operand; the bitwise OR module is used to perform a bitwise OR operation on the one-hot encodings of the multiple source operands and obtain an intermediate result, wherein the bitwise OR module is consistent with the encoding bit width of the one-hot encoder; the leading zero search module is used to search the intermediate result to obtain a leading zero search result, wherein the format of the leading zero search result is a binary data format and the data bit width is consistent with the bit width corresponding to the multiple source operands; the bitwise inversion module is used to perform a bitwise inversion on the leading zero search result to obtain the maximum value of the multiple source operands. When applied to the dot product operation of floating-point row and column vectors, the exponential sum corresponding to the mantissa product of the row and column vectors is used as the source operation, and the maximum exponential sum can be obtained by searching using the maximum value search device of the present invention, without the need for multi-level comparison through multiple comparators, which can meet both timing constraints and area constraints.

[0035] Furthermore, the present invention takes into account the reusability of the maximum value search device in floating point dot product operations, so that it can support the calculation of multiple data types. According to one embodiment of the present invention, the maximum value search device of the present invention is designed with the largest floating point dimension. Specifically, for operations in floating point formats such as FP32 / TF32 / FP16 / BF16 / FP8, the dimensions of a vector dot product are different. For the above data formats, the dimensions of each dot product (i.e., the comparison dimensions) are: 4 / 8 / 16 / 16 / 32. If the maximum value search device of the corresponding exponent sum of each data type is designed independently, it will generate a huge area overhead. If multiplexing is considered, it is necessary to be compatible with the largest comparison dimension (FP8) data. Therefore, in the embodiment of the present invention, the exponent sum corresponding to the mantissa product of floating point numbers in FP8 format is mainly used for expansion and explanation. Other data formats are similar and will not be described in detail.

[0036] Since FP8 has two types, E5M2 or E4M3, the exponent width of floating-point numbers in FP8 format is small, only 4 bits or 5 bits. For this type of data, the sum of the exponents corresponding to the product of the row and column vector mantissas is only 6-bit (two 5-bit exponents added together), and the cost of one-hot encoding is 64-bit (2 6-b it). According to one embodiment of the present invention, when the maximum value search device of the present invention is used to implement the maximum value search in the FP8 format, the one-hot encoder is designed to be 64 bits wide; the bitwise OR module is designed to be an OR gate unit with a 64-bit width. Since the floating-point number in the FP8 format has 32 dimensions, plus the previous intermediate results, the bitwise OR module needs to perform a bitwise OR operation on the 33-dimensional 64-bit wide data, and each bit implements an OR operation on the bits at the same position in the multidimensional data. Since the OR gate unit logic is a technology known to those skilled in the art, it will not be described in detail here. When the maximum value search device in this embodiment is used to search for the maximum value of the exponent and corresponding to the product of the mantissas of the row and column vectors in the FP8 format, as Figure 4 As shown, the data processing process is: the exponent sum corresponding to the product of the 32 mantissas of the current row and column vectors and the exponent sum of the partial results calculated before by the dot product unit are used as source operands and the 33 source operands are encoded by a one-hot encoder; the one-hot encoding of the source operands is obtained by bitwise OR operation to generate an intermediate result. Due to the characteristics of the one-hot encoding, the intermediate result retains the exponent information in each source operand, and the larger exponent bit will be closer to the highest bit of the intermediate result; the intermediate result is searched for leading zeros and then bitwise inverted to obtain the maximum value in the source operand. According to an example of the present invention, still referring to Figure 4 , taking the maximum exponent sum of 60 as an example, after one-hot encoding, we get a 64-bit binary value of 00010000...0 (three leading 0s, one 1, and 60 trailing 0s). After bitwise OR operation on all one-hot encodings and leading zero search, the leading zero search result is 3, which is represented by FP8 exponent and maximum bit width as 000011. The leading zero search result is 111100 after bitwise inversion, and its corresponding decimal number is the maximum exponent sum of 60. Then, the maximum exponent sum is sent to the exponent difference logic to obtain the difference between the exponent of each mantissa product and the maximum exponent, thereby obtaining the displacement.

[0037] The above embodiment describes the implementation scheme of one-level comparison based on one-hot encoding. After research, the inventors found that truncating the source operand into multiple levels according to the timing constraints can further reduce the area overhead, such as truncating into two levels, three levels or even more levels. According to one embodiment of the present invention, Figure 5As shown, a maximum value search device of the present invention includes: two layers of maximum value search units (for the convenience of description, respectively referred to as the first layer maximum value search unit and the second layer maximum value search unit), a comparison module, and a splicing module. The first layer maximum value search unit is configured to obtain a high-order source operand and calculate a high-order maximum value based on the high-order source operand, wherein the high-order source operand is a preset high-order bit (represented by k) among the multiple source operands, wherein, in the floating-point dot product operation in the FP8 format, the source operand is the exponential sum corresponding to the FP8 mantissa product (referred to as the FP8 exponential sum), and the high-order source operand is the preset high-order bit in the exponential sum corresponding to the mantissa product; the comparison module is used to obtain a low-order mask (referred to as b based on the high-order maximum value) it-mask), the low-order mask includes preset low-order bits (nk bits) in the source operand after comparison processing, wherein the comparison processing refers to setting the low-order bits of the source operands whose high-order bits in the multiple source operands are inconsistent with the high-order maximum value to zero. Since the source operands to be compared are compared in stages, multiple identical high-order bits may appear when comparing high-order bits. Therefore, the present invention can only enter the low-order bits in the source operands with consistent high-order bits into the second layer for low-order maximum value search through comparison processing. Of course, if there is only one maximum value in the high-order bit search , then there is no need to enter the second layer to search for low-order bits, and the source operand corresponding to the high-order maximum value can be directly used as the maximum value; the second-layer maximum value search unit is used to obtain the preset low-order bits in the multiple source operands, perform an AND operation with the low-order mask to obtain the low-order source operand, and calculate the low-order maximum value based on the low-order source operand; the splicing module is used to splice the high-order maximum value and the low-order maximum value to obtain the maximum value of the multiple source operands, wherein, in the dot product operation of floating-point numbers in FP8 format, the FP8 global maximum exponent sum is obtained by splicing the high-order maximum value and the low-order maximum value. For the exponential sum corresponding to the product of the mantissa of the row and column vectors of floating-point data in FP8 format, the exponential sum can be divided into two levels of comparison such as 5 bits + 1 bit, 1 bit + 5 bits, 4 bits + 2 bits, 2 bits + 4 bits, and 3 bits + 3 bits. The optimal combination in a specific scenario is optimally selected based on the timing constraints and area constraints in the specific scenario. In addition, there may be other multi-level schemes, such as 1 bit + 2 bits + 3 bits, 2 bits + 2 bits + 2 bits, 1 bit + 1 bit + 4 bits, 2 bits + 1 bit + 3 bits, etc., or even a four-level scheme. These multi-level schemes are similar to the two-level scheme in principle. They all compare the maximum values ​​of each level and then connect them to obtain the global maximum value. We will not explain them in detail here.

[0038] The above embodiment describes a maximum value search scheme based only on a one-hot encoder. The inventors have found through research that if the existing general maximum exponent logic is reused, the area overhead can be further reduced, and the exponent and maximum value searches corresponding to different floating-point data formats such as FP32 / TF32 / FP16 / BF16 / FP8 can be better compatible. According to one embodiment of the present invention, Figure 6 As shown, the maximum value search device that reuses the general maximum value search module includes a newly added FP8 maximum value search module (the FP8 maximum value search module here is designed with the same principle as the maximum value search module described in the aforementioned embodiment of the present invention, except that the data dimension processed by the FP8 maximum value search module here is different, and it can also be used for maximum value search in 6-bit width data, whether it is the exponent sum corresponding to the product of the mantissa of a floating point number or integer data) and an existing general maximum value search module (an existing maximum value search module based on the direct comparison method, for example, under the prior art, the maximum exponent logic used by the dot product unit in the tensor operation unit is a maximum value search module based on the direct comparison method. Since the scheme for maximum value search based on the direct comparison method is a technology known to those skilled in the art, it will not be described in detail here, and the general maximum value search module is used to refer to it), wherein the FP8 maximum value search module can be used for high 16-dimensional operands of FP8 The maximum value search of the high 16-dimensional operands in DP32 (i.e., the exponent sum corresponding to the high 16-dimensional mantissa product of FP8) or the maximum value search of the high 16-dimensional operands in DP32 (i.e., the exponent sum corresponding to the high 16-dimensional mantissa product of DP32) is performed. At this time, the data width processed by the FP8 maximum value search module is 6 bits and the data dimension is 17 dimensions. It is only necessary to adjust the input dimension of each OR gate unit in the bitwise OR module to adapt to the processing of 17-dimensional data; the universal maximum exponent module can be used to use a direct comparison method to perform a maximum value search on the low 16-dimensional operands of FP8 (i.e., the exponent sum corresponding to the low 16-dimensional mantissa product of FP8), the low 16-dimensional operands in DP32 (i.e., the exponent sum corresponding to the low 16-dimensional mantissa product of DP32), all 16-dimensional operands in DP16 (i.e., the exponent sum corresponding to all 16-dimensional mantissa products in DP16), or all 16-dimensional operands of TF32 / FP16 / BF16 (i.e., the exponent sum corresponding to all 16-dimensional mantissa products in TF32 / FP16 / BF16). Based on this solution, the existing general maximum value search module is reused, and a maximum value search module based on one-hot encoding is added. The two modules can work simultaneously or be used separately. It can not only be applied to maximum value searches in more data formats, but also reduce area overhead.

[0039] In order to verify the effectiveness of the scheme of the present invention, the following schemes are evaluated through DC sweep experiments: based on the existing direct comparison scheme, the FP8 maximum index search (baseline scheme), the FP8 maximum index search of the primary structure (primary encoding), the FP8 maximum index search of the 2-level structure (secondary encoding 5+1), the FP8 maximum index search of the 2-level structure (secondary encoding 4+2), and the FP8 maximum index search of the 2-level structure (secondary encoding 3+3) are implemented, and the following results are obtained: Figure 7 The experimental results are shown in Figure 2. In each solution, when the timing constraints are met, the area frequency curve is represented by a solid line, and when the timing violations occur, it is represented by a dotted line. Figure 7 It can be seen that the advantages and value of the solution of the present invention can be summarized as follows: the EMAX module using one-hot encoding for maximum exponent search has better timing performance at all frequencies than the baseline direct comparison method, and as the frequency increases, the one-hot encoding method is less sensitive to timing. Among all one-hot encoding implementations, splitting the maximum exponent module into two levels of 5+1-bit logic has the best relatively good area performance, and the timing is almost the same as the first-level leading zero calculation (onehot_emax_60). Compared with the optimal baseline solution in the same scenario, this innovative solution can reduce the area by about 20.9% at 1.5GHz and about 32.8% at 1.6GHz, and can meet the timing up to 1.9GHz before timing violations occur. The present invention can better adapt to the matrix calculation part in the Transformer model, can be flexibly promoted, adjust the search level according to timing and area requirements, and can be applied to other application scenarios with maximum / minimum comparison of multi-source operands, and optimize potential PPA problems (PPA: Power Performance Area, used to measure the comprehensive performance of the chip in three aspects: power consumption, performance, and area) to a certain extent.

[0040] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0041] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0042] Computer readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. Computer readable storage medium can include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0043] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A maximum value search device, which is used to search for the maximum value among multiple integer source operands, characterized in that: The device comprises a first maximum value search unit, wherein the first maximum value search unit comprises: A first one-hot encoder, configured to obtain a plurality of source operands and generate a one-hot encoding corresponding to each source operand; A first bitwise OR module, configured to perform a bitwise OR operation on the one-hot encodings of the plurality of source operands to obtain an intermediate result, wherein the bitwise OR module is consistent with an encoding bit width of the one-hot encoder; A first leading zero search module, configured to search the intermediate result to obtain a leading zero search result, wherein the leading zero search result is in a binary data format and has a data bit width consistent with a bit width corresponding to the plurality of source operands; The first bitwise inversion module is used to perform bitwise inversion on the leading zero search result to obtain the maximum value among the multiple source operands.

2. The device according to claim 1, characterized in that: The first one-hot encoder is 2 m bit-wide one-hot encoder; The first bitwise OR module consists of 2 m A bit-wide OR gate unit, wherein the input dimension of each bit of the OR gate unit is consistent with the dimension of the plurality of source operands; Wherein, m represents the exponent bit width of the plurality of floating-point data.

3. The device according to claim 1, characterized in that The device further comprises: a comparison module, a second maximum value search unit, and a splicing module, wherein the second maximum value search unit comprises a second one-hot encoder, a second bitwise OR module, a second leading zero search module, and a second bitwise inversion module; wherein, The first maximum value search unit is configured to obtain a high-order source operand and calculate a high-order maximum value based on the high-order source operand, wherein the high-order source operand is a preset high-order bit in the plurality of source operands; The comparison module is used to obtain a low-order mask based on the high-order maximum value, wherein the low-order mask includes preset low-order bits in the multiple source operands after comparison processing, wherein the comparison processing refers to setting the low-order bits of the source operands whose high-order bits in the multiple source operands are inconsistent with the high-order maximum value to zero; The second maximum value search unit is used to obtain preset low-order bits of the multiple source operands, perform an AND operation on the preset low-order bits and the low-order mask to obtain a low-order source operand, and calculate the low-order maximum value based on the low-order source operand by the second one-hot encoder, the second bitwise OR module, the second leading zero search module, and the second bitwise negation module; The splicing module is used to splice the high-order maximum value and the low-order maximum value to obtain the maximum value among the multiple source operations.

4. The device according to claim 3, characterized in that: The first one-hot encoder is a 2h bit width one-hot encoder; The first bitwise OR module comprises a 2h-bit wide OR gate unit; The second one-hot encoder is 2 l Bit-wide one-hot encoder; The second bitwise OR module consists of 2 l bit-wide OR gate unit; Wherein, h represents a preset high-order bit, l represents a preset low-order bit, and h+l is the exponent bit width of the plurality of floating-point data.

5. The device according to claim 1, characterized in that The device further comprises: a general maximum value search unit and a selection module; wherein, The first maximum value search unit is configured to calculate a first maximum value, the first maximum value being a maximum value of a portion of dimensions in the plurality of source operands; The universal maximum index unit is used to calculate a second maximum value by using a direct comparison method, where the second maximum value is a maximum value of remaining dimensions in the plurality of source operands; The selection module is used to select a final maximum value from the first maximum value and the second maximum value.

6. The device according to claim 5, characterized in that The first maximum value search unit is used to calculate the maximum value of the high 16-dimensional data in the plurality of source operands; The general maximum value search unit is used to calculate the maximum value of the lower 16-dimensional data in the multiple source operations.

7. A method for searching for the maximum exponent sum in a floating point dot product operation, characterized in that: The method comprises: S1. Obtain multiple exponent sums corresponding to the products of multiple mantissas of floating-point numbers; S2. Using the multiple exponential sums as source operands, searching for the maximum exponential sum among the multiple exponential sums using the apparatus described in any one of claims 1 to 6.

8. A dot product operation device for performing dot product operation of floating point row and column vectors, characterized in that: The device comprises: A multiplier array, used for performing multiplication operation on the floating-point row and column vectors to obtain a plurality of mantissa products corresponding to the row and column vectors and their corresponding exponential sums; A maximum exponent module is used to calculate the difference between the exponent sum corresponding to each mantissa product and the maximum exponent sum; wherein the maximum exponent module includes: Exponential sum logic, used to obtain the exponential sum corresponding to the multiple mantissa products; Maximum exponent logic, configured as an apparatus as claimed in any one of claims 1 to 7, for searching for a maximum exponent sum among sums of exponents corresponding to a plurality of mantissa products; Exponential difference logic, used to calculate the difference between the sum of exponents corresponding to each mantissa product and the maximum sum of exponents; An exponent alignment and shift module, configured to perform an alignment and shift operation on the sum of exponents corresponding to the products of the multiple mantissas based on the calculation result of the maximum exponent module; The multiplication accumulation module is used to accumulate the mantissa product and the corresponding exponent sum after the alignment shift operation by the exponent alignment shift module to obtain the dot product of the floating point number row and column vector.

9. A tensor operation system, characterized in that: include: Scheduling unit and tensor operation unit, where The scheduling unit is used to send a matrix multiplication and addition instruction to the tensor operation unit; The tensor operation unit is used to perform matrix multiplication and addition operations based on the multiplication and addition instructions; wherein the tensor operation unit includes: A control module, used for acquiring and caching data to be operated upon receiving a matrix multiplication and addition instruction; The calculation module comprises a dot product operation array composed of a plurality of devices as claimed in claim 8, and is used for performing dot product operation on the data to be operated cached by the control module after decoding and sparse matrix selection.

10. A computing device, characterized in that include: A host computer, and one or more tensor operation systems according to claim 9 connected to the host computer.

Images