A maximum value search device, a dot product operation device, and a tensor operation system
By employing one-hot encoding technology, the maximum exponent sum can be efficiently searched in the tensor operation unit, solving the problems of large area and timing requirements of the EMAX module. This reduces area overhead and improves timing performance, and is applicable to matrix operations in various floating-point formats.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- T-HEAD (SHANGHAI) SEMICON CO LTD
- Filing Date
- 2023-08-09
- Publication Date
- 2026-07-28
AI Technical Summary
The existing tensor operation unit, the Maximum Exponential Logic Module (EMAX), occupies a large area, and as the dimension of floating-point data increases, it cannot meet the timing and area requirements. There is an urgent need for a better EMAX solution.
One-hot encoding technology is used for maximum value search. Through one-hot encoder, bitwise OR module, leading zero search module and bitwise inversion module, the maximum exponential sum is searched efficiently, reducing area overhead and meeting timing requirements.
At the same frequency, one-hot encoding outperforms direct comparison in timing performance, reduces area by 20.9% to 32.8%, is suitable for matrix operations in various floating-point formats, and optimizes the chip's overall power consumption, performance, and area.
Smart Images
Figure CN119473214B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computers, specifically to the field of computer data processing, and more specifically to a maximum exponentiation device, a dot product operation device and chip, and a computing device. Background Technology
[0002] In the rapid development and application of artificial intelligence, the hardware devices used for Convolutional Neural Networks (CNNs) computations must be mentioned. As we all know, the data corresponding to CNNs is mainly floating-point matrices. During the training process of the neural network, floating-point matrix multiplication and addition operations are mainly involved. At this point, we must mention the tensor core, which is the unit used to perform floating-point matrix multiplication and addition operations.
[0003] In the tensor operation unit, when performing the dot product (DP) operation on row and column vectors in a floating-point matrix, before performing floating-point addition to obtain the final matrix element operation result, it needs to align the exponent sums corresponding to the mantissa products of the input row and column vectors to obtain a unified exponent sum. Before shifting and aligning the exponent sums corresponding to each mantissa product, the specific shift amount needs to be known, i.e., the difference between the exponent sum corresponding to that mantissa product and the maximum exponent sum. This means that before shifting, the maximum exponent in the mantissa products needs to be found. Furthermore, because the floating-point matrix of a convolutional neural network is quite large, the tensor unit cannot directly perform a complete calculation on the matrix each time it performs computation; it can only perform multiple batch calculations, each yielding a partial result. Therefore, when searching for the maximum exponent sum, each dot product unit not only needs to search for the maximum value among the exponent sums corresponding to the mantissa products of the currently being calculated row and column vectors, but also needs to compare these exponent sums with the exponent sums of intermediate results at the same position that have already been calculated in the floating-point matrix. Taking a typical tensor operation GPU as an example, each DP needs to find the maximum exponent of 8+1 FP16 / BF16 (data dimension of 8). The maximum exponent logic (EMAX module, Exponent Max) and the logic for finding the shift value usually need to be completed within one clock cycle. Therefore, the EMAX logic is the critical path in the timing of the entire tensor operation unit. The existing EMAX mainly adopts the general maximum exponent logic of direct comparison, which obtains the maximum exponent by comparing the exponents corresponding to all mantissa products. The higher the data dimension, the more complex the comparison logic becomes.
[0004] However, due to the large number of DP instantiations (each set of row and column vectors corresponds to a dot product unit DP) (a tensor operation unit has 32 DPs), EMAX incurs a significant area overhead, approximately 30% of the total DP area and 15% of the total Tensor Core area. Therefore, efficiently finding the maximum exponent using a smaller area becomes a key design challenge for tensor operation units. Existing EMAX, due to its comparison logic, becomes increasingly complex with the increase in floating-point data dimensions, failing to adequately meet timing and area requirements, necessitating a better EMAX solution. Summary of the Invention
[0005] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a new maximum value search device, dot product operation device and tensor operation system.
[0006] According to a first aspect of the present invention, a maximum value search apparatus is provided for searching for the maximum value among a plurality of integer source operands. The apparatus includes a first maximum value search unit, wherein the first maximum value search unit includes: a first one-hot encoder for acquiring the plurality of source operands and generating a one-hot code corresponding to each source operand; a first bitwise OR module for performing a bitwise OR operation on the one-hot codes of the plurality of source operands and obtaining an intermediate result, wherein the bitwise OR module has the same encoding bit width as the one-hot encoder; a first leading zero search module for searching the intermediate result to obtain a leading zero search result, wherein the format of the leading zero search result is binary data and the data bit width has the same as the bit width corresponding to the plurality of source operands; and a first bitwise NOT module for performing a bitwise NOT operation on the leading zero search result to obtain the maximum value among the plurality of source operands.
[0007] Preferably, the first independent thermal encoder is 2 m A one-hot encoder with a bit width; the first bitwise OR module includes 2 m A bit-width OR gate unit, wherein the input dimension of each bit of the OR gate unit is consistent with the dimension of the plurality of source operands; where m represents the exponential bit width of the plurality of floating-point data.
[0008] Preferably, the device further includes: a comparison module, a second maximum value search unit, and a splicing module. The second maximum value search unit includes a second one-hot encoder, a second bitwise OR module, a second leading zero search module, and a second bitwise inversion module. The first maximum value search unit is configured to acquire high-order source operands and calculate the high-order maximum value based on the high-order source operands, where the high-order source operands are preset high-order bits among the plurality of source operands. The comparison module is used to acquire a low-order mask based on the high-order maximum value, where the low-order mask includes preset low-order bits among the plurality of source operands after comparison processing. The comparison processing refers to setting the low-order bits of source operands whose high-order bits do not match the high-order maximum value to zero. The second maximum value search unit is used to acquire the preset low-order bits of the plurality of source operands, perform an AND operation with the low-order mask to obtain low-order source operands, and calculate the low-order maximum value based on the low-order source operands. The splicing module is used to splice the high-order maximum value and the low-order maximum value to obtain the maximum value among the plurality of source operands.
[0009] Preferably, the first uniquely thermal encoder is a 2h bit-width uniquely thermal encoder; the first bit-OR module includes a 2h bit-width OR gate unit; the second uniquely thermal encoder is a 2h bit-width uniquely thermal encoder. l Bit-width independent thermal encoder; the second bitwise OR module includes 2 l Bit width OR gate unit; where h represents the preset high-order bits, l represents the preset low-order bits, and h+l is the exponential bit width of the plurality of floating-point data.
[0010] Preferably, the apparatus further includes: a universal maximum search unit and a selection module; wherein the first maximum search unit is configured to calculate a first maximum value, the first maximum value being the maximum value of a subset of the multiple source operands; the universal maximum exponent unit is configured to calculate a second maximum value using a direct comparison method, the second maximum value being the maximum value of the remaining dimensions among the multiple source operands; and the selection module is configured to select the final maximum value from the first maximum value and the second maximum value.
[0011] Preferably, the first maximum value search unit is used to calculate the maximum value among the high 16-dimensional data of the plurality of source operands; the general maximum value search unit is used to calculate the maximum value among the low 16-dimensional data of the plurality of source operations.
[0012] According to a second aspect of the present invention, a method for searching for the maximum sum of exponents in floating-point dot product operations is provided, the method comprising: S1, obtaining multiple sums of exponents corresponding to multiple mantissa products of floating-point numbers; S2, using the multiple sums of exponents as source operands, searching for the maximum sum of exponents among the multiple sums of exponents using the apparatus described in the first aspect of the present invention.
[0013] According to a third aspect of the present invention, a dot product operation apparatus is provided for performing dot product operations on floating-point row and column vectors. The apparatus includes: a multiplier array for performing multiplication operations on the floating-point row and column vectors to obtain multiple mantissa products corresponding to the row and column vectors and their corresponding exponent sums; a maximum exponent module for calculating the difference between the exponent sum corresponding to each mantissa product and the maximum exponent sum, wherein the maximum exponent module includes exponent sum logic, maximum exponent logic, and exponent difference logic, the exponent sum logic is used to obtain the exponent sum corresponding to the multiple mantissa products, the maximum exponent logic is configured as described in the first aspect of the present invention, and is used to search for the maximum exponent sum among the exponent sums corresponding to the multiple mantissa products, the exponent difference logic is used to calculate the difference between the exponent sum corresponding to each mantissa product and the maximum exponent sum; an exponent alignment and shift module for performing an alignment and shift operation on the exponent sums corresponding to the multiple mantissa products based on the calculation result of the maximum exponent module; and a multiplication and accumulation module for accumulating the mantissa products after the alignment and shift operation performed by the exponent alignment and shift module and their corresponding exponent sums to obtain the dot product of the floating-point row and column vectors.
[0014] According to a fourth aspect of the present invention, a tensor operation system is provided, comprising: a scheduling unit and a tensor operation unit, wherein the scheduling unit is configured to send matrix multiplication and addition instructions to the tensor operation unit; the tensor operation unit is configured to perform matrix multiplication and addition operations based on the multiplication and addition instructions, wherein the tensor operation unit includes a control module and a calculation module, the control module is configured to acquire and cache data to be operated upon receiving the matrix multiplication and addition instructions, and the calculation module includes a dot product operation array composed of a plurality of devices described in the third aspect of the present invention, configured to perform dot product operations on the data to be operated cached by the control module after decoding and sparse matrix selection.
[0015] Compared with existing technologies, the advantages of this invention are as follows: This invention uses one-hot code encoding for maximum value searching, and its timing performance is superior to the baseline direct comparison method at all frequencies. As the frequency increases, the one-hot code encoding method becomes less sensitive to timing. Compared with the optimal baseline scheme in the same scenario, this invention can reduce the area by approximately 20.9% at 1.5GHz and approximately 32.8% at 1.6GHz, and can meet the timing requirement that timing violations only occur at 1.9GHz. In addition, this invention can better adapt to matrix operations of different floating-point formats, can be flexibly extended, and the search level can be adjusted according to timing and area requirements. It can also be applied to other application scenarios with multi-source operands and maximum / minimum value comparisons, optimizing potential PPA (Power Performance Area, used to measure the comprehensive performance of a chip in terms of power consumption, performance, and area) to a certain extent. Attached Figure Description
[0016] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:
[0017] Figure 1 This is a schematic diagram of the basic structure of a tensor operation unit according to an embodiment of the present invention;
[0018] Figure 2 A schematic diagram of the maximum value logic structure for implementing the FP8 floating-point dot product operation using the direct comparison method according to an embodiment of the present invention;
[0019] Figure 3 This is a schematic diagram of the maximum value search device according to an embodiment of the present invention;
[0020] Figure 4 This is a schematic diagram illustrating the working principle of the maximum value search device according to an embodiment of the present invention;
[0021] Figure 5 This is a schematic diagram illustrating the structure and working principle of a two-level maximum value search device according to an embodiment of the present invention;
[0022] Figure 6 This is a schematic diagram illustrating the structure and working principle of a maximum value search device for a reusable universal maximum value search module according to an embodiment of the present invention.
[0023] Figure 7 This is a schematic diagram of the experimental results according to an embodiment of the present invention. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention is further described in detail below through specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0025] As described in the background section of this invention, the maximum exponent logic using the direct comparison method in the prior art occupies a large area. In the face of constantly emerging new data formats, it is increasingly unable to meet the requirements of timing and area. Therefore, there is an urgent need for a new maximum value search scheme that can meet the timing requirements while reducing area overhead.
[0026] To better understand this invention, we will first introduce the basic structure of the tensor operation unit. In data center chips designed for artificial intelligence and high-performance computing, the tensor operation unit enables highly parallel processing of massive amounts of data, providing flexible mixed-precision computation while accelerating throughput in artificial intelligence network operations. For example... Figure 1As shown, the tensor operation unit includes a control section and a computation section. The computation section includes a dot product array composed of multiple dot product units. The process of performing operations on a floating-point matrix can be summarized as follows: the scheduling unit sends matrix multiplication and addition instructions to the tensor operation unit; the control section of the tensor operation unit sends a read request to the register and receives the returned data to be operated on into the internal data buffer; after decoding and sparse matrix selection, the buffered data is sent to the dot product operation unit; each dot product operation unit generates a partial result of the result matrix and continues to select other data in the buffer to calculate new partial results (each element in the final matrix operation result is actually the sum of multiple partial results corresponding to that element position. Each dot product unit accumulates the partial result obtained each time it calculates a partial result onto the partial results before the same element position until the result matrix required by the instruction is completely operated). In dot product operations, if a single dot product operation can process row / column vectors of dimension n, then the computational power of that dot product unit is called DP-n. Taking the typical tensor processing GPU chip mentioned in the background as an example, each dot product unit needs to process (1x8) row vectors multiplied by (8x1) column vectors to produce a (1x1) result. Therefore, the dot product computational power of this GPU is DP-8. In each clock cycle, the tensor processing unit of this GPU can consist of 32 dot product units (4 rows, 8 columns) to produce a 4x8 matrix multiplication and addition result. It should be noted that when performing floating-point dot product operations, before performing floating-point addition to obtain the final matrix element operation result, it is necessary to align the exponent sums corresponding to the mantissa products of the input row and column vectors to obtain a unified exponent sum. Before shifting the exponent sums corresponding to each mantissa product, the specific shift amount needs to be known, i.e., the difference between the exponent sum corresponding to each mantissa product and the maximum exponent sum. The structure of the dot product unit still refers to... Figure 1 As shown, each dot product unit includes a multiplier array, a maximum exponent module, an exponent alignment shift module, and a multiplicative accumulation module. The multiplier array performs multiplication on floating-point row and column vectors to obtain the mantissa product of the vectors. The maximum exponent module finds the maximum exponent sum among the exponent sums corresponding to the mantissa products and calculates the difference between the exponent sum of each mantissa product and the maximum exponent sum. The exponent alignment shift module performs a shift alignment operation on the exponent sum corresponding to each mantissa product based on the difference between the exponent sum of each mantissa product calculated by the maximum exponent module and the maximum exponent sum. The multiplicative accumulation module performs an accumulation operation on the mantissa products after the shift operation to obtain the dot product of the floating-point row and column vectors. Since the specific process of the floating-point row and column vector dot product operation and the basic principle of the dot product unit are known to those skilled in the art, they will not be elaborated upon here. The maximum exponent module will be the focus of the following description.
[0027] The maximum exponent module, as the most critical part of the dot product unit, involves exponent sum logic, maximum exponent logic, and exponent difference logic. Since the maximum exponent logic and exponent difference logic within the maximum exponent module typically need to be completed within one clock cycle, the maximum exponent logic is the timing critical path of the entire tensor operation unit, and its design complexity varies for different floating-point numbers. Taking the typical tensor operation GPU chip mentioned in the background technology as an example, each dot product unit needs to find 8+1 maximum exponent sums of FP16 / BF16. For other data formats, the corresponding data dimensions are different, and it may be necessary to find 16+1 maximum exponent sums, 32+1 maximum exponent sums, or even more. A typical example is the FP8 floating-point data format, which has been introduced into AI applications based on the Transformer model to increase the computational power of Attention matrix calculations. For floating-point matrices in FP8 data format, each dot product unit of the tensor arithmetic unit requires processing the inner product of 32-dimensional FP8 vectors. Furthermore, EMAX needs to select the maximum value from 33 exponents, further increasing its complexity. The current common solution is the direct comparison method, which involves grouping and comparing multiple times to obtain the maximum value. Taking a three-to-one selection logic as an example, EMAX (with inputs s0, s1, and s2, where the three-to-one logic requires comparing s0 vs s1, s0 vs s2, and s1 vs s2 respectively to obtain the maximum value among s0, s1, and s2) contains three 2-input comparators and one 3mux1 logic. Similarly, a four-to-one selection consists of six 2-input comparators and one 4mux1. Extending this to an n-to-one selection, a total of n(n-1) / 2 2-input comparators and one n-to-1 selector are needed.
[0028] Due to timing constraints, a maximum of two levels of comparisons can be performed. Assuming a scheme using multiple m-to-1 logic options is used to complete the maximum value comparison in FP8, such as... Figure 2 As shown, the first layer requires 10 three-choice options and 1 two-choice option, while the second layer requires 1 twelve-choice option. Since the principle of the direct comparison method is known to those skilled in the art, implementing the maximum value logic through multiple three-choice schemes is also easy to understand, so it will not be discussed here. Figure 2 To elaborate further.
[0029] Due to time constraints, under the general direct comparison method, other possible combinations are traversed. Area constraints are then considered within the time-constrained combinations. The optimal combinations among these possible combinations are shown in Table 1.
[0030] Table 1
[0031] 4 select 1 7 select 1 5 select 1 8 select 1 80 two select 1 comparators
[0032] As can be seen from Table 1, for the dot product unit of FP8 floating-point data format, the optimal solution among all maximum value logic schemes is composed of seven 4mux1s plus one 5mux1 in the first stage and one 8mux1 in the second stage. After synthesis at a frequency of 1.5 GHz, this scheme has a relatively high area overhead ratio (about 30%) in the tensor operation unit containing 32 dot product units corresponding to the FP8 floating-point data format. This is a very large area overhead for the dot product unit, so further optimization is needed.
[0033] In view of this, this invention proposes a maximum value search scheme based on one-hot encoding, which achieves more efficient finding of the maximum value when there are many comparison sources, and reduces timing and surface-based overhead. To better understand this invention, the concept of one-hot encoding is first introduced. One-hot encoding, also known as one-bit valid encoding, uses an N-bit state register to encode N states. Each state has its own independent register bit, and at any given time, only one bit is valid. For example, when performing one-hot encoding on a string of binary values with a width of 3 bits, the one-hot codes corresponding to 000, 001, 010, 011, 100, 101, 110, and 111 are: 00000001 (0 trailing zeros), 00000010 (1 trailing zero), 00000100 (2 trailing zeros), 00001000 (3 trailing zeros), 00010000 (4 trailing zeros), 00100000 (5 trailing zeros), 01000000 (6 trailing zeros), and 10000000 (7 trailing zeros). From the characteristics of one-hot encoding, it can be simply understood as using the number of trailing zeros to represent the value of the encoded value. The cost of one-hot encoding is determined by the maximum encoded binary value. For example, in the example above, the maximum encoded binary value is 111, and the cost of one-hot encoding is 8 bits, or 2^32 bits. 3 Bits. Since one-hot encoding is a technique known to those skilled in the art, it will not be elaborated upon here.
[0034] Based on the characteristics of one-hot encoding, according to one embodiment of the present invention, such as Figure 3As shown, a maximum value search device of the present invention includes: a one-hot encoder, a bitwise OR module, a leading zero search module, and a bitwise NOT module. The one-hot encoder is used to acquire multiple source operands and generate a one-hot code corresponding to each source operand. The bitwise OR module is used to perform a bitwise OR operation on the one-hot codes of the multiple source operands to obtain an intermediate result, wherein the bitwise OR module has the same encoding bit width as the one-hot encoder. The leading zero search module is used to search the intermediate result to obtain a leading zero search result, wherein the format of the leading zero search result is binary data and the data bit width is the same as the bit width corresponding to the multiple source operands. The bitwise NOT module is used to perform a bitwise NOT operation on the leading zero search result to obtain the maximum value among the multiple source operands. When applied to the dot product operation of floating-point row and column vectors, the maximum exponent sum can be obtained by using the maximum value search device of the present invention as the source operation, without the need for multi-level comparisons through multiple comparators, thus satisfying both timing and area constraints.
[0035] Furthermore, considering the reusability of the maximum value search device in floating-point dot product operations, enabling it to support calculations for multiple data types, according to one embodiment of the present invention, the maximum value search device is designed with the largest floating-point dimension. Specifically, the dimension of a single vector dot product is different for floating-point formats such as FP32 / TF32 / FP16 / BF16 / FP8. For the above data formats, the dimensions of each dot product (i.e., comparison dimensions) are 4 / 8 / 16 / 16 / 32 respectively. If a maximum value search device for the corresponding exponent sum of each data type is designed independently, it will generate a huge area overhead. If reusability is considered, it needs to be compatible with the largest comparison dimension (FP8) data. Therefore, in this embodiment of the present invention, the exponent sum corresponding to the mantissa product of floating-point numbers in FP8 format is mainly used as the object of explanation. Other data formats are similar and will not be described in detail.
[0036] Because FP8 has two types, E5M2 and E4M3, the exponent width of floating-point numbers in FP8 format is relatively small, only 4 bits or 5 bits. For the exponent sum corresponding to the product of the row and column vector mantissas of this type of data, the maximum value is only 6-bit (the sum of two 5-bit exponents). The cost of one-hot encoding is 64-bit (2^32 - 1 / 2) * 1 / 2 ... 6-b it). According to an embodiment of the present invention, when implementing maximum value search in FP8 format based on the maximum value search device of the present invention, the one-hot encoder is designed with a 64-bit width; the bitwise OR module is designed with a 64-bit width OR gate unit. Since the floating-point number in FP8 format has 32 dimensions, plus the previous intermediate results, the bitwise OR module needs to perform bitwise OR operation on the 33-dimensional 64-bit width data. Each bit implements the OR operation on the bits at the same position in the multi-dimensional data. Since the OR gate unit logic is a technology known to those skilled in the art, it will not be described in detail here. When the maximum value search device in this embodiment is used to perform maximum value search on the exponent sum corresponding to the mantissa product of the row and column vectors in FP8 format, such as Figure 4 As shown, the data processing procedure is as follows: The source operands are the sum of the exponents corresponding to the product of the 32 mantissas of the current row and column vectors, and the sum of the exponents of the partial results previously calculated for the dot product unit. These 33 source operands are encoded by a one-hot encoder. A bitwise OR operation is performed on the one-hot encoded source operands to obtain an intermediate result. Due to the characteristics of one-hot encoding, the intermediate result retains the exponent information of each source operand, and the larger exponent bits are closer to the highest bit of the intermediate result. A leading zero search is performed on the intermediate result, followed by a bitwise inversion, to obtain the maximum value among the source operands. According to an example of the present invention, reference is still made to... Figure 4 Taking a maximum exponent sum of 60 as an example, after one-hot encoding, we get a 64-bit binary value 00010000……0 (three leading zeros, one 1, and 60 trailing zeros). After performing a bitwise OR operation on all one-hot codes and searching for leading zeros, we get a leading zero search result of 3. The maximum bit width of the exponent sum in FP8 is represented as 000011. After inverting the leading zero search result bitwise, we get 111100, which corresponds to the maximum exponent sum of 60 in decimal. Then, we feed the maximum exponent sum into the exponent difference logic to obtain the difference between the exponent of each mantissa product and the maximum exponent, thus obtaining the displacement.
[0037] The above embodiments describe an implementation scheme for a single-hot comparison based on one-hot coding. The inventors, through research, discovered that further reducing area overhead by truncating the source operands into multiple levels according to timing constraints, such as truncating them into two, three, or even more levels, is possible. According to one embodiment of the present invention, as... Figure 5As shown, a maximum value search device of the present invention includes: two-layer maximum value search units (referred to as the first-layer maximum value search unit and the second-layer maximum value search unit for convenience), a comparison module, and a concatenation module. The first-layer maximum value search unit is configured to obtain high-order source operands and calculate the high-order maximum value based on the high-order source operands. The high-order source operands are preset high-order bits (denoted as k) among the plurality of source operands. In the FP8 format floating-point dot product operation, the source operands are the exponent sums corresponding to the FP8 mantissa products (referred to as the FP8 exponent sum), and the high-order source operands are the preset high-order bits in the exponent sums corresponding to the mantissa products. The comparison module is used to obtain the low-order mask (referred to as b) based on the high-order maximum value. The low-order mask includes a preset low-order bit (nk bit) in the source operands after comparison processing. The comparison processing involves setting the low-order bits of source operands whose high-order bits do not match the maximum high-order value to zero. Since the source operands to be compared are compared in stages, multiple identical high-order bits may appear when comparing them. Therefore, this invention, through comparison processing, can only include the low-order bits of source operands with identical high-order bits in the second layer for low-order maximum value search. Of course, if there is only one maximum value in the high-order bit search... Then, there is no need to enter the second layer for low-bit search; the source operand corresponding to the highest bit maximum value can be directly used as the maximum value. The second-layer maximum value search unit is used to obtain the preset low bit among the multiple source operands, perform an AND operation with the low bit mask to obtain the low-bit source operand, and calculate the low-bit maximum value based on the low-bit source operand. The concatenation module is used to concatenate the highest bit maximum value and the lowest bit maximum value to obtain the maximum value among the multiple source operands. In the floating-point dot product operation of FP8 format, the global maximum exponent sum of FP8 is obtained by concatenating the highest bit maximum value and the lowest bit maximum value. For the exponent sum corresponding to the product of the row and column vector mantissas of FP8 format floating-point data, the exponent sum can be divided into two levels of comparison: 5 bits + 1 bit, 1 bit + 5 bits, 4 bits + 2 bits, 2 bits + 4 bits, and 3 bits + 3 bits. The optimal combination in a specific scenario is optimally selected based on the timing constraints and area constraints of the specific scenario. In addition, there are other multi-level schemes, such as 1 bit + 2 bits + 3 bits, 2 bits + 2 bits + 2 bits, 1 bit + 1 bit + 4 bits, 2 bits + 1 bit + 3 bits, and even a four-level scheme. These multi-level schemes are similar in principle to the two-level schemes. They all obtain the global maximum value by comparing the maximum value of each level and then concatenating them. We will not go into further detail here.
[0038] The above embodiments describe a maximum value search scheme based solely on a one-hot encoder. The inventors have discovered that reusing existing general-purpose maximum exponent logic can further reduce area overhead and better support exponent and maximum value searches for different floating-point data formats such as FP32 / TF32 / FP16 / BF16 / FP8. According to one embodiment of the present invention, as... Figure 6 As shown, the maximum value search device that reuses the general maximum value search module includes a newly added FP8 maximum value search module (the FP8 maximum value search module here is designed using the same principle as the maximum value search module described in the previous embodiments of the present invention, except that the data dimension processed by the FP8 maximum value search module here is different. It can also be used for maximum value search in 6-bit wide data, whether it is the exponent sum corresponding to the mantissa product of floating-point numbers or integer data) and an existing general maximum value search module (an existing maximum value search module based on the direct comparison method. For example, in the prior art, the maximum exponent logic used by the dot product unit in the tensor operation unit is a maximum value search module based on the direct comparison method. Since the scheme of maximum value search based on the direct comparison method is a technology known to those skilled in the art, it will not be described in detail here, and will be referred to as the general maximum value search module). The FP8 maximum value search module can be used for the high 16-dimensional operands of FP8. (i.e., the sum of exponents corresponding to the mantissa products of the higher 16 dimensions of FP8) or the maximum value search of the higher 16-dimensional operands in DP32 (i.e., the sum of exponents corresponding to the mantissa products of the higher 16 dimensions of DP32). In this case, the data width processed by the FP8 maximum value search module is 6 bits, and the data dimension is 17 dimensions. It is only necessary to adjust the input dimension of each OR gate unit in the bitwise OR module to adapt to the processing of 17-dimensional data. The general maximum exponent module can be used to perform maximum value search on the lower 16-dimensional operands of FP8 (i.e., the sum of exponents corresponding to the mantissa products of the lower 16 dimensions of FP8), the lower 16-dimensional operands in DP32 (i.e., the sum of exponents corresponding to the mantissa products of the lower 16 dimensions of DP32), all 16-dimensional operands in DP16 (i.e., the sum of exponents corresponding to the mantissa products of all 16 dimensions of DP16), or all 16-dimensional operands in TF32 / FP16 / BF16 (i.e., the sum of exponents corresponding to the mantissa products of all 16 dimensions of TF32 / FP16 / BF16) using the direct comparison method. Based on this scheme, the existing general maximum value search module is reused, and a new maximum value search module based on one-hot encoding is added. The two modules can work simultaneously or be used separately. This not only makes it applicable to maximum value search for more data formats, but also reduces area overhead.
[0039] To verify the effectiveness of the present invention, the following schemes were evaluated through DC frequency sweep experiments: FP8 maximum exponent search (baseline scheme), FP8 maximum exponent search with a single-level structure (first-level coding), FP8 maximum exponent search with a two-level structure (second-level coding 5+1), FP8 maximum exponent search with a two-level structure (second-level coding 4+2), and FP8 maximum exponent search with a two-level structure (second-level coding 3+3), yielding the following results: Figure 7 The experimental results are shown. For each scheme, the area-frequency curve is represented by a solid line when the timing constraints are met, and by a dashed line when timing violations occur. From... Figure 7 The advantages and value of this invention can be summarized as follows: The EMAX module using one-hot code encoding for maximum exponent search outperforms the baseline direct comparison method in timing performance across all frequencies. As the frequency increases, the one-hot code encoding method becomes less sensitive to timing. Among all one-hot code encoding implementations, splitting the maximum exponent module into two levels of 5+1 bit logic yields the best area performance, while maintaining timing performance almost identical to that of a single-level leading zero calculation (onehot_emax_60). Compared to the optimal baseline scheme in the same scenario, this innovative scheme reduces the area by approximately 20.9% at 1.5GHz and by approximately 32.8% at 1.6GHz, while maintaining timing compliance until 1.9GHz where timing violations occur. This invention can be better adapted to the matrix calculation part in the Transformer model, can be flexibly extended, the search level can be adjusted according to timing and area requirements, and it can be applied to other application scenarios with multi-source operands and maximum / minimum value comparisons, and can optimize potential PPA problems (PPA: Power Performance Area, used to measure the comprehensive performance of a chip in three aspects: power consumption, performance, and area) to a certain extent.
[0040] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.
[0041] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.
[0042] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, including but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.
[0043] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.
Claims
1. A maximum value search device for searching for the maximum value among a plurality of integer source operands, characterized in that, The device includes a first maximum value search unit, wherein the first maximum value search unit includes: The first one-hot encoder is used to acquire multiple source operands and generate a one-hot code corresponding to each source operand. The first bitwise OR module is used to perform a bitwise OR operation on the one-hot encoding of the plurality of source operands to obtain an intermediate result, wherein the bit width of the first bitwise OR module is the same as that of the first one-hot encoder. A first leading zero search module is used to search the intermediate results to obtain a leading zero search result, wherein the format of the leading zero search result is binary data format and the data bit width is consistent with the bit width corresponding to the plurality of source operands; a first bit-inverting module is used to invert the leading zero search result bit-wise to obtain the maximum value among the plurality of source operands.
2. The apparatus according to claim 1, characterized in that: The first unique thermal encoder is Bit-width independent thermal encoder; The first bitwise OR module includes A bit-wide OR gate unit, wherein the input dimension of each bit of the OR gate unit is consistent with the dimension of the plurality of source operands; in, The exponent width of multiple floating-point data.
3. The apparatus according to claim 1, characterized in that, The device further includes: a comparison module, a second maximum value search unit, and a splicing module. The second maximum value search unit includes a second one-hot encoder, a second bitwise OR module, a second leading zero search module, and a second bitwise inversion module. The first maximum value search unit is configured to obtain high-order source operands and calculate the high-order maximum value based on the high-order source operands. The high-order source operands are preset high-order bits among the plurality of source operands. The comparison module is used to obtain the low-bit mask based on the maximum value of the high-bit. The low-bit mask includes a preset low-bit bit in the plurality of source operands after comparison processing. The comparison processing refers to setting the low-bit bit of the source operand whose high-bit bit does not match the maximum value of the high-bit to zero. The second maximum value search unit is used to obtain the preset low-order bits of the plurality of source operands, perform an AND operation with the low-order mask to obtain the low-order source operands, and the second one-hot encoder, the second bitwise OR module, the second leading zero search module, and the second bitwise inversion module calculate the low-order maximum value based on the low-order source operands. The splicing module is used to splice the high-bit maximum value and the low-bit maximum value to obtain the maximum value among the multiple source operations.
4. The apparatus according to claim 3, characterized in that: The first unique thermal encoder is Bit-width independent thermal encoder; The first bitwise OR module includes Bit-width OR gate unit; The second thermal encoder is Bit-width independent thermal encoder; The second bitwise OR module includes Bit-width OR gate unit; in, This indicates the preset high-order bits. This indicates the preset low-order bit. It is the exponential bit width of multiple floating-point data.
5. The apparatus according to claim 1, characterized in that, The device further includes: a universal maximum value search unit and a selection module; wherein... The first maximum value search unit is configured to calculate a first maximum value, which is the maximum value of a subset of the plurality of source operands; The general maximum value search unit is used to calculate the second maximum value using the direct comparison method. The second maximum value is the maximum value of the remaining dimensions among the plurality of source operands. The selection module is used to select the final maximum value from the first maximum value and the second maximum value.
6. The apparatus according to claim 5, characterized in that, The first maximum value search unit is used to calculate the maximum value among the high 16 dimensions of the plurality of source operands; the general maximum value search unit is used to calculate the maximum value among the low 16 dimensions of the plurality of source operations.
7. A method for searching the maximum sum of exponents in floating-point dot product operations, characterized in that, The method includes: S1. Obtain the sum of multiple exponents corresponding to the product of multiple mantissas of a floating-point number; S2. Using the sum of the plurality of exponents as the source operands, employ the method described in claim 1. The apparatus described in any of the 6 searches for the largest sum of exponents among the plurality of exponents.
8. A dot product operation device for performing floating-point row and column vector dot product operations, characterized in that, The device includes: A multiplier array is used to perform multiplication operations on the floating-point row and column vectors to obtain multiple mantissa products corresponding to the row and column vectors and their corresponding exponent sums; A maximum exponent module is used to calculate the difference between the sum of the exponents corresponding to each mantissa product and the maximum exponent sum; wherein, the maximum exponent module includes: Exponent and logic, used to obtain the exponent sum corresponding to the multiple mantissa products; Maximum exponential logic, which is configured as described in claim 1 6. Any of the aforementioned apparatuses is used to search for the largest sum of exponents among the sums of exponents corresponding to multiple mantissa products; Exponential difference logic is used to calculate the difference between the exponential sum of each mantissa product and the maximum exponential sum; The exponent alignment shift module is used to perform an alignment shift operation on the exponent sum corresponding to the product of the multiple mantissas based on the calculation result of the maximum exponent module. The multiply-accumulate module is used to accumulate the mantissa product after alignment and displacement operations by the exponent alignment and displacement module and its corresponding exponent sum to obtain the dot product of the floating-point row and column vectors.
9. A tensor operation system, characterized in that, include: Scheduling unit and tensor operation unit, wherein, The scheduling unit is used to send matrix multiplication and addition instructions to the tensor operation unit; The tensor operation unit is used to perform matrix multiplication and addition operations based on the multiplication and addition instructions; wherein, the tensor operation unit includes: The control module is used to obtain and cache the data to be operated on when a matrix multiplication and addition instruction is received; The computing module includes a dot product array composed of multiple devices as described in claim 8, used to perform dot product operations on the data to be computed cached by the control module after decoding and sparse matrix selection.
10. A computing device, characterized in that, include: A host computer, and one or more tensor computation systems as described in claim 9 connected to the host computer.