A low-power floating-point multiplication-accumulation method for neural network inference acceleration

Abstract

The present disclosure discloses a low-power floating-point multiplication and accumulation method for accelerating neural network reasoning. By preprocessing the input floating-point number, rounding the decimal part and adjusting its exponent bit before calculation, the data that may be rounded is pre-rounded and regularized, eliminating the calculation of unnecessary precision. At the same time, in order to enable the accumulation circuit to execute quickly, the decimal part of the exponent represented by the scientific and technological method abandons the original original code representation method and adopts the complement representation method instead. The present disclosure utilizes the characteristics of multiplication and accumulation operations and uses fixed-point operation resources that consume less resources to achieve accuracy close to that of floating-point operations, thereby solving the problem of balancing calculation accuracy and hardware implementation complexity in the neural network reasoning process.

Description

Technical Field

[0001] The present disclosure belongs to the field of processors and computing technologies, and in particular relates to a low-power floating-point multiplication-accumulation operation method for accelerating neural network inference. Background Art

[0002] With breakthroughs in artificial intelligence algorithms, inference algorithms, represented by deep convolutional neural networks, are widely used across various industries. However, deep convolutional neural networks require extremely high computing power, which traditional CPUs cannot provide. While GPUs can provide sufficient computing power, their performance-to-power ratio is too low, making them unsuitable for mobile devices.

[0003] Academia and industry have proposed a large number of specialized neural network accelerators (NPUs) to address these issues. Deep convolutional neural networks are insensitive to data precision. With the support of well-developed quantization algorithms, fixed-point arithmetic can achieve near-floating-point accuracy even with a limited dynamic range. NPU designs can be divided into two types based on the precision of the computational data: one type is a neural network accelerator that processes floating-point data, exemplified by NVIDIA's mobile GPUs. While these facilitate network deployment, their power consumption is suboptimal. The other type is a neural network accelerator that processes fixed-point data. By using fixed-point arithmetic instead of complex floating-point operations, their performance-to-power ratio is very high. However, network deployment requires a specialized quantization process, and their accuracy and dynamic range are limited.

[0004] Therefore, how to balance the precision advantage of floating-point operations in neural network accelerators with the energy advantage of fixed-point operations is a key issue that needs to be solved urgently. Summary of the Invention

[0005] In view of this, the present disclosure provides a low-power floating-point multiplication-accumulation method for accelerating neural network inference, which is characterized by:

[0006] By preprocessing the floating-point input, rounding the decimal fraction and adjusting the exponent before calculation, data that might otherwise be rounded is pre-rounded and normalized, eliminating unnecessary precision calculations. Furthermore, to ensure fast execution of the accumulation circuit, the decimal fraction of the exponent represented in scientific notation is replaced by its two's complement representation, rather than the original original code.

[0007] Preferably,

[0008] Scheme 1 of the method is specifically:

[0009] for Index bit sequence of group data , take the maximum value among them ,for The uniform exponent of the group data. Group data express, Each data in for:

[0010]

[0011] in Based on the uniform exponent ,Will Shift right. , the number of bits that need to be shifted right is ,and ;

[0012] In more extreme cases, when hour, It will become 0 due to the shift;

[0013] For each , both The data of the bit is directly discarded;

[0014] Similarly, for arrays Do the same process, and the approximate array is , Each data in for:

[0015]

[0016] in , is based on Will Move right The obtained bits are, ;

[0017] After the conversion, and All are 8-bit integer data and can be sent to the MAC array for calculation;

[0018] The result obtained by the set point multiplication part is :

[0019]

[0020] For the 8-term summation of i=0,1,2,…,7 in this formula, since it is fixed-point addition, it is obtained using a Wallace tree adder and a fixed-point adder;

[0021] Finally, combine the calculation of the decimal part with the operation of the exponential part:

[0022] Set the point multiplication output sequence as an array , then the array Each number in for:

[0023]

[0024] Since the resulting array The exponents of are unified, so the fixed-point adder can be used directly for addition. Let the result be , so we have:

[0025] ;

[0026] The method performs normalization operation only at the end of a group of multiplication and addition operations, which can save a lot of resources in both the multiplication part and the addition part;

[0027] The method pre-processes the data and determines the indexes of the respective groups before the two arrays to be multiplied enter the multiplication array.

[0028] Preferably,

[0029] If we wait until the two arrays enter the multiplication array, that is, after obtaining the exponent bit information of the two sets of data at the same time, and then jointly determine the exponent of the calculation result, we can improve the accuracy of the calculation result. Specifically:

[0030] Add the exponentials of the corresponding products of the two arrays first and find the maximum value:

[0031]

[0032] set up The maximum value in , Is the exponent of the final product result; when shifting according to the adjusted exponent, only Decimal number of groups To shift, The number of groups remains unchanged; The number of bits to be shifted left is ,but:

[0033]

[0034] After obtaining, the rest of the subsequent operations are the same as the above solution 1;

[0035] Compared with solution 1, the final approximate result is for:

[0036] .

[0037] As a result, the present disclosure implements a low-power floating-point multiplication-accumulation method for neural network inference acceleration, as well as a matrix multiplication-accumulation circuit for use in an end-to-end neural network inference accelerator. By leveraging the characteristics of multiplication-accumulation operations, the present disclosure uses relatively low-consumption fixed-point computing resources to achieve near-floating-point accuracy, thereby addressing the balance between computational accuracy and hardware implementation complexity during neural network inference. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the data structure of floating point numbers in the prior art;

[0039] Figure 2 This is a schematic diagram of a calculation using a standard floating-point multiplier in the prior art;

[0040] Figure 3 It is a schematic diagram of a standard floating-point adder in the prior art;

[0041] Figure 3A It is a schematic diagram of a circuit implemented according to a method in the prior art;

[0042] Figure 4 Schematic diagram of a Wallace number adder used in one embodiment of the present disclosure;

[0043] Figure 5 This is a schematic diagram of the calculation process of Solution 1 in one embodiment of the present disclosure;

[0044] Figure 6 This is a schematic diagram of the calculation process of Solution 2 in one embodiment of the present disclosure;

[0045] Figure 7 This is a schematic diagram of the application of Solution 1 in a fixed-point MAC array in one embodiment of the present disclosure;

[0046] Figure 8 This is a schematic diagram of the application of Solution 2 in a MAC array in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present disclosure.

[0048] Therefore, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the present disclosure as claimed, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present disclosure.

[0049] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0050] In the description of the present disclosure, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present disclosure and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present disclosure.

[0051] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. Throughout the present disclosure, "plurality" means two or more, unless otherwise specifically defined.

[0052] In this disclosure, unless otherwise expressly specified or limited, terms such as "mounted," "connected," "connect," and "fixed" should be understood broadly. For example, they may refer to fixed connections, detachable connections, or integration; they may refer to direct connections or indirect connections through an intermediate medium; they may refer to internal communication between two components or interaction between two components. Those skilled in the art will understand the specific meanings of these terms in this disclosure based on specific circumstances.

[0053] In the present disclosure, unless otherwise expressly specified or limited, a first feature being "above" or "below" a second feature may include the first and second features being in direct contact, or may include the first and second features being in contact not directly but through another feature between them. Moreover, a first feature being "above," "above," and "above" a second feature includes the first feature being directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below," "below," and "below" a second feature includes the first feature being directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0054] The present disclosure is described in detail below.

[0055] 1. The general concept of the present disclosure is:

[0056] The present disclosure discloses a matrix multiplication-accumulation operation circuit for an edge-side neural network inference accelerator. By utilizing the characteristics of multiplication-accumulation operations and using fixed-point operation resources that consume fewer resources, the circuit achieves accuracy close to that of floating-point operations, thereby solving the problem of balancing computational accuracy and hardware implementation complexity during the neural network inference process.

[0057] 2. Technical solution

[0058] 2.1 Origin

[0059] Now let’s take the most common multiplication and accumulation operation in neural networks as an example to illustrate. According to the traditional solution of the prior art, the two floating-point arrays used for multiplication and accumulation are and ,by Taking group data as an example, The group data contains a total of 8 numbers, and each data is a floating point number represented in BF16 format.

[0060] exist In the data set, The sign bit of the data is recorded as , occupies 1 bit; the exponent bit is recorded as , occupies 8 bits; decimal places are recorded as , occupies 7 bits. . No. The data is composed of Figure 1 As shown in the figure, when performing multiplication, the sign bit of the result can be directly obtained by performing an XOR operation on the sign bits of the two multiplicands. Therefore, we are more concerned with the operation rules of the exponent and decimal part. Remove the sign bit from the group data. The number is .

[0061] Floating-point numbers can be divided into regular numbers and irregular numbers according to their representation. They can both be represented using scientific notation. Regular numbers can be represented as follows:

[0062]

[0063] Non-infinite numbers in irregular numbers can be expressed in the following unified form:

[0064]

[0065] Numbers representing infinity require special notation.

[0066] against The unsigned portion of the group data will have:

[0067]

[0068] In binary, the base , that is, Move left Bit.

[0069] for The same applies to groups. The sign bit is recorded as , the exponent is recorded as , the decimal places are recorded as , remove the sign bit of the data and use express.

[0070]

[0071] Convolution or matrix multiplication are often used in various algorithms today. These complex operations can usually be broken down into a large number of various vector inner product operations. Group data and The group data are all considered as vectors. Now we need to calculate their inner product , that is, and Perform multiplication and accumulation operations on the data of the group. Calculation results The exact result should be:

[0072]

[0073] In formula (3) It is a symbol used in this article for the convenience of mathematical expression.

[0074]

[0075] remember Group and The first group of data The XOR result of the sign bit of the bit is , so there is . The symbol here represents the logical exclusive OR operation. When , the result is positive. ;on the contrary, When , the result is negative, .

[0076] 2.2 General calculation method

[0077] If formula (3) is implemented according to the traditional standard method, according to the general IEEE754 standard, 8 floating-point multipliers and 7 floating-point adders are required.

[0078] Each IEEE754 standard multiplier is as follows Figure 2 As shown in Figure 2. The signs of the two numbers are XORed, the exponents are added, and the decimals are multiplied.

[0079] Because multiplication operations may cause carries or shifts, the resulting number must be normalized to convert it to a standard floating-point number. However, the bit width of this standard floating-point number may not meet the output requirements, so its precision must be rounded. However, rounding operations may also cause carry shifts, so normalization is required after rounding.

[0080] Each normalization also includes three operations: "finding the most significant bit (counting leading zero)", "fractional shift" and "exponent bit adjustment shift".

[0081] Floating-point multiplication requires calling a fixed-point multiplication to implement the product of the decimal part, a low-bit adder to implement the product of the exponential part, and two sets of comparators and shifters to implement the shifting and rounding operations of the product result.

[0082] See also Figure 2 , which illustrates the calculation using a standard floating-point multiplier.

[0083] After using the standard multiplier to calculate, 8 floating point results are obtained, and these 8 numbers are used in an array Indicates that. Since the array There are 8 numbers in total, and 7 additions are required to add them all. The exponents of are not uniform, so a floating-point adder is required for addition.

[0084] The general process of each floating-point adder is as follows Figure 3As shown. The exponent bits of floating-point addition need to be compared first to find the larger value, and the larger exponent bit will be used as the predetermined exponent. The decimal bit corresponding to the smaller exponent bit will be shifted and adjusted to the same as the larger exponent bit. The number of shifted bits is the exponent difference. In this way, two floating-point numbers with the same exponent bits can be obtained. These two numbers are added with a sign. The subsequent operations are similar to floating-point multiplication, and normalization, rounding, and re-normalization operations need to be performed. The floating-point adder needs to call 1 comparator and 1 shifter to implement the sorting of input data, call 1 fixed-point adder to implement the sum operation of the decimal part, call 1 CLZ logic, 2 groups of comparators and 2 shifters to implement data normalization and rounding operations.

[0085] After seven accumulations, the final result can be obtained .

[0086] See also Figure 3 The standard floating-point adder is shown in the figure. The steps of the general calculation method are summarized as follows:

[0087] S100: From the group and group Extract a pair of floating point numbers and , use the floating-point multiplier to calculate and get the floating-point number .

[0088] S200: Repeat S100 until all the data in group A and group W are calculated and the array is obtained. In this example, there are 8 data in the group, which need to be repeated 8 times. The 8 operations can be performed in parallel, such as Figure 2 shown.

[0089] S300: Use floating point adder to add array Accumulate and get the calculation result as a floating point number In this case, the array The accumulation of 8 numbers requires 7 floating-point addition operations.

[0090] Therefore, an operation such as formula (3) requires the use of 8 floating-point multipliers and 7 floating-point adders. If the above steps are implemented using a circuit, the overall structural relationship is as follows Figure 3A shown.

[0091] Before introducing its specific embodiments, this disclosure briefly introduces the prior art, general computing methods, and their implementation. The specific circuit structure relationships of the general computing methods in the prior art are not described, and are provided for illustrative purposes only. The focus is on comparing the prior art with the solutions disclosed in this disclosure. The hardware resource consumption of the circuits implemented by the general computing methods in the prior art is summarized in Table 1:

[0092] Table 1

[0093] floating-point multiplier floating-point adder total CLZ Logic 0 7 7 shifter 16 21 37 Comparator 16 21 37 9-bit fixed-point multiplier 8 0 8 6-bit fixed-point adder 8 0 8 18-bit fixed-point adder 0 7 7

[0094] As can be seen, when using the general approach to perform calculations like those in formula (3), multiple rounding operations are performed on the data before and after the calculations. In fact, the general approach still performs standard high-precision calculations on data that may be rounded. Such unnecessary calculations waste a lot of hardware resources and take up a certain amount of runtime.

[0095] 2.3 Fast floating-point algorithm solution 1

[0096] Given that using general calculation methods consumes a large amount of hardware resources and slows down execution, this paper discloses a resource-saving multiplication-accumulation device. For calculations like those in Formula (3), a simple and fast method for approximating the inner product of floating-point vectors is designed. This method is based on a fixed-point MAC array (fixed-point multiplication-addition array) in hardware, making minor hardware modifications to the array compatible with floating-point operations.

[0097] The key idea behind the simplified algorithm is to preprocess the floating-point input and replace it with fixed-point multiplication-accumulation operations in two's complement format, thereby achieving higher-speed, lower-power approximate floating-point operations. The decimal fraction is rounded and the exponent is adjusted before calculation. This pre-rounding and normalization of potentially rounded data eliminates the need for unnecessary precision calculations. Furthermore, to ensure fast execution of the accumulation circuit, the decimal fraction of the exponent represented in scientific notation is replaced with two's complement representation, thereby supporting large-scale multiplication-accumulation operations.

[0098] It is worth noting that the fixed-point data used for multiply-accumulate operations uses two's complement format rather than the original format of floating-point data. This approach helps reduce the resource overhead of fixed-point operations and has a wider instruction fetch range.

[0099] The specific operations and methods are as follows:

[0100] for Index bit sequence of group data , take the maximum value among them ,for The uniform exponent of the group data. Group data express, Each data in for:

[0101]

[0102] in Based on the uniform exponent ,Will Shift right. , the number of bits that need to be shifted right is ,and In more extreme cases, when hour, It will become 0 due to the shift.

[0103] It can be seen that for each , both The bits of data are discarded. This may result in some loss of precision, but they will not be used in subsequent multiplications and sums, allowing us to omit subsequent rounding and normalization operations. The precision loss caused by this method will be discussed separately later.

[0104] Similarly, for arrays Do the same process, and the approximate array is , Each data in for:

[0105]

[0106] in , is based on Will Move right Among them, .

[0107] After the conversion, and All are 8-bit integer data and can be sent to the MAC array for calculation. The result of the set point multiplication part is :

[0108]

[0109] Then, for the formula (7) Since it is a fixed-point addition, it can be obtained by using a Wallace tree adder and a fixed-point adder. Figure 4 As shown in Figure 1, this is the Wallace number adder used in this example.

[0110] Finally, the calculation of the decimal part is combined with the operation processing of the exponential part.

[0111] Set the point multiplication output sequence as an array , then the array Each number in for:

[0112]

[0113] Since the resulting array The exponents of are unified, so the fixed-point adder can be used directly for addition. Let the result be , so we have:

[0114]

[0115] The flow chart for solution 1 is as follows: Figure 5 As shown:

[0116] If the two groups of floating-point numbers to be processed are and In summary, the steps of Solution 1 are summarized as follows:

[0117] S100: Take the exponent bits respectively and The maximum values ​​obtained are and .

[0118] S200: Decimal places and Shift them separately to get and For each , the number of bits that need to be shifted right is ,and Similarly, for each , the number of bits that need to be shifted right is ,and Combined with S100, the current results can be characterized by equations (5) and (6).

[0119] S300: Directly and Add together to get the calculation result The exponent bit.

[0120] S400: Each pair and Perform fixed-point multiplication operations on both, and get . Each pair of sign bits and Do XOR operation on them and get At this time, we get the result in formula (7): .

[0121] S500: Send it to the Wallace tree adder to get the calculation result Thus, combined with the exponent bit obtained in S300, the final result is a floating point number. , which is the result in formula (9).

[0122] S100 and S200 can be considered as pre-processing, which can be performed before the data is sent to the PE. The remaining steps must be executed in the PE.

[0123] S400 to S500 are fixed-point multiplication and addition steps, while S300 is a fixed-point multiplication and addition step that requires additional integration.

[0124] 2.4 Fast floating-point algorithm solution 2

[0125] Solution 1 preprocesses the data, determining the exponents for each group before the two arrays to be multiplied enter the multiplication array. If the exponents for the result are determined after both arrays enter the multiplication array—that is, after the exponents for both groups are simultaneously known—then the result's accuracy can be improved.

[0126] The specific steps are:

[0127] Add the exponents of the corresponding product numbers of the two arrays first and find the maximum value.

[0128]

[0129] set up The maximum value in , This is the exponent of the final product result. When shifting according to the adjusted exponent, only Decimal number of groups To shift, The number of groups remains unchanged. The number of bits to be shifted left is ,but:

[0130] (11)

[0131] The rest of the method is the same as that of solution 1, and the approximate results are for:

[0132] (12)

[0133] The calculation flow diagram of Scheme 2 is as follows: Figure 6 As shown:

[0134] If the two groups of floating-point numbers to be processed are and In summary, the steps of Solution 2 are summarized as follows:

[0135] S100: Set the exponents of the two sets of data and Add the corresponding , then ask The maximum value of . Directly as calculation result The exponent bit.

[0136] S200: Decimal places Shift and get For each , the number of bits that need to be shifted right is ,and . For decimal places No shifting is done.

[0137] S300: Each pair and Perform fixed-point multiplication operations on both, and get . Each pair of sign bits and Do XOR operation on them and get .

[0138] S400: Send it to the Wallace tree adder to get the calculation result Thus, combined with the exponent bit obtained in S300, the final result is a floating point number. , which is the result in formula (12).

[0139] In Solution 2, there's no preprocessing; all steps must be executed in the PE. S300 and S400 in this solution are similar to S400 through S500 in Solution 1 and can be considered fixed-point multiply-accumulate-add steps. If the fixed-point multiply-accumulate-add unit is used as an extension, S100 and S200 in Solution 2 must be integrated into the PE.

[0140] 3 Other embodiments

[0141] It should be noted that the design concept of this disclosure is to expand and modify the hardware of an existing fixed-point multiply-accumulate MAC array to enable floating-point operations. The floating-point arithmetic circuit obtained by this disclosure is highly reused with the fixed-point multiply-accumulate circuit, enabling efficient floating-point operations with minimal loss of precision.

[0142] 3.1 Application of Solution 1 in Fixed-Point MAC Array

[0143] The computing resources used in this method are highly consistent with the computing resources used in the fixed-point accumulator MAC array. Figure 7As shown, the green part is the computing resources used in the fixed-point MAC array. Figure 7 As can be seen in the figure, the additional arithmetic units for floating-point data processing are mostly implemented in the preprocessing phase, so no major modifications are required to the fixed-point MAC array. For each pair of arrays performing floating-point multiplication and accumulation, only the exponent summation unit needs to be added to the MAC array.

[0144] like Figure 7 As shown in FIG, a floating-point operation circuit is obtained by expanding the MAC array using the scheme.

[0145] Figure 7 The functions and connection methods of each module are introduced as follows:

[0146] Connection method:

[0147] Solid single arrow: indicates the transmission of multiple bits of data backward. Example:

[0148] Hollow thick arrow: indicates the transmission of a group of multi-bit data backward. Example:

[0149] In this example, a set of data has , expressed in the form of a superscript. Represents a group There are 8 data in a group.

[0150] Module Introduction: The green shaded area in the figure represents a PE compute unit in the existing fixed-point MAC array. It can perform multiplication and accumulation operations on eight fixed-point numbers, outputting a fixed-point number representing the multiplication and accumulation result. A PE compute unit contains eight fixed-point multipliers, capable of simultaneously performing multiplication operations on eight pairs of signed fixed-point numbers. It also includes a Wallace tree adder and a fixed-point adder for rapidly summing the eight products generated by the multipliers. The MAC array is composed of 64 such PE units, which are used to rapidly perform fixed-point matrix and vector multiplication operations in parallel.

[0151] MAX: Find the maximum value and output it from the passed data.

[0152] Sum: A fixed-point addition operation.

[0153] Shift: means shifting the data on the left side of the block diagram to the right, and outputting the shifted data on the right side. The number of shifted bits is input from the top of the block diagram.

[0154] Two sets of floating-point numbers: The input of the entire circuit, the input contains two sets of floating-point numbers and .

[0155] : The output of the entire circuit, expressed as a floating point number , which also consists of decimal, floating point and exponential parts, is an approximate solution to formula (3).

[0156] The remaining letters indicate intermediate variables in circuit calculations, see 2.3 above.

[0157] Preprocessing operations such as two-array shifts and exponent maximization can be performed before entering the MAC array. Compared to fixed-point multiply-accumulate operations, floating-point preprocessing can be divided into additional pipeline stages. This allows floating-point operations to achieve nearly the same hardware consumption, computational pipeline, and computation cycles as fixed-point operations.

[0158] In order to use such a fixed-point MAC array to perform floating-point operations, we need to make relevant modifications to each PE computing unit.

[0159] The plan is the same Figure 7 As shown, the circuitry to the left of the red dashed line can be considered data preprocessing circuitry, which can be distributed outside the MAC array. After preprocessing, it can be seen that compared to fixed-point arithmetic, only the addition of the exponent part is added. Therefore, each PE unit in the MAC array only needs to add a data port for the exponent bit and a fixed-point adder.

[0160] As shown in the figure, before the data enters the MAC array, the circuits that need to be added in the pre-processing part are:

[0161] Max Value Search: Two maximum value operations. In the figure, this operation for two sets of data is represented by the MAX box. It is used to find the maximum exponent within the set of data to be operated on, and this maximum value is used as the unified exponent of the set of data.

[0162] Determining the number of shifts: Two groups of 16 adders are used. In the figure, this operation on the two groups of data is represented by the μ_i and η_i boxes, respectively. This operation is used to shift the decimal places of the data by the group's unified exponent. The shift amount is the unified exponent minus the original exponent. Adders are used here to calculate the number of shifts.

[0163] Shift: Two groups of 16 shift operations, used to shift the decimal places of data according to the required shift number.

[0164] Because this solution only performs normalization at the end of a set of multiplication-add operations, both the multiplication and addition stages can save significant resources. Each multiplication uses two shifters and one fixed-point multiplier, and the multiplication stage has seven comparators to select the value with the largest exponent. The addition stage only performs normalization on the final result. The hardware resource consumption of this implementation is shown in the figure below:

[0165] Table 2: Hardware resource consumption of solution 1

[0166] Floating-point multiplication part Floating-point addition part total CLZ Logic 0 1 1 shifter 16 1 17 Comparator 7 1 8 9-bit fixed-point multiplier 8 0 8 6-bit fixed-point adder 0 0 0 18-bit fixed-point adder 0 7 * 7

[0167] * Since fixed-point number addition can be simplified using a Wallace tree adder, its hardware resources and timing are less than those of seven independent full adders.

[0168] 3.2 Application of Solution 2 in Fixed-Point MAC Array

[0169] The second solution is to perform quantization and exponential comparison operations after all data enters the MAC. This will give a higher precision result, but it will also require a greater modification to the fixed-point MAC array. Figure 8 As shown in the figure, the dark shaded area in the lower right corner is the fixed-point MAC array. If you want to use the second solution in the fixed-point MAC array, you must make significant changes to the MAC array. Figure 8 As shown, operations such as data decimal shifting, exponential comparison to find the maximum value, etc. need to be placed in the MAC array.

[0170] The circuit structure of Option 2 is as follows Figure 8 Most of the specific modules and connection methods have been introduced in the specific embodiment of solution one.

[0171] The additional modules introduced in Solution 2 are:

[0172]

[0173] Fixed-point addition operations, performing calculations .

[0174] This will have higher accuracy than solution 1, but may incur greater hardware consumption and latency.

[0175] As shown in the figure, the MAC array is expanded using the second scheme to obtain a floating-point operation circuit.

[0176] The difference between Solution 2 and Solution 1 is that the exponential processing in Solution 2 is all performed in the PE unit. Therefore, Solution 2 does not have a data preprocessing part. The PE unit designed according to Solution 2 can directly operate on floating-point numbers.

[0177] As shown in the figure, the green part is the same as in Solution 1, which is the PE unit in the fixed-point MAC array. Since there is no data preprocessing, the additional floating-point processing needs to be integrated into the PE unit.

[0178] The circuits that need to be additionally integrated in the PE unit are:

[0179] Exponential addition: 8 fixed-point adders. Use boxes in the figure Calculate the sum of the exponentials of the corresponding data in the two sets of data.

[0180] Maximum value search: an operation to find the maximum value. In the figure, this operation is used The box represents the maximum exponential value among the sum of the exponentials calculated in the previous step, and takes this maximum value as the exponent of the calculation result.

[0181] Determine the number of shifts: a group of 8 adders. In the figure, the operation of the two sets of data is performed using Grid representation. Used to shift the decimal places of a set of data according to the determined exponent. The shift amount is the exponent of the result minus the original exponent of the data. Here, an adder is used to implement the shift calculation.

[0182] Shift: A set of 8 shift operations. Used to shift the decimal places of data according to the required shift number.

[0183] It can be seen that when using the second design method, there is no preprocessing circuit, but the changes to each PE unit are relatively large.

[0184] Compared with Solution 1, this solution adds exponential operations to the multiplier to improve calculation accuracy. In addition, since only one multiplier is shifted, the shifter on another path is saved. The hardware resource consumption of this implementation is shown in the figure below:

[0185] Table 3: Hardware resource consumption of solution 2

[0186] Floating-point multiplication part Floating-point addition part total CLZ Logic 0 1 1 shifter 8 1 9 Comparator 8 1 9 9-bit fixed-point multiplier 8 0 8 6-bit fixed-point adder 8 0 8 18-bit fixed-point adder 0 7 * 7

[0187] * Since fixed-point number addition can be simplified using a Wallace tree adder, its hardware resources and timing are less than those of seven independent full adders.

[0188] Compared to Option 1, Option 2 requires a larger area per PE unit and consumes more power, but also offers higher computational accuracy without floating-point preprocessing circuitry. While Option 2 offers nearly identical accuracy to standard floating-point calculation circuits, it omits a significant number of floating-point adders and floating-point rounding circuitry, resulting in faster operation and lower power consumption. Furthermore, it offers a high degree of reuse with fixed-point calculation circuitry, resulting in greater scalability.

[0189] 4. Scalability and Application Scope of the Present Disclosure

[0190] In addition to being applicable to neural network convolution operation circuits, the approximate calculation method disclosed in this disclosure can also be applied to other high-performance, low-power circuit designs that require multiple multiplication and accumulation. Taking large-scale non-sparse matrix multiplication as an example, a large-scale matrix can divide its elements into multiple 8x8 sub-matrices. Large-scale matrix multiplication can use sub-matrices as elements to perform multiplication and accumulation operations, thereby improving computing power.

[0191] The approximate calculation method disclosed in this disclosure is mainly for the floating-point format of BF16. In addition, it can also be used for floating-point formats such as FP16, FP32, and FP64. FP16 is used as an example for analysis. Compared with BF16, the main difference of FP16 is that its decimal places total 11 bits and the exponent places total 5 bits. The rest of the characteristics are the same. Therefore, the fixed-point multiplication and accumulation unit is expanded to a 12-bit signed fixed-point multiplication and accumulation array, and the exponent-related calculations are reduced to 5-bit operations, so that the multiplication and accumulation method of the FP16 format can be processed.

[0192] The approximate calculation method disclosed in this disclosure is mainly used for multiplication and accumulation circuits of two groups of 8 elements. In addition, it can also be optimized for two groups of any number of elements. More common array sizes include 12x12, 16x16, etc.

[0193] The approximate computation method disclosed in this disclosure primarily optimizes multiply-accumulate operations. In addition, it can also optimize Map-reduce computations such as multiply-accumulate-subtract, multiply-Max, and multiply-Min. Taking Max pooling as an example, this operation does not sum data but instead calculates the maximum value for each set of data. Therefore, this can be achieved by simply replacing the fixed-point multiply-accumulate unit with comparison and selection operations.

[0194] In addition, the present disclosure also has the following outstanding features:

[0195] (1) A matrix multiplication and accumulation circuit for edge-side neural network inference accelerators is disclosed. By utilizing the characteristics of multiplication and accumulation operations, the circuit uses fixed-point computing resources with relatively low resource consumption to achieve accuracy close to that of floating-point operations, thereby solving the problem of balancing computational accuracy and hardware implementation complexity during neural network inference.

[0196] (2) In view of the characteristics of neural network data, the two’s complement + exponential form is used to represent floating-point numbers, rather than the traditional original code + exponential form, to provide support for large-scale accumulation operations;

[0197] (3) By utilizing the fixed pattern of the multiplication-accumulation algorithm, normalization is not performed directly after the multiplication is completed. Instead, normalization is performed uniformly after the final summation, thereby greatly simplifying the resource consumption of the multiplication-accumulation circuit.

[0198] Although the embodiments of the present disclosure have been described above with reference to the accompanying drawings, the present disclosure is not limited to the specific embodiments and application areas described above. The specific embodiments described above are merely illustrative and instructive, and not restrictive. A person of ordinary skill in the art, guided by this specification and without departing from the scope of protection of the claims of the present disclosure, may devise various other forms, all of which fall within the scope of protection of the present disclosure.

Claims

1. A low-power floating-point multiplication-accumulation method for accelerating neural network inference, characterized by: By preprocessing the input floating-point number, the decimal part is rounded and the exponent is adjusted before calculation, which can pre-round and normalize the data that may be rounded, eliminating the calculation of unnecessary precision. At the same time, in order to ensure the fast execution of the accumulation circuit, the decimal part of the exponent expressed in scientific notation abandons the original code representation method and adopts the two's complement representation method instead. Wherein, scheme 1 of the method is specifically as follows: for Index bit sequence of group data , take the maximum value among them ,for The uniform exponent of the group data is approximated Group data express, Each data in for: ， in Based on the uniform exponent ,Will Shift right to get, for each , the number of bits that need to be shifted right is ,and ; In more extreme cases, when hour, It will become 0 due to the shift; For each , both The data of the bit is directly discarded; Similarly, for arrays Do the same process, and the approximate array is , Each data in for: ， in , is based on Will Move right The obtained bits are, ; After the conversion, and All are 8-bit integer data and can be sent to the MAC array for calculation; The result obtained by the set point multiplication part is : ,For the 8-term summation of i=0,1,2,…,7 in the formula, since it is a fixed-point addition, it is obtained using a Wallace tree adder and a fixed-point adder; Finally, combine the calculation of the decimal part with the operation of the exponential part: Set point multiplication output sequence as an array , then the array Each number in for: ， Since the resulting array The exponents of are unified, so we can directly use the fixed-point adder to add, and let the result be , so we have: ； The method performs normalization operation only at the end of a group of multiplication and addition operations, which can save a lot of resources in both the multiplication part and the addition part; The method pre-processes the data and determines the indexes of the respective groups before the two arrays to be multiplied enter the multiplication array.

2. The method according to claim 1, wherein If we wait until the two arrays enter the multiplication array, that is, after obtaining the exponent bit information of the two sets of data at the same time, and then jointly determine the exponent of the calculation result, we can improve the accuracy of the calculation result. Specifically: Add the exponentials of the corresponding products of the two arrays first and find the maximum value: ， set up The maximum value in , Is the exponent of the final product result; when shifting according to the adjusted exponent, only Decimal number of groups To shift, The number of groups remains unchanged; The number of bits to be shifted left is ,but: ， After obtaining, the rest of the subsequent operations are the same as the above solution 1; Compared with solution 1, the final approximate result is for: 。

Images