Index operation implementation method and device of activation function and electronic equipment

Abstract

The invention provides an index operation implementation method and device of an activation function, electronic equipment and a non-instantaneous computer readable storage medium, and the method comprises the steps: (1) calculating a 16-bit floating point value corresponding to the index operation of the activation function, the 16-bit floating point value is an index operation hardware storage value of the BF16 format of the activation function obtained according to a table look-up method; (2) dividing the 16-bit floating point numerical value into a plurality of segments according to the numerical value distribution of the 16-bit floating point numerical value; (3) performing curve fitting on each segment to obtain a corresponding fitting function; (4) calculating an error between the fitting function and a corresponding 16-bit floating point value; and (5) under the condition that the error is greater than a preset error threshold value, dividing the segment corresponding to the fitting function into a plurality of sub-segments, and repeatedly executing the steps (3)-(4) for each sub-segment until the calculated error of each sub-segment is smaller than the error threshold value. According to the embodiment, the operation precision is guaranteed, and waste of hardware resources is avoided.

Description

Technical Field

[0001] This invention relates to the field of deep learning model execution, and more specifically, to a method and apparatus for implementing exponential operations of activation functions, an electronic device, and a non-transitory computer-readable storage medium. Background Technology

[0002] In the attention mechanism of large-scale Transformer models, the Softmax activation function is responsible for converting the similarity scores between the query (Q) and the key (K) into a probability distribution. Its computational accuracy directly affects the model's ability to model long-range dependencies. However, traditional software implementations rely on 32-bit floating-point operations (FP32), which, while ensuring numerical stability (e.g., avoiding exponential overflow / underflow), become bottlenecks at the scale of hundreds of billions of parameters due to FP32's high memory consumption (4 bytes per parameter) and computational latency (pipeline blockage after matrix multiplication followed by Softmax). Some edge scenarios (such as mobile deployments) and low-power requirements (such as edge AI chips) necessitate further compression of computational accuracy. Directly downgrading to 16-bit floating-point operations (FP16) or 8-bit integer operations (INT8) leads to insufficient dynamic range for exponential operations, easily causing numerical overflow, ultimately disrupting the normalization properties of the probability distribution, resulting in distorted attention weights and model output illusions or logical errors.

[0003] To address the aforementioned issues, a hardware solution for lookup tables (LUTs) based on floating-point 16-bit operations (FP16) balances accuracy and efficiency through a combination of pre-computation and hardware acceleration. The core idea of ​​this solution is to pre-build a high-precision reference table in hardware storage units (such as SRAM or on-chip cache) for the most critical exponentiation operation (exp(x)) and normalization step (1 / sum(exp(x))) in the Softmax activation function. During actual inference, approximate values ​​are quickly obtained by looking up the table, rather than through real-time computation. Compared to pure FP16 real-time computation (which has a limited dynamic range and is prone to overflow), the lookup table hardware utilizes FP16 as the input / output transmission format (saving memory bandwidth). Simultaneously, it leverages the FP32-level precision (or higher precision reserved bits) of the reference table to compensate for the inherent dynamic range limitations of FP16. This ensures computational speed (lookup table access latency is only in the nanosecond range) while effectively mitigating the numerical risks of exponentiation operations through "precise mapping within a limited range," providing a compromise of "low storage overhead + high precision guarantee" for large models.

[0004] The hardware implementation based on FP16 lookup tables essentially solves the core contradiction of the Softmax activation function in large models through "precision compensation design." This means satisfying the energy efficiency and storage compression requirements of low-precision computation (FP16) while avoiding numerical distortion due to insufficient dynamic range. Its value lies not only in ensuring the precision of activation function operations but also in driving an efficiency upgrade of the entire attention mechanism through hardware-level acceleration.

[0005] The essence of the FP16-based lookup table method is to approximate a continuous function with finite discrete points. The small mantissa of the floating-point format makes the "nearest neighbor interpolation / rounding error" more easily amplified, especially in nonlinear regions such as exp(), 1 / x, and log. The error exhibits systematic rather than random behavior, affecting the stability and interpretability of the attention weights. Taking the BF16 format as an example, with 1 sign bit, 8 exponent bits, and 7 mantissa bits, the input dynamic range is relatively wide (8-bit exponent). To obtain acceptable accuracy across the entire range, LUT entries need to grow exponentially, which is unacceptable in terms of on-chip storage and power consumption. Therefore, in engineering, it can only cover the "high-frequency range," and scaling or backtracking paths are used for inputs outside the range. This brings three costs: First, switching across ranges introduces additional control and branching overhead; second, the O(N²) memory access mode of the attention matrix overlaps with LUT access, which can easily form bandwidth bottlenecks and cache pollution; third, the long tail of the distribution or dynamic drift during training will reduce the hit rate, forcing more backtracking to high-precision calculations or generating additional latency. For inference / training of large-scale models (such as tens to hundreds of billions of parameters), these structural bottlenecks are difficult to solve by simply increasing the size of the LUT. Summary of the Invention

[0006] The present invention aims to provide a method and apparatus for implementing exponential operations of activation functions, an electronic device, and a non-transient computer-readable storage medium, so as to solve the problem of high storage and power consumption requirements in the exponential operation of activation functions.

[0007] According to one aspect of the present invention, a hardware implementation method for exponential operation in an activation function is proposed, comprising: (1) Calculate the 16-bit floating-point value corresponding to the exponentiation operation of the activation function, wherein the 16-bit floating-point value is the hardware storage value of the exponentiation operation of the activation function in BF16 format obtained by the lookup table method. (2) Divide the 16-bit floating-point value into multiple segments according to the numerical distribution of the 16-bit floating-point value; (3) Perform curve fitting on each segment to obtain the corresponding fitting function; (4) Calculate the error between the fitting function and the corresponding 16-bit floating-point value; (5) If the error is greater than the preset error threshold, the segment corresponding to the fitting function is divided into multiple sub-segments, and steps (3)-(4) are repeated for each sub-segment until the error of each sub-segment is less than the preset error threshold.

[0008] According to some embodiments, after step (3) and before step (4), the method further includes: Check whether the coefficients of the fitted function are within the range of values ​​in 16-bit floating-point format; If the coefficients of the fitted function are not within the range of 16-bit floating-point format, the coefficients of the fitted function are quantized to the range of 16-bit floating-point format.

[0009] According to some embodiments, when the coefficients of the fitting function are not within the numerical range of 16-bit floating-point format, quantizing the coefficients of the fitting function to the numerical range of 16-bit floating-point format includes: If the coefficients of the fitted function are not within the range of 16-bit floating-point format, the coefficients of the fitted function are quantized to the range of 16-bit floating-point format by truncation or rounding.

[0010] According to some embodiments, the hardware implementation method further includes: Evaluate the hardware implementation cost corresponding to the fitted function; If the hardware implementation cost is greater than a preset hardware cost threshold, repeat steps (3)-(5) until the calculated hardware implementation cost of each segment is less than the preset hardware cost threshold.

[0011] According to some embodiments, the hardware implementation method further includes: Evaluate the hardware implementation cost corresponding to the fitted function; If the hardware implementation cost exceeds a preset hardware cost threshold, a lookup table is used instead of the fitting function.

[0012] According to some embodiments, the hardware implementation cost corresponding to the fitting function is evaluated, including: The fitting function is decomposed to obtain the hardware-executable basic computing unit corresponding to the fitting function; If the basic operation unit includes division, recursive operation and / or higher-order polynomials, then the hardware implementation cost of the fitting function is considered to be greater than a preset hardware cost threshold.

[0013] According to some embodiments, the hardware implementation method further includes: Normalization is performed on the coefficients of the fitted function, and the coefficients of the fitted function are stored in the lookup table to simplify multiple fitted functions into one function; The number of basic computational units after the fitting function is decomposed is counted to provide a quantitative basis for the hardware implementation scheme of the exponential operation.

[0014] According to one aspect of the present invention, a hardware implementation apparatus for an activation function is provided, comprising: A 16-bit floating-point numerical calculation unit is used to calculate the 16-bit floating-point value corresponding to the exponential operation of the activation function; The segmentation unit is used to divide the 16-bit floating-point value into multiple segments according to the numerical distribution of the 16-bit floating-point value. The fitting unit is used to perform curve fitting on each segment to obtain the corresponding fitting function; An error calculation unit is used to calculate the error between the fitting function and the corresponding 16-bit floating-point value; The repeated calculation unit is used to divide the segment corresponding to the fitting function into multiple segments when the error is greater than a preset error threshold, and to use the fitting unit and the error calculation unit multiple times until the error of each segment is less than the preset error threshold.

[0015] According to one aspect of the present invention, an electronic device is provided, comprising: a processor; and a memory storing a computer program that, when executed by the processor, causes the processor to perform a hardware implementation method as described in any of the preceding embodiments.

[0016] According to one aspect of the present invention, a non-transitory computer-readable storage medium is provided, on which computer-readable instructions are stored, which, when executed by a processor, cause the processor to perform the method as described in any of the preceding embodiments.

[0017] According to an embodiment of the present invention, by performing multiple segmented function fitting on the 16-bit floating-point hardware storage value of the exponential operation in the activation function, the waste of hardware resources is avoided and the implementation cost of the product is reduced while ensuring the accuracy of the exponential operation in the activation function.

[0018] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit the invention. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. The above and other objectives, features, and advantages of the present invention will become more apparent from the detailed description of exemplary embodiments with reference to the accompanying drawings.

[0020] Figure 1 A flowchart illustrating a hardware implementation method for exponential operation in an activation function according to an exemplary embodiment of the present invention is shown.

[0021] Figure 2 A block diagram of a hardware implementation apparatus for an activation function according to an exemplary embodiment of the present invention is shown.

[0022] Figure 3 A block diagram of an electronic device according to an exemplary embodiment of the present invention is shown. Detailed Implementation

[0023] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that the invention will be thorough and complete, and the concept of the exemplary embodiments will be fully conveyed to those skilled in the art. The same drawings in the figures show the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0024] The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of these specific details, or other methods, components, materials, apparatus, or operations may be employed. In these cases, well-known structures, methods, apparatuses, implementations, materials, or operations will not be shown or described in detail.

[0025] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0026] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0027] Bfloat16, or BF16 for short, is a 16-bit floating-point format designed for artificial intelligence applications such as deep learning. It aims to balance numerical range, precision, storage efficiency, and computation speed. The BF16 structure includes one sign bit, eight exponent bits, and seven mantissa bits. Compared to the 32-bit IEEE 754 single-precision floating-point number (float32), BF16 retains the same 8-bit exponent, thus maintaining the same numerical range as float32. However, the mantissa bits are reduced from 23 bits to 7 bits, resulting in lower precision. This design allows BF16 to be directly converted from float32 by truncating the mantissa portion, simplifying the process and eliminating the need for complex scaling or quantization.

[0028] As mentioned earlier, taking the BF16 format as an example, it has 1 sign bit, 8 exponent bits, and 7 mantissa bits, resulting in a wide input dynamic range (8-bit exponent). To obtain acceptable accuracy across the entire range, a hardware storage scheme based on lookup tables would require exponential growth of LUT entries, which would be unacceptable in terms of on-chip storage and power consumption. Therefore, in engineering practice, it can only cover the "high-frequency range," and scaling or rollback paths are used for inputs outside the range.

[0029] Taking the activation function Softmax as an example, the most complex operation in Softmax is the floating-point BF16 exponentiation. BF16 is 16 bits long. By directly looking up the table, the exponentiation operation of the Softmax function requires storing 2 to the power of 16, a total of 65,536 operation results. Each result occupies 2 bytes, and the storage space is 128KB. Occupying 128KB of storage space for just one function operation is a significant cost for FPGA or ASIC implementation. Although storing the LUT table in off-chip DDR reduces the internal storage space of the chip, it will bring problems such as bandwidth bottlenecks, low computational efficiency, and high power consumption.

[0030] The inventors of this invention discovered that LUT hardware for BF16 requires a delicate balance between "range partitioning, segmented interpolation, and fallback mechanisms (such as out-of-range scaling / rollback)." Different models / data distributions have different preferences for "hotspot regions," making fixed LUTs unreliable and often requiring online calibration, dynamic reconstruction, or mixed-precision strategies, significantly increasing architectural and microarchitectural complexity. Furthermore, the coupling of the LUT with the main computational pipeline introduces additional timing convergence pressure and area overhead, necessitating careful cost-effectiveness evaluation at advanced process nodes.

[0031] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0032] Figure 1 A flowchart illustrating a hardware implementation method for exponential operation in an activation function according to an exemplary embodiment of the present invention is shown, as follows: Figure 1The method shown includes steps S101, S103, S105, S107, and S109. The following will use... Figure 1 Taking an example, a hardware implementation method for exponential operation in an activation function according to an exemplary embodiment of the present invention will be described in detail.

[0033] like Figure 1 As shown, in step S101, the 16-bit floating-point value corresponding to the exponential operation of the activation function is calculated.

[0034] According to an embodiment of the present invention, the activation function includes the Softmax function.

[0035] In a specific embodiment, the 16-bit floating-point value is a hardware storage scheme for the exponential operation of the Softmax function based on the lookup table method. That is, the 16-bit floating-point value is the hardware storage value of 65536 exponential operations of the activation function in BF16 format obtained by the lookup table method.

[0036] In step S103, the 16-bit floating-point value is divided into multiple segments according to the numerical distribution of the 16-bit floating-point value.

[0037] According to an embodiment of the present invention, in order to facilitate the analysis of the numerical distribution of the 16-bit floating-point value, before step S103, it is also necessary to draw the 16-bit floating-point value into a scatter plot, so as to divide the 16-bit floating-point value into multiple segments according to the size of the 16-bit floating-point value and the corresponding variable value range.

[0038] In step S105, curve fitting is performed on each segment to obtain the corresponding fitting function.

[0039] The image of a 16-bit floating-point value in BF16 format of the Softmax function is similar in shape to FP32 and the mathematically ideal exponential function, both approaching 0 when x is negative infinity, equaling 1 when x=0, and then rising rapidly as x increases. According to an embodiment of the present invention, in step S105, a fitting is performed based on the image shape of the 16-bit floating-point value scatter plot. The fitting function includes, but is not limited to, polynomial functions, exponential functions, and / or logarithmic functions.

[0040] In step S107, the error between the fitting function and the corresponding 16-bit floating-point value is calculated.

[0041] According to an embodiment of the present invention, after step S105 and before step S107, Figure 1 The hardware implementation methods shown also include: Check whether the coefficients of the fitted function are within the range of values ​​in 16-bit floating-point format; If the coefficients of the fitted function are not within the range of 16-bit floating-point format, quantize the coefficients of the fitted function to the range of 16-bit floating-point format; or If the coefficients of the fitted function are not within the range of 16-bit floating-point format, the coefficients of the fitted function are quantized to the range of 16-bit floating-point format by truncation or rounding.

[0042] According to an embodiment of the present invention, by calculating the error between the fitting function and the corresponding 16-bit floating-point value, it is determined whether to segment the current 16-bit floating-point value range.

[0043] In step S109, if the error is greater than a preset error threshold, the segment corresponding to the fitting function is divided into multiple sub-segments, and steps S105-S107 are repeated for each sub-segment until the error of each sub-segment is less than the preset error threshold.

[0044] According to an embodiment of the present invention, in step S109, based on the error calculated in step S107, it is determined whether to divide the segment corresponding to the fitting function into multiple sub-segments. If the segment corresponding to the fitting function is divided into multiple sub-segments, function fitting is performed for each sub-segment until the calculated error for each sub-segment is less than a preset error threshold.

[0045] According to an embodiment of the present invention, after step S109, Figure 1 The hardware implementation method shown also requires evaluating the hardware implementation cost corresponding to the fitting function; if the hardware implementation cost is greater than a preset hardware cost threshold, steps S105-S107 are repeated until the hardware implementation cost of each segment is less than the preset hardware cost threshold; or if the hardware implementation cost is greater than the preset hardware cost threshold, a lookup table is used instead of the fitting function. This embodiment significantly reduces the size of the lookup table by using a hybrid structure combining polynomials and lookup tables, while maintaining accuracy close to that of a floating-point library. Furthermore, while maintaining high accuracy, it achieves a high-throughput, low-latency, and low-area / power-consumption hardware implementation, making it adaptable to FPGA or ASIC platforms.

[0046] In some embodiments, when evaluating the hardware implementation cost corresponding to the fitting function, the fitting function is first decomposed to obtain the hardware-executable basic operation unit corresponding to the fitting function; if the basic operation unit includes division, recursive operation and / or higher-order polynomial, then the hardware implementation cost corresponding to the fitting function is considered to be greater than a preset hardware cost threshold.

[0047] In a specific embodiment, during the execution of steps S105-S107, multiple fitting schemes with acceptable accuracy are generated (each scheme may contain multiple equations). Among equations of the same accuracy, a hardware-friendly design is selected, for example, a scheme whose equations contain x. 2,4,8 Methods that involve addition, subtraction, and integer multiplication are preferred to reduce the consumption of hardware resources.

[0048] In other embodiments, for hardware-incompatible implementations, or for certain items in the equations such as division, exponentiation, logarithmic operations, or decimal operations, a lookup table is used, and the input range of the lookup table is controlled within the range of 8 and 9 (storing 256,512 data).

[0049] Since numerical overflow affects subsequent calculation modules, in some other embodiments, when overflow or superflow occurs during the calculation process, bidirectional boundary correction is performed on the overflow value.

[0050] For example, assuming the range of subsequent calculations is -127 to +127, output data less than -127 during the calculation process will be clamped to -127, and output data greater than 127 will be clamped to 127.

[0051] This embodiment evaluates the hardware implementation cost of the fitting function, selects a hardware-friendly fitting function, or uses a lookup table instead of a solution with a high hardware implementation cost, thereby avoiding waste of hardware resources and reducing the implementation cost of the product.

[0052] In some embodiments, after determining the hardware implementation scheme of the exponential operation in the activation function in step S109, it is also necessary to perform a normalization operation on the function coefficients and store the coefficients of the fitted function in the lookup table to simplify multiple fitted functions into one function. This allows the hardware to perform calculations by simply looking up the corresponding coefficients in the lookup table based on the input data range and using the simplified function to complete the calculation, thereby reducing the complexity of hardware calculations.

[0053] According to the hardware architecture embodiment of the present invention, after determining the hardware implementation scheme of the exponential operation in the activation function in step S109, it is also necessary to count the number of basic operation units after the fitting function is decomposed, so as to provide a quantitative basis for the hardware implementation scheme of the exponential operation.

[0054] For example, suppose the final function is divided into three segments for fitting as shown in formulas (1) to (3): y1 = -0.0001649649x² + 0.0070842073x + 3.9975295101 (1) y2 = -0.0000210524x² + 0.0027668325x + 4.0299098208 (2) y3 = 0.0000004615x² + 0.0004863585x + 4.0903423830 (3) As can be seen from formulas (1) to (3), this embodiment requires 6 multipliers, 6 adders, and a calculator for x. 2 Logic circuits, etc. As large models develop, the curves to be fitted become increasingly complex, and the requirements for accuracy become increasingly higher, corresponding to more piecewise equations and higher-order polynomials. Therefore, hardware resource consumption increases exponentially.

[0055] To optimize the hardware architecture implementation, this embodiment adopts a hybrid structure of "single polynomial + LUT", which significantly reduces the lookup table size while maintaining near-floating-point precision.

[0056] The coefficients of formulas (1) to (3) are normalized to the interval [-1, 1] to obtain the corresponding normalized coefficients scale, and the normalized coefficients are stored in the lookup table.

[0057] After normalization, formulas (4) to (6) are obtained, and the corresponding normalization coefficient scale is 4.0903423830. Where, y1 = y1' * scale, y2 = y2' * scale, y3 = y3' * scale.

[0058] y1' = 0.00004033x 2 + 0.001732x + 0.9773(4) y2' = 0.00000515x 2 + 0.000676x + 0.9852 (5) y3' = 0.00000011x 2 + 0.0001189x + 1.0 (6) Based on formulas (4) to (6), we obtain a single polynomial as shown in formula (7), and store the coefficients of (4) to (6) in a lookup table. For example, a 3x3 lookup table.

[0059] x² + x + 1 (7) As can be seen from formula (7), the hardware resources required in this embodiment are 3 multipliers, 2 adders, a lookup table, and a calculator for x. 2 Logic circuits, etc.

[0060] From a hardware perspective, formulas (1) to (3) calculate results for different input ranges and use three different sets of logic to select the corresponding output values. Formulas (4) to (6), through normalization, reduce the range of each coefficient in the formula by using the cost of a multiplier, thereby reducing the data width of the lookup table and achieving the goal of reducing resources. By caching all the coefficients in the formula into the lookup table, and substituting the coefficients into the formula for calculation according to different input ranges, only one formula is used here, which greatly reduces the hardware resources occupied.

[0061] According to the embodiments of this application, even if the coefficients of the multiple polynomial fitting functions determined in the end are different, they can be solved by formula and lookup table as shown in formula (7), thereby achieving the purpose of reducing resource consumption and further reducing hardware resource consumption.

[0062] according to Figure 1 The embodiment shown in the figure uses a multi-segment function fitting method to fit the 16-bit floating-point hardware storage value of the exponentiation operation in the activation function. This ensures the accuracy of the exponentiation operation in the activation function while avoiding the waste of hardware resources and reducing the implementation cost of the product.

[0063] The foregoing mainly describes the embodiments of the present invention from a methodological perspective. Those skilled in the art should readily recognize that, in conjunction with the operations or steps of the various examples described in connection with the embodiments disclosed herein, the present invention can be implemented in hardware or a combination of hardware and computer software. Those skilled in the art can implement the described functions in different ways for each specific operation or method, and such implementations should not be considered beyond the scope of the present invention.

[0064] The following describes embodiments of the apparatus of the present invention. For details not described in the embodiments of the apparatus of the present invention, please refer to the embodiments of the method of the present invention.

[0065] Figure 2 A block diagram of a hardware implementation apparatus for an activation function according to an exemplary embodiment of the present invention is shown, such as... Figure 2The hardware implementation device shown includes a 16-bit floating-point value calculation unit 201, a segmentation unit 203, a fitting unit 205, an error calculation unit 207, and a repetitive calculation unit 209. Specifically, the 16-bit floating-point value calculation unit 201 calculates the 16-bit floating-point value corresponding to the exponential operation of the activation function; the segmentation unit 203 divides the 16-bit floating-point value into multiple segments based on its numerical distribution; the fitting unit 205 performs curve fitting on each segment to obtain the corresponding fitting function; the error calculation unit 207 calculates the error between the fitting function and the corresponding 16-bit floating-point value; and the repetitive calculation unit 209, when the error exceeds a preset error threshold, divides the segment corresponding to the fitting function into multiple segments and repeatedly uses the fitting unit and the error calculation unit until the error of each segment is less than the preset error threshold.

[0066] Figure 3 An electronic device according to an exemplary embodiment of the present invention is shown below. Figure 3 To describe an electronic device 200 according to this embodiment of the present invention. Figure 3 The electronic device 200 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0067] like Figure 3 As shown, the electronic device 200 is presented in the form of a general-purpose computing device. The components of the electronic device 200 may include, but are not limited to: at least one processing unit 210, at least one storage unit 220, a bus 230 connecting different system components (including storage unit 220 and processing unit 210), a display unit 240, etc.

[0068] The storage unit stores program code, which can be executed by the processing unit 210, causing the processing unit 210 to perform the methods described in this specification according to various exemplary embodiments of the present invention. For example, the processing unit 210 can perform, for example... Figure 1 The method shown.

[0069] Storage unit 220 may include readable media in the form of volatile storage units, such as random access memory (RAM) 2201 and / or cache memory 2202, and may further include read-only memory (ROM) 2203.

[0070] Storage unit 220 may also include a program / utility 2204 having a set (at least one) program module 2205, such program module 2205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0071] Bus 230 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0072] Electronic device 200 can also communicate with one or more external devices 300 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 200, and / or with any device that enables electronic device 200 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 250. Furthermore, electronic device 200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 260. Network adapter 260 can communicate with other modules of electronic device 200 via bus 230. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0073] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. The technical solutions of the embodiments of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, portable hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the methods described above according to the embodiments of the present invention.

[0074] Software products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0075] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0076] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0077] The aforementioned computer-readable medium carries one or more programs, which, when executed by a device, cause the computer-readable medium to perform the aforementioned functions.

[0078] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0079] According to an embodiment of the present invention, a computer program is provided, comprising a computer program or instructions, which, when executed by a processor, can perform the methods described above.

[0080] The embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of the present invention, its specific implementation methods, and its application scope, are all within the scope of protection of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A hardware implementation method for exponential operation in an activation function, characterized in that, include: (1) Calculate the 16-bit floating-point value corresponding to the exponentiation operation of the activation function, wherein the 16-bit floating-point value is the hardware storage value of the exponentiation operation of the activation function in BF16 format obtained by the lookup table method. (2) Divide the 16-bit floating-point value into multiple segments according to the numerical distribution of the 16-bit floating-point value; (3) Perform curve fitting on each segment to obtain the corresponding fitting function; (4) Calculate the error between the fitting function and the corresponding 16-bit floating-point value; (5) If the error is greater than the preset error threshold, the segment corresponding to the fitting function is divided into multiple sub-segments, and steps (3)-(4) are repeated for each sub-segment until the error of each sub-segment is less than the preset error threshold.

2. The hardware implementation method according to claim 1, characterized in that, After step (3) and before step (4), it also includes: Check whether the coefficients of the fitted function are within the range of values ​​in 16-bit floating-point format; If the coefficients of the fitted function are not within the range of 16-bit floating-point format, the coefficients of the fitted function are quantized to the range of 16-bit floating-point format.

3. The hardware implementation method according to claim 2, characterized in that, When the coefficients of the fitted function are not within the range of 16-bit floating-point format, quantizing the coefficients of the fitted function to the range of 16-bit floating-point format includes: If the coefficients of the fitted function are not within the range of 16-bit floating-point format, the coefficients of the fitted function are quantized to the range of 16-bit floating-point format by truncation or rounding.

4. The hardware implementation method according to claim 1, characterized in that, Also includes: Evaluate the hardware implementation cost corresponding to the fitted function; If the hardware implementation cost is greater than a preset hardware cost threshold, repeat steps (3)-(5) until the calculated hardware implementation cost of each segment is less than the preset hardware cost threshold.

5. The hardware implementation method according to claim 1, characterized in that, Also includes: Evaluate the hardware implementation cost corresponding to the fitted function; If the hardware implementation cost exceeds a preset hardware cost threshold, a lookup table is used instead of the fitting function.

6. The hardware implementation method according to claim 4 or 5, characterized in that, Evaluating the hardware implementation cost corresponding to the fitted function includes: The fitting function is decomposed to obtain the hardware-executable basic computing unit corresponding to the fitting function; If the basic operation unit includes division, recursive operation and / or higher-order polynomials, then the hardware implementation cost of the fitting function is considered to be greater than a preset hardware cost threshold.

7. The hardware implementation method according to claim 6, characterized in that, Also includes: Normalization is performed on the coefficients of the fitted function, and the coefficients of the fitted function are stored in the lookup table to simplify multiple fitted functions into one function; The number of basic computational units after the fitting function is decomposed is counted to provide a quantitative basis for the hardware implementation scheme of the exponential operation.

8. A hardware implementation device for an activation function, characterized in that, include: A 16-bit floating-point numerical calculation unit is used to calculate the 16-bit floating-point value corresponding to the exponential operation of the activation function; The segmentation unit is used to divide the 16-bit floating-point value into multiple segments according to the numerical distribution of the 16-bit floating-point value. The fitting unit is used to perform curve fitting on each segment to obtain the corresponding fitting function; An error calculation unit is used to calculate the error between the fitting function and the corresponding 16-bit floating-point value; The repeated calculation unit is used to divide the segment corresponding to the fitting function into multiple segments when the error is greater than a preset error threshold, and to use the fitting unit and the error calculation unit multiple times until the error of each segment is less than the preset error threshold.

9. An electronic device, comprising: processor; as well as A memory storing a computer program that, when executed by the processor, causes the processor to perform the exponentiation implementation method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having stored computer-readable instructions thereon, which, when executed by a processor, cause the processor to perform the exponentiation implementation method as described in any one of claims 1-7.

Images