Calculation device and calculation method thereof

Abstract

An arithmetic device and an arithmetic method thereof are provided. The arithmetic device is used for converting input data in a neural network into output data and comprises an arithmetic logic circuit and a control circuit which are connected with each other, and the control circuit sends a configuration setting sequence in one or more periods to configure the arithmetic logic circuit connected with each other. The arithmetic logic circuit jointly performs at least one of a normalized exponent operation or a linear unit operation on the input data. The arithmetic logic circuits connected to each other include an exponent circuit that generates a first exponent of an input decimal portion and a second exponent of an input integer portion, a multiplier circuit that multiplies the first exponent and the second exponent into a first intermediate variable, and an accumulator circuit that sums a plurality of intermediate variables related to a plurality of input exponents. And a reciprocal circuit generating a reciprocal of the second intermediate variable input to the reciprocal circuit.

Description

Technical Field

[0001] The embodiments disclosed herein relate to a computing device and a computing method thereof, and more particularly to a computing device and a computing method thereof for an arithmetic logic unit of a computing processor. Background Technology

[0002] Artificial intelligence (AI) systems can be built on software-based neural network models implemented on one or more AI accelerators, such as graphics processing units (GPUs), tensor processing units (TPUs), and / or the like. AI accelerators may contain specialized components and / or devices to accelerate the execution of AI and machine learning workloads. Existing AI accelerators and / or processors rely heavily on software frameworks and libraries to perform complex computational tasks. Due to the intense computational demands of AI systems, the power consumption of such AI accelerators and / or processors can be substantial. Summary of the Invention

[0003] One embodiment of this disclosure provides a computing device. This computing device is used to transform input data in a neural network into output data, and includes: a plurality of interconnected arithmetic logic circuits and a control circuit. The plurality of interconnected arithmetic logic circuits include: an exponentiation circuit for generating a first exponent of a fractional part of an input and a second exponent of an integer part of the input; a multiplier circuit for multiplying the first exponent and the second exponent to obtain a first intermediate variable; an accumulator circuit for summing a plurality of intermediate variables associated with a plurality of exponents of a plurality of inputs; and a reciprocal circuit for generating a reciprocal of a second intermediate variable input to the reciprocal circuit. The control circuit is used to send a plurality of configured sequences to configure the plurality of interconnected arithmetic logic circuits within one or more cycles, such that one or more of the plurality of interconnected arithmetic logic circuits jointly perform at least one of a normalized exponentiation operation or a sigmoid unit operation on the input data according to the plurality of configured sequences.

[0004] One embodiment of this disclosure provides a computational method for transforming input data in a neural network into output data using multiple interconnected arithmetic logic circuits. The method includes the following steps: generating a first exponent of a fractional part of an input and a second exponent of an integer part of the input via an exponentiation circuit; multiplying the first exponent and the second exponent by a multiplier circuit to obtain a first intermediate variable; summing multiple intermediate variables related to multiple exponents of multiple inputs via an accumulator circuit; generating a reciprocal of a second intermediate variable input to the reciprocal circuit via a reciprocal circuit; sending a multiple configured sequence via a control circuit to configure the multiple interconnected arithmetic logic circuits within one or more cycles; and jointly performing at least one of a normalized exponentiation operation or a sigmoid unit operation on the input data via one or more of the multiple interconnected arithmetic logic circuits according to the multiple configured sequences.

[0005] One embodiment of this disclosure provides a computational method for constructing a circuit that transforms input data in a neural network into output data. The method includes the following steps: placing a plurality of interconnected arithmetic logic circuits on the circuit, wherein the plurality of interconnected arithmetic logic circuits include: an exponentiation circuit that generates a first exponent of a fractional portion of an input and a second exponent of an integer portion of the input; a multiplier circuit that multiplies the first exponent and the second exponent to obtain a first intermediate variable; an accumulator circuit that sums a plurality of intermediate variables associated with a plurality of exponents of a plurality of inputs; a reciprocal circuit that generates a reciprocal of a second intermediate variable input to the reciprocal circuit; and a control circuit that sends a plurality of configured sequences within one or more cycles to configure the plurality of interconnected arithmetic logic circuits, thereby jointly performing at least one of a normalized exponentiation or a sigmoid unit operation on the input data according to the plurality of configured sequences. Attached Figure Description

[0006] The present disclosure describes an embodiment in relation to the accompanying documents. Figure 1 The best way to understand this text is by referring to the detailed description below. Note that, according to industry standards, the features are not drawn to scale. In practice, the dimensions of the features can be arbitrarily increased or decreased for clarity of explanation.

[0007] Figure 1 These are examples of neural network models relating to computational operations for performing classification tasks, according to one or more embodiments described herein;

[0008] Figure 2This is a simplified diagram of an instance structure of a hardware-based computational arithmetic logic unit (ALU) for performing normalized exponent (Softmax) operations or sigmoidal linear unit (SiLU) operations in a neural network, according to one or more embodiments described herein.

[0009] Figure 3 According to one or more embodiments described herein Figure 2 A simplified circuit diagram of the input or output register shown.

[0010] Figure 4 According to one or more embodiments described herein Figure 2 A simplified circuit diagram of the multiplexer-demultiplexer shown in the figure;

[0011] Figure 5 According to one or more embodiments described herein Figure 2 A simplified diagram of the circuit structure of the transmission gate array shown;

[0012] Figure 6A According to one or more embodiments described herein Figure 2 A simplified diagram of the circuit structure of the exponential circuit (ES) shown.

[0013] Figure 6B According to one or more embodiments described herein Figure 6A A simplified diagram of the circuit structure of ES is shown below;

[0014] Figure 6C The illustration is based on one or more embodiments described herein. Figure 6B A simplified diagram of an example structure of the divider circuit described in the diagram;

[0015] Figure 7A According to one or more embodiments described herein Figure 6C A simplified diagram of the divider circuit described in the diagram operating in dequantization mode;

[0016] Figure 7B According to one or more embodiments described herein Figure 6B The simplified diagram of ES operation in dequantization mode described in the figure corresponds to Figure 7A The splitter operation described in the document;

[0017] Figure 8A According to one or more embodiments described herein Figure 6C A simplified diagram of the divider circuit described in the diagram operating in quantization mode;

[0018] Figure 8B According to one or more embodiments described herein Figure 6B The simplified diagram of ES operation in quantization mode described in the figure corresponds to Figure 8A The splitter operation described in the document;

[0019] Figure 9A According to one or more embodiments described herein Figure 2 A simplified circuit diagram of the multiplier circuit shown;

[0020] Figure 9B Use according to one or more embodiments described herein Figure 9A A simplified diagram of the operation of the softmax function of the MP circuit described in the figure;

[0021] Figure 10A According to one or more embodiments described herein Figure 2 A simplified diagram of the AC circuit structure shown;

[0022] Figure 10B According to one or more embodiments described herein Figure 10A A simplified diagram of the AC circuit structure shown;

[0023] Figures 11A to 11D Operation according to one or more embodiments described herein Figures 10A to 10B A simplified diagram of multiple cycles of the AC circuit shown;

[0024] Figure 12 According to one or more embodiments described herein Figure 2 A simplified diagram of the circuit structure of the RC circuit shown;

[0025] Figure 13 According to one or more embodiments described herein Figure 2 A simplified diagram of the control circuit structure shown;

[0026] Figure 14 It is a configuration of instance control signals for different operations;

[0027] Figure 15 This is based on one or more embodiments described herein. Figure 2 A simplified diagram of the exponential calculation operation performed by the ALU is shown.

[0028] Figure 16 This is based on one or more embodiments described herein. Figure 2 A simplified diagram of a single accumulation operation performed by the ALU is shown below;

[0029] Figure 17This is based on one or more embodiments described herein. Figure 2 A simplified diagram of a single reciprocal operation performed by the ALU is shown below;

[0030] Figure 18 This is based on one or more embodiments described herein. Figure 2 A simplified diagram of a single multiplication operation performed by the ALU is shown below;

[0031] Figure 19 This is a simplified diagram of a combination operation based on one or more embodiments described herein, which combines... Figures 15 to 18 The multiple single operations described in the text are used to... Figure 2 The ALU shown performs the Softmax operation;

[0032] Figure 20 This is a simplified diagram of a combination operation based on one or more embodiments described herein, which combines... Figures 15 to 18 The multiple single operations described in the text are used to... Figure 2 The ALU shown performs SiLU operations;

[0033] Figure 21 This is a simplified diagram of a combination operation based on one or more embodiments described herein, which combines... Figures 15 to 18 The multiple single operations described in the text are used to... Figure 2 The ALU shown performs an INT8 softmax operation (dequantizes the input and then quantizes the output);

[0034] Figure 22 This is a use according to the embodiments described herein. Figures 2 to 21 The example logic flowchart describes the process by which multiple interconnected arithmetic logic circuits transform input data in a neural network into output data.

[0035] Figure 23 It is according to one embodiment described herein, in the inclusion Figures 1 to 22 A simplified diagram of a computing device that implements neural networks on an AI accelerator with the circuit structure described in the figure.

[0036] [Symbol Explanation]

[0037] 100: Neural Network Model

[0038] 102: Input Data

[0039] 105: Neuron

[0040] 110: Floor

[0041] 115:logits / logarithmic odds

[0042] 120: softmax operation

[0043] 130: Output data

[0044] 200:ALU

[0045] 202: Control Circuit

[0046] 204:IR

[0047] 206a~206b:TGA

[0048] 208a~208d:MD

[0049] 210:MP

[0050] 210a~210d:MP

[0051] 212:AC

[0052] 215:ES

[0053] 216:RC

[0054] 220:OR

[0055] 221: Control signal

[0056] 222: Clock Signal

[0057] 223: Input data value

[0058] 224:RSTB

[0059] 230: Output value

[0060] 300: Temporary Register Circuit

[0061] 302:RSTB

[0062] 304: Clock signal

[0063] 306: Input value

[0064] 308: Output value

[0065] 400: Circuit Structure

[0066] IN A IN B. IN C Input signal

[0067] 410: Demultiplexer

[0068] OUT A ,OUT B Output signal

[0069] 420: Multiplexer

[0070] 500: TGA circuit

[0071] 501: Selection Signal

[0072] 602: Adder

[0073] 603: Input data value

[0074] 604: Divider Circuit

[0075] 606: Demultiplexer

[0076] 610: LUT circuit

[0077] 612~614: Multiplexer

[0078] 620: Taylor Term Calculation Circuit

[0079] 622: Integer part

[0080] 624: Decimal part

[0081] 631: Adder

[0082] 632: Shift Counter

[0083] 634: Shifter

[0084] 640: Plus / Minus Combinator

[0085] 642: Normalized Circuit

[0086] 903: Multiplication Output

[0087] 905: Multiplexer

[0088] 910: Connector Array

[0089] 1002: Input value

[0090] 1004: Output value

[0091] 1010~1018: Multiplexer

[0092] 1019: Temporary Register

[0093] 1020: Temporary Register

[0094] 1022: Adder

[0095] 1101~1107: Data Path

[0096] 1202:IN

[0097] 1204:OUT

[0098] 1901–1905: Data Path

[0099] 2001–2003: Data Path

[0100] 2101~2106: Data Path

[0101] 2200: Method

[0102] 2202~2210: Steps

[0103] 2300: Computing device

[0104] 2310: Processor

[0105] 2315: Data Interface

[0106] 2320: Memory

[0107] 2330: AI Accelerator

[0108] 2331: Neural Networks

[0109] 2340: Input

[0110] 2350: Output Detailed Implementation

[0111] The following disclosure provides numerous different embodiments or instances for implementing various features of this disclosure. Specific examples of components and configurations are described below to simplify one embodiment of this disclosure. These are, of course, merely examples and are not intended to be limiting. For instance, in the following description, the formation of a first feature above or on a second feature may include embodiments where the first and second features are formed in direct contact, and may also include embodiments where additional features may be formed between the first and second features such that the first and second features are not in direct contact. Furthermore, references to numbers and / or letters may be repeated in various instances of an embodiment of this disclosure. This repetition is for simplicity and clarity and does not, in itself, indicate a relationship between the various embodiments and / or configurations discussed.

[0112] Furthermore, for ease of description, spatial relative terms such as “below,” “under,” “lower,” “above,” “upper,” and the like are used herein to describe the relationship between one element or feature illustrated in the figures and another element(s). Spatial relative terms are intended to cover different orientations of the device during use or operation, other than those depicted in the figures. Devices may be oriented in other ways (rotated 90 degrees or otherwise), and the spatial relative descriptors used herein can be interpreted similarly.

[0113] As used herein, the term “network” may include any hardware- or software-based framework, including any artificial intelligence network or system, neural network or system, and / or any training or learning model implemented thereon or in conjunction with it.

[0114] As used herein, the term "module" may encompass a hardware- or software-based framework that performs one or more functions. In some embodiments, the module may be implemented on one or more neural networks.

[0115] In recent years, the rapid development of artificial intelligence (AI) and machine learning has had a significant impact on various industries, from healthcare and finance to automotive and consumer electronics. As AI systems become increasingly complex, their computational demands are also escalating, driving the need for more efficient and powerful processing solutions. Traditional central processing units (CPUs) often struggle to keep up with these demands, leading to the widespread adoption of AI accelerators such as graphics processing units (GPUs) and tensor processing units (TPUs). GPUs and TPUs are better suited for AI applications than CPUs because they can handle the massive amounts of parallel processing required for AI and machine learning tasks. Unlike CPUs, which are optimized for general-purpose computing, GPUs and TPUs are designed to perform thousands of operations simultaneously, making them ideal for handling large datasets and complex algorithms. This parallelism significantly accelerates the training and inference processes in AI models, resulting in faster and more efficient computation. Furthermore, GPUs and TPUs are optimized for the specific mathematical operations that underpin AI workloads, further enhancing their performance in these applications.

[0116] AI accelerators have become a key component for deploying AI models, especially in tasks requiring massive parallel processing capabilities, such as deep learning. These specialized hardware components are designed to optimize the performance of AI workloads, enabling faster processing times and more efficient resource utilization. However, this performance improvement often comes at the cost of higher power consumption, posing significant challenges in energy efficiency and thermal management.

[0117] This application relates to computing circuits, and more specifically, to methods and apparatus for using hardware-based application-specific integrated circuits (ASICs) to perform neural network operations across different input data formats, such as softmax operations, sigmoidal linear unit (SiLU) operations, and / or the like. The embodiments described herein provide an arithmetic logic unit (ALU) circuit for calculating complex neural network operations, such as softmax operations or sigmoidal linear unit (SiLU), on input data values ​​such as Floating Point 16-bit (BF16), Half-Point 16-bit (FP16), 16-bit floating-point data types primarily used for machine learning and AI computation, or 8-bit integers (INT8).

[0118] In one embodiment, the ALU circuit supports both softmax and SiLU operations, enabling data and hardware reuse across different digital formats. The ALU hardware includes arithmetic units (e.g., addition, multiplication, reciprocal, and exponentiation) interleaved with Mux-Demux (MD) units to create different data paths between arithmetic units. Hardware reuse between complex functions and basic arithmetic operations is achieved by decomposing complex functions (such as softmax, SiLU) into basic arithmetic operations (e.g., addition, multiplication, reciprocal, and exponentiation) and controlling the MD units to control the input and output data paths corresponding to different arithmetic operations.

[0119] In one embodiment, hardware reuse between number formats is achieved by dividing operand bits into {sign, exponent, and mantissa} fields according to the definitions of FP16, BF16, and INT8.

[0120] In one embodiment, the ALU circuitry further supports softmax in integer mode, where hardware reuse is achieved by sharing the integer dequantization and quantization units within the exponent unit of a Taylor-based floating-point lookup table (LUT). The dequantization step is replaced by an existing LUT within the ALU. Integer inputs are converted to LUT addresses, and the LUT provides the exponent output in floating-point format, such as... Figures 7A to 7B As shown. The quantization step is replaced by an exponential divider and a 128x readjustment ratio. For example... Figures 8A to 8BAs further described, the 128-fold readjustment ratio is implemented by adding seven to the exponent bits (or subtracting seven from the exponent deviation).

[0121] In this way, computational ALUs can be used as on-chip ALUs for complex computations (e.g., softmax and SiLU, and / or similar) in AI accelerators. This type of hardware-based computation allows for fast convergence and high accuracy, as well as efficient circuit area utilization and low power consumption. Similarly, compared to software-based computation on GPUs, hardware-based computational ALU units require less GPU memory access. Therefore, the hardware efficiency of neural network deployments is improved.

[0122] Figure 1 The illustration shows an example of a neural network model 100 relating to computational operations for performing a classification task according to one or more embodiments described herein. In one embodiment, the neural network 100 comprises a computational system built on a collection of connected units or nodes (referred to as neurons 105). Neurons are often connected by edges, and adjustable weights are often associated with the edges. Neurons are often clustered into layers 110 such that different layers can perform different transformations on individual inputs and output the transformed input data to the next layer.

[0123] For example, as each neuron receives input signals, the input layer receives input data 102, performs a weighted summation of the input according to the weights assigned to each connection, and then applies an activation function associated with the individual neuron to the result. The output of the activation function is passed to neurons in the next layer or used as the final output of the network. The activation functions may be the same or different across different layers. Instance activation functions include, but are not limited to, Sigmoid, hyperbolic tangent, Rectified Linear Unit (ReLU), Leaky ReLU, softmax, SiLU, and / or similar. In this way, after many layers, the input data 102 received at the input layer is transformed into considerably different values ​​representing data characteristics corresponding to the tasks that the neural network structure is designed to perform.

[0124] For example, input data 102 may contain images, and neural network 100 may be a classification model trained to classify objects in the input images. The input images may be processed by layer 110 containing transformations of activation functions such as SiLU. For example, the activation of SiLU is calculated by multiplying its input by a sigmoid function:

[0125]

[0126] Where x i This represents the input data value for SiLU.

[0127] The output layer can output logits 115, indicating the probability that the input image may contain one of a predefined object category (e.g., apple, orange, ..., dog, cat). A softmax operation 120 can be performed to generate output probabilities for each category based on the output logits at the output layer. For example, the softmax operation is a normalized exponential function:

[0128]

[0129] Where x represents the output probability vector for n categories. In this process, the computation of the neural network 100 involves a large number of exponential calculations, such as in softmax operations, SiLU operations, and / or similar operations.

[0130] Traditionally, these complex operations, such as softmax or SiLU, have mostly been performed by software implemented on GPUs, TPUs, or other AI accelerators. Known AI accelerators typically have dedicated hardware for different digital formats, leading to higher circuit area overhead. Furthermore, performing complex activation functions (such as softmax) in integer mode within neural networks typically involves (1) dequantizing the integer input to floating-point, (2) performing floating-point arithmetic, and (3) quantizing the floating-point output back to integer. Dequantization and quantization operations are often again handled by additional dedicated hardware hardwired in silicon, further increasing circuit area overhead.

[0131] Figure 2 This is a simplified diagram illustrating an example structure of a hardware-based computational ALU 200 for performing softmax or SiLU operations in a neural network according to one or more embodiments described herein. The ALU 200 may include multiple logic circuits, such as an input register (IR) 204, a transmission gate array (TGA) 206a-b, multiple multiplexers-demultiplexers (Mux-Demux) 208a-d, an accumulator circuit (AC) 212, a multiplier circuit (MP) 210, an exponential circuit (ES) 215, a reciprocal circuit (RC) 216, an output register (OR) 220, and / or the like. These arithmetic logic circuits may be interconnected according to different control configurations to jointly perform calculations on input data values ​​223 and output an output value 230.

[0132] In one embodiment, ALU 200 may further include control circuitry 202 that receives control signal 221 and subsequently configures various circuit modules within ALU 200. For example, control signal 221 may include a reset (RST) bit, bits indicating the mode of the input data value (MODE = numeric format, FP8 or BF16), bits indicating the configuration of one of ES, AC, RC, or MP (CFG), bits indicating the operation of one of ES, AC, RC, or MP (OP), and / or similar. Control circuitry 202 may receive control signal 221 and subsequently configure circuits ES, MD, MP, AC, and RC based on control signal 221.

[0133] In one embodiment, control circuitry 202 and IR 204 can be synchronized via clock signal 222 and / or reset by RSTB 224. In this manner, input data values ​​223 in the format FP8, BF16, FP16, and / or similar (e.g., representing...) Figure 1 The intermediate variables in one of the layers 110 of the neural network 100 shown are input to the ALU 200 to calculate output values ​​230 in the formats FP8, BF16, FP16, and / or similar, respectively. Depending on the type of computation configured by various control settings according to the control signal 221, the output value 230 may represent the softmax operation result, SiLU operation result, and / or similar of the input value 223.

[0134] In one embodiment, the ALU 200 can support input data value 223 using both FP16 and BF16 data formats. For example, the exponent and mantissa of data types FP16 and BF16 have different bit counts: FP16 = {1b sign, 5b exponent, 10b mantissa}, BF16 = {1b sign, 8b exponent, 7b mantissa}. Each arithmetic unit, such as ES, MD, MP, AC, or RC, can accommodate the maximum bit width of each component in FP16 and BF16, while data propagation between units remains in either FP16 or BF16.

[0135] Specifically, within each arithmetic unit ES, MD, MP, AC, or RC, data variables in FP16 or BF16 format can be padded with leading or trailing zeros to adopt the form {1b sign, 8b exponent, 10b mantissa}. For example, for both FP16 and BF16 data types, the original 1-bit sign remains unchanged. For the FP16 data type, the 8-bit exponent can be padded with zeros to {3'b0, 5b exponent} or {5b exponent, 3'b0}, while the 10-bit mantissa remains unchanged. For the BF16 data type, the original 8-bit exponent remains unchanged, and the 10-bit mantissa is padded with zeros to {3'b0, 7b mantissa} or {7b mantissa, 3'b0}. Across arithmetic units ES, MD, MP, AC, or RC, the data type FP16 or BF16 remains unchanged.

[0136] In one embodiment, one or more multiplexer-demultiplexer (mux-demux) units MD 208a-208d can interleave with multiple interconnected arithmetic logic circuits. These MD 208a-208d are controlled by one or more selection signals to form one or more data paths between the interconnected arithmetic logic circuits. The one or more data paths correspond to sequences of arithmetic operations performed by one or more interconnected arithmetic logic circuits. When combined in a specific order, sequences of arithmetic operations such as exponentiation, accumulation, multiplication, and reciprocal can be combined to form complex operations such as softmax or SiLU operations.

[0137] Figure 3 The illustration is based on one or more embodiments described herein. Figure 2 The diagram shows a simplified circuit structure of the register for IR 204 or OR 220. In one embodiment, the register circuit 300 may have synchronous inputs and outputs. The input side of the register circuit 300 may include multiple input pins for a one-bit reset signal (RSTB) 302 for resetting the register, a one-bit clock signal 304 for synchronization, and multiple bits for input value 306. The input value 306 may be captured by the register circuit 300 at the rising edge of the clock signal 304, and then output as output value 308 from the register circuit 300 at the rising edge of the clock signal 304.

[0138] Figure 4 The illustration is based on one or more embodiments described herein. Figure 2 The diagram shows a simplified circuit structure of MD 208a-208d. Figure 4 As shown, MD 208a-208d can adopt circuit structure 400, which includes a demultiplexer 410 connected to the multiplexer 420. The multiplexer and the demultiplexer jointly control the input signal IN. AIN B The data path between them, to output the output signal OUT A OUT B For example, when S IN When the value is 1, the demultiplexer 410 can receive a one-bit input signal IN. B And output it to the output signal OUT. B Otherwise, when S IN When the value is 0, the input signal 402 is transmitted to the input terminal of the multiplexer 420.

[0139] In one embodiment, the input signal IN A The signal is transmitted to multiplexer 420. In this way, when the selection signal S... OUT When it is 00, the output signal OUT A The value is 0; when the selection signal S is selected. OUT When it is 0 or 1, the input signal IN will be... A Transmitted to output signal OUT A When the selection signal S OUT When the value is 10, the input signal IN will be... B Transmitted to output signal OUT A When the selection signal S OUT When the input signal is 11, the input signal IN will be... C Transmitted to output signal OUT A .

[0140] Figure 5 The illustration is based on one or more embodiments described herein. Figure 2 The diagram shows a simplified circuit structure of TGA 206a-206b. In one embodiment, TGA circuit 500 may include n pairs of transmission gates, each connected to the input bit IN and the ground signal. The input data IN[0:n] is then selectively passed to the output terminal OUT[0:n]. When the selection signal 501 is 1'b1 (e.g., a 1-bit width unsigned integer value = 1), the input is passed to the output OUT, for example, IN[i] = OUT[i], i = 0, 1, ..., n. Otherwise, when the selection signal 501 is 1'b0 (a 1-bit width unsigned integer value = 0), the output terminal OUT[i] = GND.

[0141] Figure 6A The illustration is based on one or more embodiments described herein. Figure 2The diagram shows a simplified circuit structure of the ES215. In one embodiment, the ES215 may receive a clock signal CLK, an enable signal EN, an input data value IN (e.g., in FP16 or BF16 data format), a mode signal MODE indicating the input data format, and a control setting CFG. Specifically, CFG[1:0] indicates the number of cycles for exponential calculation, and CFG[2] indicates whether the negative exponent increment mode is enabled to calculate 1+e for a given input value x. -x .

[0142] The operation of the ES215 can be synchronized via the clock signal CLK. For example, when EN=1 and CLK is on the rising edge, the ES215 can divide IN into operator IN1. A (Integer part) and IN B (The decimal part) is calculated separately to calculate the exponent of IN. Figure 6B The code further provides the computation operator IN. A (Integer part) and IN B Additional circuit details for the exponent (decimal part).

[0143] In one embodiment, as defined by CFG[1:0], the operator IN A (Integer part) and IN B The calculated exponential result (decimal part) is driven to OUT after N cycles. A and OUT B The ES215 can contain temporary registers, allowing OUT to... A and OUT B Both have been temporarily stored.

[0144] Specifically, if CFG[2] = 1'b1, meaning the negative exponent plus one mode is enabled, then the sign bits of the input IN are reversed, and one of the outputs (i.e., pre_OUT) is processed at adder 602. A Add one to produce the final OUT. A .

[0145] Figure 6B The illustration is based on one or more embodiments described herein. Figure 6A The diagram shows a simplified circuit structure of the ES215. Figure 6B As shown, the ES215 includes a splitter circuit 604, a demultiplexer 606, a lookup table (LUT) circuit 610, a Taylor term calculation circuit 620, and multiple multiplexers 612 and 614.

[0146] In one embodiment, neural network weights can be quantized, such as using FP16 or BF16 data formats, to reduce the precision of the weights from a higher bit width (e.g., 16-bit floating-point) to a lower bit width (e.g., 8-bit integer). Weight quantization helps reduce the memory footprint and computational requirements of the neural network, making it more efficient to deploy, especially in resource-constrained environments such as mobile devices and edge computing. For example, for 8-bit quantization, a continuous range of weights might be mapped to the integer range [-128, 127]. During inference, the quantized weights are dequantized back to floating-point for computation, but retain their reduced precision. For example, for the softmax operation, which requires an exponential normalization function to produce a fractional output, in integer mode, the integer input is first converted to floating-point (e.g., dequantization), floating-point arithmetic is performed, and then the floating-point output is converted back to integer mode (e.g., quantization). Traditionally, both dequantization and quantization require separate dedicated hardware. Here, as... Figures 6B to 6C As shown, the ES215 operates using the same hardware circuitry in quantization or dequantization modes, using control signals CFG_INT8[1] and CFG_INT8[0].

[0147] In one embodiment, given input data value 603 (e.g., similar to...), Figure 6A The IN value (which can be in FP16 or BF16 format) is used by the divider circuit 604 to divide the input data value 603 into an integer part and a fractional part. The following... Figure 6C The example circuit structure of the divider circuit 604 is further described below.

[0148] The integer portion can be passed to a demultiplexer 606 controlled by the control signal CFG_INT8, which indicates whether the integer portion is undergoing dequantization or quantization. For example, the ES215 supports integer mode computation (e.g., input data 604 has INT8 format), where hardware reuse is achieved by sharing the integer dequantization and quantization units in floating-point LUT-Taylor based units 610 and 620.

[0149] Specifically, when CFG_INT8[1] is 0, this means that the dequantization mode is selected for the INT8 input data value 603, and the integer part is passed to the LUT circuit 610. The integer part is converted to the LUT address of LUT 610, and LUT 610 can retrieve the pre-stored exponent value e in the floating point. integer Here, when the input data value 603 is an integer, the fractional part of the input data value 603 is 0, so the exponent of the fractional part is output as 1. When CFG_INT8[1] is 1, this means that the quantization mode is selected, and a 128-fold readjustment ratio is implemented by adding 7 to the exponent bits (or subtracting 7 from the exponent deviation).

[0150] In one embodiment, the fractional part from the divider circuit 604 can be sent to the Taylor term calculation circuit 620, which can then calculate e as the sum of a finite number of Taylor expansion terms of the fractional part. fractional The approximate value. Similar to the integer part, the Taylor term calculation circuit 620 is enabled or disabled using the control signals CFG_INT8[1] or CFG_INT8[0], which indicate whether dequantization mode or quantization mode is occurring. For example, when CFG_INT8[1] indicates that dequantization mode is selected, the Taylor term calculation circuit 620 is not disabled (and is therefore enabled) because the fractional part is 0, so e fractional =1. Alternatively, when CFG_INT8 indicates that quantization mode is selected, the fractional part becomes zero after being readjusted by 128, therefore e fractional =1. The multiplexer 614 can further select the exponent, such as 1 or e, to pass the output 624 as a fractional part based on the control signal CFG_INT8[1] indicating whether the circuit is operating in quantization / dequantization mode or FP mode. fractional .

[0151] exist Figure 6B In the exponent calculation circuit shown, the Taylor term calculation circuit 620 uses a predefined fixed number of Taylor expansion terms, for example, N = 3, 4, 5, etc.

[0152] Traditionally, after calculating the exponent of the integer part 622 and the exponent of the fractional part 624, a multiplier is used to multiply these two parts to produce the final exponent value of the input value 603. Figures 6A to 6B In the ES215, the exponent of the integer part 622 and the exponent of the fractional part 624 can be output separately and reused. Figure 2 Multiplication is performed in MP 210a of ALU 200.

[0153] Figure 6C The illustration is based on one or more embodiments described herein. Figure 6B A simplified diagram of an example structure of the divider circuit 604 described herein. The divider circuit 604 may include an adder 631, a shift counter 632, a shifter 634, a sign combiner 640, and a normalization circuit 642.

[0154] In one embodiment, the input data value 603 (e.g., in BF16 or FP16 data format) can be decomposed into its sign, mantissa, and exponent, such as Figure 2As shown. Specifically, adder circuit 631 can add 7 to the exponent according to the control signal CFG_INT8[1] indicating whether dequantization or quantization mode is in effect. For example, when CFG_INT8[1] = 1'b1, indicating quantization mode, adder circuit 631 adds 7 to the exponent or subtracts 7 from the exponent.

[0155] The adder result is passed to shift counter 632, which then shifts the exponent bits by a number of bits to obtain the number of shifted bits and the fractional flag (indicating the presence of a fractional part). These two outputs from shift counter 632, along with the mantissa bits, are then passed to shift circuit 634. Shift circuit 634 then shifts the bits to produce an unsigned integer part INT and an unsigned fractional part FRAC.

[0156] The sign combiner circuit 640 combines the sign bits, fraction flag, and unsigned integer INT from the input value 603 to output the integer part integer(x). The normalization circuit 642 further combines the sign bits, fraction flag, and unsigned fractional part FRAC, and further normalizes the unsigned fractional part FRAC to output the fractional part fractional(x).

[0157] Figure 7A The illustration is based on one or more embodiments described herein. Figure 6C A simplified diagram of the operation of the divider circuit 604 in dequantization mode as described in the diagram. The divider circuit 604 operates in dequantization mode when the control signal CFG_INT8[0] = 1 and CFG_INT8[1] = 0. The input data value 603 (i.e. x) can represent the neural network weights that have been mapped to 8-bit integers (INT8) in the range of [-128, 127].

[0158] In this case, when the control signal CFG_INT8[1] = 0, the adder circuit 631, under the control of the control signal CFG_INT8[1] = 0, will not add any value to the exponent of the input data value 603. In other words, the original exponent bits are passed to the shift counter, as shown in the following... Figure 6C As shown. In this way, the sign combiner circuit 640 outputs the original integer part of the input data value 603. For the input data value 603 in INT8 form, the fractional part is zero, so the normalization circuit 642 outputs fractional(x) as zero.

[0159] Figure 7B The illustration is based on one or more embodiments described herein. Figure 6B The simplified diagram of the ES215 operation in dequantization mode described in the figure corresponds to Figure 7A The divider operation described in the text. When CFG_INT8[0] = 1 and CFG_INT8[1] = 0 indicates dequantization mode, the divider circuit 604 outputs the original integer part (integer) and the fractional part (0) of the input data value 603. Under the control of the control signal CFG_INT8[1] = 1, the demultiplexer 606 can pass the integer part (integer) to the LUT circuit 610, which can retrieve the pre-stored exponent value (i) in the floating point. nteger Next, under the control of the control signal CFG_INT8[1] = 0, the multiplexer 612 can convert the FP e from the LUT circuit 610 into PFE. nteger The value is chosen as the exponent of the integer part.

[0160] In one embodiment, for a fractional part of 0, the multiplexer 614 can further select e under the control of the control signal CFG_INT8[0]=1. 0 =1 passes the exponent as the decimal part.

[0161] Figure 8A The illustration is based on one or more embodiments described herein. Figure 6C A simplified diagram of the operation of the divider circuit 604 in quantization mode, as described in the diagram. The divider circuit 604 operates in quantization mode when the control signal CFG_INT8[0] = 0 and CFG_INT8[1] = 1. The input value 603 (i.e., x) can represent a floating-point value, which may be the result of operations such as softmax, 0.045, -0.132, or 0.913 with 16-bit precision.

[0162] Specifically, in order to map the FP input data value 603 to INT8 (quantization), the adder circuit 631, under the control of the control signal CFG_INT8[1] = 1, can add 7 to the exponent of the input data value 603 (equivalent to multiplying the input data value 603 by 2). 7 =128). In other words, add 7 to the original exponent bits and pass them to the shift counter, as about Figure 6C As shown. In this manner, the sign combiner circuit 640 outputs an integer portion, which is the original integer portion of the input data value 603 divided by 2. 7 = 128 scaling (e.g., integer × 128).

[0163] For the fractional part, after the input value x is mapped to an 8-bit integer (INT8) in the range of [-128, 127], the fractional part from the divider circuit 604 is zero.

[0164] Figure 8B The illustration is based on one or more embodiments described herein. Figure 6BThe simplified diagram of the ES215 operation in quantization mode described in the figure corresponds to Figure 8A The divider operation described in [the document] is as follows. In one embodiment, when CFG_INT8[0] = 0 and CFG_INT8[1] = 1 indicates quantization mode, the divider circuit 604 outputs an integer × 128 integer part and a fractional part of 0. The demultiplexer 606 can, under the control of the control signal CFG_INT8[1] = 1, pass the integer part of the integer × 128 directly from the divider circuit 604 to the multiplexer 612. The LUT 610 is no longer needed here because the ES215 is used to quantize the output (the exponent has already been calculated). Then, the multiplexer 612 can, under the control of the control signal CFG_INT8[1] = 1, select the integer × 128 value as the integer part of the output.

[0165] In one embodiment, for a fractional part of 0, the multiplexer 614 can further select e under the control of the control signal CFG_INT8[1]=1. 0 =1 passes the output as the decimal part.

[0166] Figure 9A The illustration is based on one or more embodiments described herein. Figure 2 The diagram shows a simplified circuit structure of the MP 210. In one embodiment, the MP 210 can generate an input signal IN based on various control signals. A IN B The multiplication output is 903. The MP 210 uses the clock signal CLK and... Figure 2 The other circuits in the ALU 200 shown are synchronized and can be reset via the RST signal. The EN signal enables the MP circuit 210; the MODE signal indicates the digital format of the MP circuit 210; the CFG control signal contains two bits to indicate three configurations, for example, when CFG[1:0] = 2'b00, the EN signal in each MP element... A With IN B Multiplication is performed between them; when CFG[1:0] = 2'b01, the IN values ​​of different MP elements are multiplied. A Multiplication is performed between them; when CFG[1:0] = 2'b1X, in the IN of each MP element A With IN B Multiplication is performed between [0] (multicast).

[0167] In one embodiment, the CFG[1] control signal can be selected by the multiplexer 905 to whether to send the input signal IN. B Or from the output and input signal IN of RC circuit 216 A Multiplication. Combination. Figures 9A to 9BAdditional details are described regarding the operation of MP circuit 210 in conjunction with other circuit modules in ALU 200.

[0168] Figure 9B The illustration shows the use according to one or more embodiments described herein. Figure 9A A simplified diagram of the operation of the softmax function computation elements of the MP circuit 210 described herein. In one embodiment, it should be noted that... Figure 9B In the example shown, the four MP circuits 210a-210d are placed in parallel in one implementation. In another implementation, only one MP circuit 210 can be used to perform multiplication operations sequentially.

[0169] In one embodiment, the connection array 910 can convert the input signal IN according to the control signal CFG[1:0]. A IN B The bits are connected to different MP circuits.

[0170] For example, for the first time step, when the control signal CFG[1:0] = 2'b00, the connector array 910 uses a solid arrow to indicate the input bit IN. A and input IN B Connect to individual MP circuits 210a to 210d; in this case, MP circuit 210a will generate OUT[3] = IN. A [3]×IN B [3], MP circuit 210b will generate OUT[2]=IN A [2]×IN B [2], MP circuit 210c will generate OUT[1]=IN A [1]×IN B [1], MP circuit 210d will generate OUT[0]=IN A [0]×IN B [0].

[0171] For the second time step, when the control signal CFG[1:0] = 2'b01, the connector array 910 uses the dashed arrow to direct the input bit IN. A and input IN B Connect to individual MP circuits 210a to 210d; in this case, MP circuit 210a will generate OUT[3] = 0, and MP circuit 210b will generate OUT[2] = IN. A [2]×IN A [3], MP circuit 210c will generate OUT[1] = 0, MP circuit 210d will generate OUT[0] = IN A [0]×IN A [1].

[0172] For the third timing step, when the control signal CFG[1:0] = 2'b1X, the connector array 910 uses a dotted arrow to direct the input bit IN. A and input IN B Connect to individual MP circuits 210a to 210d; in this case, MP circuit 210a will generate OUT[3] = IN. A [3]×IN B [0], MP circuit 210b will generate OUT[2]=IN A [2]×IN B [0], MP circuit 210c will generate OUT[1]=IN A [1]×IN B [0], MP circuit 210d will generate OUT[0]=IN A [0]×IN B [0].

[0173] In this way, MP circuits 210a-210d can be used to calculate the softmax operation, for example, for the third step, IN A [3] = x1, IN A [2] = x2, IN A [1] = x3, IN A [0] = x4, IN B [0] = 1 / (x1+x2+x3+x4), and then MP circuits 210a to 210d generate x1 / (x1+x2+x3+x4), x2 / (x1+x2+x3+x4), x3 / (x1+x2+x3+x4), and x4 / (x1+x2+x3+x4) respectively.

[0174] Figure 10A The illustration is based on one or more embodiments described herein. Figure 2 The diagram shows a simplified circuit structure of the AC circuit 212. In one embodiment, the AC circuit 212 can calculate the sum (e.g., "accumulation") between the input value 1002 and the output value 1004. The AC circuit 212 and... Figure 2Other circuits in the ALU 200 shown are synchronized using the clock signal CLK and can be reset via the RST signal. AC circuit 212 includes a register that allows the output value OUT to be temporarily stored and driven out at the rising edge of the CLK signal. The EN signal enables AC circuit 212; the MODE signal indicates the digital format of AC circuit 212; the CFG control signal contains two bits to indicate four configurations, for example, CFG[1:0] = 2'b00, channel-by-channel accumulation OUT[t=1] = IN[t=1] + OUT[t=0]; CFG[1:0] = 2'b01, cross-channel accumulation OUT[n] = OUT[n] + OUT[n+1]; CFG[1:0] = 2'b10, cross-channel accumulation OUT[n] = OUT[n] + OUT[n+2]; CFG[1:0] = 2'b11, cross-channel accumulation OUT[n] = OUT[n] + OUT[n+4]. The following is combined with... Figure 10B Describes additional operations based on control signals CFG[1:0].

[0175] Figure 10B The illustration is based on one or more embodiments described herein. Figure 10A A simplified diagram of the circuit structure of AC circuit 212 shown. Figure 10B As shown, AC circuit 212 can receive 8-bit integers IN[0:7]. Each bit IN[0] to IN[7] can be selectively fed to an adder through one or more multiplexers to calculate the cumulative sum.

[0176] In one embodiment, the control signal CFG[1:0] may be further mapped to selection signals for various multiplexers 1010, 1014, 1016, 1018, and / or similar devices within the AC circuit 212. For example, CFG[1:0] may be mapped to selection signal ST0 of multiplexer 1016, selection signal ST1 of multiplexer 1014, ST2 of multiplexer 1010, SP[1] of multiplexer 1018, and / or similar devices. Table 1 provides the mapping from CFG[1:0] to ST0, ST1, ST2, and SP[1:7].

[0177] Table 1. AC Control Signals

[0178]

[0179]

[0180] In one embodiment, the output bits can be accumulated. For example, OUT[1] calculated from a previous time step can be output from register 1020 and added at adder 1022 to OUT[0] from register 1019. Similarly, OUT[2], ..., OUT[7] can be accumulated to another output bit. Additional details regarding the accumulation operation of circuit structure 212 can be found in the following combination Figures 11A to 11D Describe it.

[0181] Figures 11A to 11D The illustration shows the operation according to one or more embodiments described herein. Figures 10A to 10B A simplified diagram of multiple cycles of the AC circuit 212 shown. (See diagram below.) Figure 11A As shown, at period #1, the control signal CFG[1:0] = 2'b00, which, according to Table 1, is mapped to control signals ST0 = 1'b0, ST1 = 1'b0, ST2 = 1'b0, and SP[1:7] = 7'b0000000. The data paths are shown with bold arrows. Therefore, assuming all OUT = 0 before period #1, the input IN[0] can be passed to the adder via data path 1101 through multiple multiplexers controlled by selection signals ST0, ST1, and ST2, which will produce OUT[0] = IN[0]. IN[1] is also passed to the adder to produce OUT[1] = IN[1]. IN[2] is passed to the adder via data path 1103, which will produce OUT[2] = IN[2]. Similarly, IN[3], IN[4], IN[5], IN[6], and IN[7] are all passed to individual adders, for example, via data paths 1105, 1107, and / or similar, to produce OUT[n]|t=1=IN[n]|t=1.

[0182] At period #2, the control signal CFG[1:0] = 2'b00, which, according to Table 1, maps to control signals ST0 = 1'b0, ST1 = 1'b0, ST2 = 1'b0, and SP[1:7] = 7'b0000000. Then, IN[n] continues along the same data path 1101, 1103, 1105, 1107 to the individual adders. Therefore, the adders will produce OUT[n]|t = 2 = OUT[n]|t = 1 + IN[n]|t = 2.

[0183] like Figure 11BAs shown, at period #3, the control signal CFG[1:0] = 2'b01. According to Table 1, this signal is mapped to control signals ST0 = 1'b1, ST1 = 1'b0, ST2 = 1'b0, and SP[1:7] = 7'b1010101. Therefore, the data path formed by the multiplexer controlled by the control signal is shown with a bold arrow. For example: OUT[0]|t = 3 = OUT[0]|t = 2 + OUT[1]|t = 2 = sum(OUT[1:0])|t = 2;

[0184] OUT[1]|t=3=OUT[1]|t=2+IN[1]|t=3;

[0185] OUT[2]|t=3=OUT[2]|t=2+OUT[3]|t=2=sum(OUT[3:2])|t=2;

[0186] OUT[3]|t=3=OUT[3]|t=2+IN[3]|t=3;

[0187] OUT[4]|t=3=OUT[4]|t=2+OUT[5]|t=2=sum(OUT[5:4)|t=2;

[0188] OUT[5]|t=3=OUT[5]|t=2+IN[5]|t=3;

[0189] OUT[6]|t=3=OUT[6]|t=2+OUT[7]|t=2=sum(OUT[7:6)|t=2;

[0190] OUT[7]|t=3=OUT[7]|t=2+IN[7]|t=3.

[0191] like Figure 11C As shown, at period #4, the control signal CFG[1:0] = 2'b10. According to Table 1, this signal is mapped to control signals ST0 = 1'b1, ST1 = 1'b1, ST2 = 1'b1, and SP[1:7] = 7'b1110111. Therefore, the data path formed by the multiplexer controlled by the control signals is shown with bold arrows. For example: OUT[0]|t = 4 = sum(OUT[3:0]|t = 3);

[0192] OUT[4]|t=4=sum(OUT[7:4]|t=3).

[0193] like Figure 11DAs shown, at period #5, the control signal CFG[1:0] = 2'b11. According to Table 1, this signal is mapped to control signals ST0 = 1'b1, ST1 = 1'b1, ST2 = 1'b1, and SP[1:7] = 7'b1111111. Therefore, the data path formed by the multiplexer controlled by the control signals is shown with bold arrows. For example: OUT[0]|t = 5 = OUT[4]|t = 4 + OUT[0]|t = 4 = sum(OUT[7:0]|t = 3)

[0194] Therefore, in this way, AC circuit 212 accumulates the input bits passed to the output.

[0195] Figure 12 The illustration is based on one or more embodiments described herein. Figure 2 The diagram shows a simplified circuit structure of the RC circuit 216. In one embodiment, RC 216 calculates the reciprocal of input IN 1202 and drives the result to output OUT 1204 at the rising edge of the CLK signal. RC 216 uses the clock signal CLK and... Figure 2 The other circuits in the ALU 200 shown are synchronized and can be reset via the RST signal. The EN signal enables the RC circuit 216; the MODE signal indicates the digital format of the RC circuit 216; the CFG control signal contains two bits to indicate the desired number of cycles.

[0196] Figure 13 The illustration is based on one or more embodiments described herein. Figure 2 The diagram shows a simplified circuit structure of the control circuit 202. In one embodiment, the control circuit 202 can be configured... Figures 2 to 12 The various control signals described herein. Control circuit 202 utilizes the clock signal CLK and... Figure 2 The other circuits in the ALU 200 shown are synchronized and can be reset via the RST signal. The EN signal enables control circuit 202; the MODE signal indicates the digital format of RC circuit 216. Figure 14 Displays the instance control signal configuration for different operations.

[0197] Figure 15 The illustration is based on one or more embodiments described herein. Figure 2 A simplified diagram of the exponential calculation operation of ALU 200 is shown. According to... Figure 14For exponential operations, the control signals EN_EX = 1, IR_DM = 2'b00, OR_MX = 3'b000, CFG_EX[2] = 0, CFG_EX[1:0] define N, SIN_0 = 1'b0, SOUT_0 = 1'b0, SIN_1 = 1'b1, SOUT_1[1:0] = 2'b00, SIN_2 = 1'b00, SOUT_2[1:0] = 2'b00, SIN_3 = 1'b0, and SOUT_3[1:0] = 2'b00. Therefore, the input value IN 223 can pass through IR 204, TGA206a, MD 208a, and ES215 along the data path (indicated by the bold arrow).

[0198] In one embodiment, TGA 206a can pass IN to MD 208a when S OUT _0=1'b01 control Figure 4 When the multiplexer 420 is used, the MD 208a then selects to allow IN to pass. The ES215 can then calculate the exponent value of the input IN, such as regarding... Figures 6A to 6C As stated above. At MD 208b, when S IN_1 =1'b1 control Figure 4 When demultiplexer 410 is used, the exponentiation result from ES215 is selected as the output. Therefore, the exponentiation result from ES215 can eventually be passed to OR 220 and output from ALU 200.

[0199] Figure 16 The illustration is based on one or more embodiments described herein. Figure 2 A simplified diagram of a single accumulation operation of the ALU 200 is shown. According to... Figure 14 For the accumulation operation, the control signals EN_AC = 1, IR_DM = 2'b01, OR_MX = 3'b001, CFG_AC[1:0] define the AC type, SIN_0 = 1'b0, SOUT_0 = 1'b00, SIN_1 = 1'b0, SOUT_1[1:0] = 2'b01, SIN_2 = 1'b1, SOUT_2[1:0] = 2'b01, SIN_3 = 1'b0, SOUT_3[1:0]: 2'b00. Therefore, the input value IN 223 can pass through IR 204, TGA206a, MD 208d, and AC 212 along the data path (indicated by the bold arrow).

[0200] In one embodiment, TGA 206a can pass IN to MD 208b when SOUT_ = 1'b01 control Figure 4When the multiplexer 420 is in the circuit, MD 208b then selects to allow IN to pass. AC circuit 212 can then calculate the accumulation of the input over time, such as regarding... Figures 10A to 10B and Figures 11A to 11D As stated above. At MD 208d, when S IN _3=1'b1 control Figure 4 When the demultiplexer 410 is used, the accumulated result from the AC circuit 212 is selected as the output. Therefore, the accumulated result from ES215 can eventually be passed to OR202 and output from ALU 200.

[0201] Figure 17 The illustration is based on one or more embodiments described herein. Figure 2 A simplified diagram of a single reciprocal operation of the ALU 200 is shown. According to... Figure 14 For the reciprocal operation, the control signals EN_MP = 1, IR_DM = 2'b11, OR_MX = 3'b011, and CFG_RC[1:0] define N, SIN_1 = 1'b0, SOUT_1[1:0] = 2'b01, SIN_2 = 1'b1, SOUT_2[1:0] = 2'b00, SIN_3 = 1'b0, and SOUT_3[1:0] = 2'b00. Therefore, the input value IN 223 can pass through IR 204, TGA 206a, MD 208d, and RC 216 along the data path (indicated by the bold arrow).

[0202] In one embodiment, TGA 206a can pass IN to MD 208d when S OUT _3=1'b01 control Figure 4 When the multiplexer 420 is used, MD 208d then selects to allow IN to pass. The RC circuit 216 can then calculate the reciprocal of the input over time, such as regarding... Figure 12 As shown. The reciprocal result from RC circuit 216 is passed to OR 202 and output from ALU 200.

[0203] Figure 18 The illustration is based on one or more embodiments described herein. Figure 2 A simplified diagram of a single multiplication operation of the ALU 200 is shown. According to... Figure 14For the reciprocal operation, the control signals EN_RC = 1, IR_DM = 2'b10, OR_MX = 3'b010, CFG_RC[1:0] define N, SIN_1 = 1'b0, SOUT_1[1:0] = 2'b00, SIN_2 = 1'b0, SOUT_2[1:0] = 2'b00, SIN_3 = 1'b0, SOUT_3[1:0] = 2'b01. Therefore, the input value IN 223 can pass through IR 204, TGA 206a, MD 208b, MP 210, and MD 208c along the data path (indicated by the bold arrow).

[0204] In one embodiment, TGA 206a can pass IN to MD 208b when S OUT _1=1'b01 control Figure 4 When the multiplexer 420 is in the circuit, MD 208b then selects to allow IN to pass. MP circuit 210 can then multiply the input IN by the reciprocal result from RC 216, as shown in the diagram. Figures 9A to 9B The multiplication result is passed to MD 208c, when SIN_2 = 1'b1 controls... Figure 4 When the demultiplexer 410 is used, MD 208c then outputs the multiplication result. The multiplication result is finally passed to OR 202 and output from ALU 200.

[0205] Figure 19 This is a simplified diagram illustrating a combination operation according to one or more embodiments described herein, which combines... Figures 15 to 18 The multiple single operations described in the text are used to... Figure 2 The ALU 200 shown performs a softmax operation. In one embodiment, the softmax operation is as follows:

[0206]

[0207] It can be decomposed into, for example Figures 15 to 18 The single operation sequence shown.

[0208] For example, given an input value x representing the logits vector, Figure 14 The operation "ES+MP" calculates the exponent of the integer part and the exponent of the fractional part of each xi, and then multiplies these two parts to obtain x. i. The exponent. This operation is performed via data path 1901 through IR, MD0, ES, MD1, followed by multiplying the integer part by the exponent of the fractional part via MP. In this operation, it can be determined according to... Figure 14 Configure the control signal using the "ES reMP" column, which contains the operation name.

[0209] Next, Figure 14The operation of first ES then MP then AC calculates e. xi ×e –max(x) This operation is performed via data path 1903, through MP, MD2 (and then AC). In this operation, it can be determined according to... Figure 14 Configure the control signals using the "ES first, then MP, then AC" column for the operation name.

[0210] Next, Figure 14 The calculation of the denominator of softmax using the operations of MP, AC, and RC is as follows: xi ×e –max(x) The sum. This operation is performed via data path 1904 via AC, MD3 (then RC). In this operation, it can be determined according to... Figure 14 The control signals are configured using the "MP then AC then RC" column, which lists the operation names.

[0211] Next, Figure 14 The operation of AC followed by RC then OR is used to calculate the reciprocal of the denominator of the softmax function. This operation is performed via data path 1905, which is connected to RC (followed by MP). In this operation, the reciprocal of the denominator can be calculated based on... Figure 14 The control signals are configured using the "AC then RC then OR" column, which lists the operation names.

[0212] The final step, Figure 14 The operation OR followed by MP followed by OR calculates the numerator of the softmax by multiplying the reciprocal of the denominator, resulting in the final softmax. This operation is performed via data path 1902, which passes through OR, MD1, MP, MD1, and OR. Specifically, it retrieves the calculated and temporarily stored e from the operation "ES followed by MP followed by AC" from OR. xi ×e –max(x) This is then multiplied by the reciprocal of the denominator of the softmax result from the previous operations (AC, RC, OR) to produce the final softmax result. In this operation, it can be determined according to... Figure 14 The control signal is configured using the "OR then MP then OR" column, which lists the operation names.

[0213] Therefore, the ALU 200 can perform softmax operations on input values ​​and output the softmax result through the following sequence of operation commands: ES then MP, ES then MP then AC, MP then AC then RC, AC then RC then OR, OR then MP then OR. In this operation sequence, MP is used twice, and intermediate data is stored between each arithmetic unit. OR is only accessed in the last operation to reduce read and write operations to the registers, thereby achieving high hardware efficiency.

[0214] Figure 20This is a simplified diagram illustrating a combination operation according to one or more embodiments described herein, which combines... Figures 15 to 18 The multiple single operations described in the text are used to... Figure 2 The ALU 200 shown performs SiLU operations. In one embodiment, the SiLU operation is as follows:

[0215]

[0216] It can be decomposed into, for example Figures 15 to 18 The single operation sequence shown.

[0217] For example, given an input value x representing an intermediate variable vector between layers of a neural network, Figure 14 The operation "ES Neg Plus One then RC" calculates the denominator of SiLU, for example, 1+e -x This operation is performed via data path 2001 through IR, MD0, ES, and MD1, and then temporarily stored at OR. In this operation, it can be based on... Figure 14 The operation name "ES Neg + one followed by RC" column is used to configure the control signal.

[0218] Next, Figure 14 The "ES Neg plus RC" operation calculates the reciprocal of the denominator of SiLU, for example, 1 / (1+e -x This operation is performed via OR (to extract the calculated and temporarily stored denominator 1+e from the previous step). -x The data paths MD3, RC, and MP are used to execute this operation. In this operation, it can be performed according to... Figure 14 The operation name "ES Neg plus one more RC" column is used to configure the control signal.

[0219] The final step, Figure 14 The first MP followed by OR and then OR operation calculates the numerator of SiLU by multiplying the reciprocal of the denominator, resulting in the final SiLU. This operation is performed via OR (to retrieve the temporary value x), MD1, and MP (to combine the retrieved x with the calculated 1 / (1+e) from the previous step). -x The process involves multiplying the input value x by MD2, followed by OR in the data path 2002. Specifically, MP will multiply the previously calculated 1 / (1+e) from the ES Neg plus RC. -x Multiply by to produce the final SiLU result. In this operation, it can be determined according to... Figure 14 The control signal is configured using the "OR then MP then OR" column, which lists the operation names.

[0220] Therefore, the ALU 200 can perform SiLU operations on the input value and output the SiLU result through the following sequence of operation commands: EX Neg plus one then RC, EX Neg plus one then RC then MP, and OR then MP then OR.

[0221] Figure 21 This is a simplified diagram illustrating a combination operation according to one or more embodiments described herein, which combines... Figures 15 to 18 The multiple single operations described in the text are used to... Figure 2 The ALU 200 shown performs an INT8 softmax operation (dequantizing the input and then quantizing the output). In this operation, the input value x can be received in INT8 (8-bit integer) format and first converted (dequantized) to 16-bit floating-point. Then, the dequantized input value can be used to calculate the softmax result, which is then converted (quantized) back to INT8.

[0222] For example, given an input value x in INT8 format, Figure 14 The operation "dequantize ES re-MP" in the middle uses Figures 7A to 7B The inverse quantization mode described herein calculates the exponents of the integer and fractional parts of each xi. This operation is performed via data path 2101 through IR, MD0, ES, and MD1. In this operation, it is possible to determine the exponents of the integer and fractional parts of each xi. Figure 14 The operation name "Dequantize ES ReMP" column is used to configure the control signal. The output of this operation is a 16-bit floating point.

[0223] Next, Figure 14 The calculation of e is performed by first performing DES, then MP, and finally AC. xi ×e –max(x) This operation is performed via data path 2102, which passes through MP, MD2 (and then AC). In this operation, it can be determined according to... Figure 14 The control signals are configured using the "DES first, then MP, then AC" column, which specifies the operation name.

[0224] Next, Figure 14 The calculation of the denominator of softmax, i.e., e, is performed by first performing MP, then AC, and then RC. xi ×e –max(x) The sum. This operation is performed via data path 2103 via AC, MD3 (then RC). In this operation, it can be based on Figure 14 The control signals are configured using the "MP then AC then RC" column, which lists the operation names.

[0225] Next, Figure 14The operation of AC followed by RC then OR calculates the reciprocal of the denominator of the softmax function. This operation is performed via data path 2104, which is connected to RC (followed by OR). In this operation, the reciprocal of the denominator can be calculated based on... Figure 14 The control signals are configured using the "AC then RC then OR" column, which lists the operation names.

[0226] Next, Figure 14 The OR-MP-AC operation calculates the numerator of the softmax by multiplying the reciprocal of the denominator, resulting in the final softmax. This operation is performed via data path 2105, which passes through OR, MD1, MP, MD1, and then OR again. Specifically, it retrieves the calculated and temporarily stored e from the OR operation "ES-MP-AC". xi ×e –max(x) It is then multiplied by the reciprocal of the denominator of the softmax from the previous operation of AC then RC then OR to produce a 16-bit floating-point softmax result.

[0227] The final step, Figure 14 In this context, the OR operation followed by quantization and then OR again will quantize the floating-point softmax result to INT8. This operation is performed via OR, MD0, ES, MD1, and the OR data path 2106. This quantization is similar to... Figures 8A to 8B The embodiments described herein. In this operation, it can be based on... Figure 14 The control signal is configured using the "OR first, then Quant, then OR" column, which lists the operation names.

[0228] Therefore, the ALU 200 can perform INT8 softmax operations on INT8 input values ​​and output the softmax result in INT8 using the following sequence of commands: dequantization ES then MP, DES then MP then AC, MP then AC then RC, AC then RC then OR, OR then MP then OR, and finally OR then quantization then OR. This sequence of commands differs from... Figure 19 The softmax operation sequence described in [the document] has additional operations to dequantize the input and then quantize the output.

[0229] Figure 22 This is an instance logic flowchart illustrating the use of the embodiments described herein. Figures 2 to 21 The process 2200 described herein involves multiple interconnected arithmetic logic circuits transforming input data in a neural network into output data. One or more of the processes in method 2200 may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium, which, when run by one or more processors, causes one or more processors to execute one or more of the processes. In some embodiments, method 2200 corresponds to... Figures 2 to 21 The operation of ALU 200 is shown in the figure.

[0230] As shown in the figure, method 2200 includes many enumeration steps, but the state of method 2200 may include additional steps before, after, and between the enumeration steps. In some states, one or more of the enumeration steps may be omitted or performed in a different order.

[0231] In one embodiment, multiple interconnected arithmetic logic circuits are configured with different settings depending on the type of input data, such as 16-point half-precision floating-point (FP16), 16-point big-point floating-point (BF16), and 8-bit integer (INT8). Therefore, the multiple interconnected arithmetic logic circuits are compatible and thus support operations on different data formats.

[0232] At step 2202, the exponential circuit (e.g., Figure 2 The ES215 in the code can generate a first exponent of the fractional part of the input and a second exponent of the integer part of the input. For example, the configuration settings include at least one control signal used to configure the exponent circuit to operate in dequantization mode or in quantization mode, as per [reference to...]. Figures 6A to 8B As stated above.

[0233] At step 2204, the multiplier circuit (e.g., Figure 2 MP 210 in the original text can multiply the first exponent by the second exponent to obtain a first intermediate variable. For example, the configuration settings include at least one control signal used to configure the multiplier circuit to multiply the two inputs of the multiplier circuit, or to multiply one input of the multiplier circuit by another input of a different multiplier circuit, as per the context... Figures 9A to 9B As stated above.

[0234] At step 2206, the accumulator circuit (e.g., Figure 2 AC 212 in the example can sum multiple intermediate variables related to the exponents of multiple inputs. For instance, the configuration settings include at least one control signal to configure the accumulator circuit to either add the inputs and outputs of the same adder or add the two outputs of two different adders, as per the example regarding... Figures 10A to 10B As stated above.

[0235] At step 2208, the reciprocal circuit (e.g., Figure 2 The RC 216 in the circuit can generate the reciprocal of the second intermediate variable input to the reciprocal circuit.

[0236] At step 2210, the control circuit (e.g., control 202 in Figure 2) may send a configuration setting sequence (e.g., Figure 14The control settings in the configuration allow for the configuration of multiple interconnected arithmetic logic circuits within one or more cycles. Then, one or more of these interconnected arithmetic logic circuits can jointly perform at least one of softmax or sigmoid linear unit (SiLU) operations on the input data according to the configuration settings sequence.

[0237] In one implementation of step 2210, a configuration setting sequence is used to configure multiple interconnected arithmetic logic circuits to jointly perform a softmax operation. This operation is decomposed into a sequence of operations within multiple cycles, including: exponentiation, followed by a first multiplication operation, followed by an accumulation operation, followed by a reciprocal operation, and finally a second multiplication operation. Both the first and second multiplication operations are performed by multiplication circuits. Figure 19 As shown in the image.

[0238] In one implementation of step 2210, a configuration setting sequence is used to configure multiple interconnected arithmetic logic circuits to jointly perform the SiLU operation. This operation is decomposed into a sequence of operations within multiple cycles, including: a negative exponent addition operation, followed by a reciprocal operation, and finally a multiplication operation, such as... Figure 20 As shown in the image.

[0239] In one implementation of step 2210, a configuration setting sequence is configured to configure multiple interconnected arithmetic logic circuits to jointly perform a softmax operation on the INT8 input. This softmax operation is decomposed into a sequence of operations over multiple cycles, including: an exponentiation operation in dequantization mode (dequantizing the INT8 input), followed by a first multiplication operation, an accumulation operation, a reciprocal operation, a second multiplication operation, and finally a quantization operation (quantizing the INT8 output). Here, the quantization operation is performed by the exponentiation circuit in quantization mode, such as... Figure 21 As shown in the image.

[0240] Figure 23 The illustration is based on an embodiment described herein and includes... Figures 1 to 22 A simplified diagram of a computing device implementing a neural network on an AI accelerator, as described in the circuit structure. Figure 23As shown, computing device 2300 includes a processor 2310 coupled to memory 2320. The operation of computing device 2300 is controlled by processor 2310. Although only one processor 2310 is shown in computing device 2300, it should be understood that processor 2310 may represent one or more central processing units, microcontrollers, multi-core processors, microprocessors, digital signal processors, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), graphics processing units (GPUs), and / or the like in computing device 2300. Computing device 2300 may be implemented as a standalone subsystem, added to a board of a computing device, and / or implemented as a virtual machine.

[0241] Memory 2320 may be used to store software executed by computing device 2300 and / or one or more data structures used during operation of computing device 2300. Memory 2320 may contain one or more types of machine-readable media. Some common forms of machine-readable media may include floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punched cards, paper tapes, any other physical media with a perforated pattern, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media suitable for reading by a processor or computer.

[0242] Processor 2310 and / or memory 2320 can be configured in any suitable physical configuration. In some embodiments, processor 2310 and / or memory 2320 may be implemented on the same board, in the same package (e.g., system-in-package), on the same chip (e.g., system-on-a-chip), and / or the like. In some embodiments, processor 2310 and / or memory 2320 may include distributed, virtualized, and / or containerized computing resources. According to such embodiments, processor 2310 and / or memory 2320 may reside in one or more data centers and / or cloud computing facilities.

[0243] In another embodiment, processor 2310 may include multiple microprocessors and / or memory 2320 may include multiple registers and / or other memory elements, such that processor 2310 and / or memory 2320 can be configured in the form of a hardware neural network, such as... Figures 1 to 21 Further details are provided below.

[0244] In some instances, memory 2320 may contain non-transitory, tangible, machine-readable media containing executable code that, when run by one or more processors (e.g., processor 2310), causes one or more processors to perform methods described further in detail herein. For example, as shown in the figures, memory 2320 contains instructions for operating neural network 2331.

[0245] Memory 2320 may be further coupled to AI accelerator 2330, which may include an ALU, such as Figure 2 The ALU 200 described in the document, and Figures 3 to 21 The various circuits described in the document.

[0246] Data interface 2315 may include a communication interface, a user interface (such as a voice input interface, a graphical user interface, and / or the like). For example, computing device 2300 may receive input 2340 (such as a training dataset) from a networked database via the communication interface. Alternatively, computing device 2300 may receive input 2340 from a user, such as an input image, via the user interface and generate output 2350 (such as... Figure 1 (130 in the middle).

[0247] Some examples of computing devices, such as computing device 2300, may include non-transitory, tangible, machine-readable media containing executable code that, when run by one or more processors (e.g., processor 2310), enables one or more processors to perform the process of a method. Some common forms of machine-readable media that may contain the process of a method include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punched cards, paper tapes, any other physical media with a perforated pattern, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media suitable for reading by a processor or computer.

[0248] The computing device 2300 may be included in a system for running one or more neural networks. The system includes a segmenter circuit for dividing input data values ​​into integer and fractional parts, a compensation circuit for generating a compensated fractional part based on at least a first output from a first comparator circuit comparing the fractional and integer parts, a scheduler circuit for dynamically determining the number of terms for approximating the fractional part's exponent, and a Taylor expansion calculation circuit for calculating the sum of the Taylor expansion terms for the compensated fractional part.

[0249] In one exemplary embodiment, this disclosure relates to a computational device for transforming input data in a neural network into output data. The computational device includes a plurality of interconnected arithmetic logic circuits and a control circuit. The plurality of interconnected arithmetic logic circuits include an exponent circuit that generates a first exponent for the fractional part of the input and a second exponent for the integer part; a multiplier circuit that multiplies the first exponent and the second exponent to obtain a first intermediate variable; an accumulator circuit that sums a plurality of intermediate variables related to the exponents of the plurality of inputs; and a reciprocal circuit that generates the reciprocal of a second intermediate variable input to a reciprocal circuit. The control circuit sends a configuration setting sequence to configure the plurality of interconnected arithmetic logic circuits within one or more cycles, such that one or more of the plurality of interconnected arithmetic logic circuits jointly perform at least one of a softmax operation or a sigmoidal linear unit (SiLU) operation on the input data according to the configuration setting sequence.

[0250] In some embodiments, the computing device further includes one or more multiplexer-demultiplexer (mux-demux) units interleaved with a plurality of interconnected arithmetic logic circuits. This configuration includes one or more selection signals for the one or more mux-demux units to form one or more data paths among the interconnected arithmetic logic circuits. These one or more data paths correspond to sequences of arithmetic operations performed by one or more of the interconnected arithmetic logic circuits.

[0251] In some embodiments, multiple interconnected arithmetic logic circuits are configured with different configuration settings according to different types of input data. The different types of input data include 16-point half-precision floating-point (FP16), 16-point big-point floating-point (BF16), and 8-bit integer (INT8).

[0252] In some embodiments, each of a plurality of interconnected arithmetic logic circuits performs operations on a data format that accommodates the maximum bit width of the sign, exponent, and mantissa in FP16 and BF16.

[0253] In some embodiments, the configuration setting includes at least one control signal to configure the exponential circuit to operate in dequantization mode or in quantization mode.

[0254] In some embodiments, the configuration setting includes at least one control signal to configure the multiplier circuit to multiply two inputs of the multiplier circuit or to multiply one input of the multiplier circuit with another input of a different multiplier circuit.

[0255] In some embodiments, the configuration setting includes at least one control signal to configure the accumulator circuit to sum the input and output of the same adder, or to sum the two outputs of two different adders.

[0256] In some embodiments, a configuration setting sequence is configured with multiple interconnected arithmetic logic circuits to jointly perform a softmax operation, which is decomposed into a sequence of operations within multiple cycles, including: an exponentiation operation, followed by a first multiplication operation, followed by an accumulation operation, and finally a reciprocal operation, followed by a second multiplication operation, both of which are performed by multiplication circuits.

[0257] In some embodiments, a configuration setting sequence is configured to configure multiple interconnected arithmetic logic circuits to jointly perform SiLU operations, which are decomposed into a sequence of operations within multiple cycles, including: negative exponent addition, followed by reciprocal operations, and finally multiplication operations.

[0258] In some embodiments, a configuration setting sequence is configured with multiple interconnected arithmetic logic circuits to jointly perform a softmax operation on the INT8 input. The softmax operation is decomposed into a sequence of operations within multiple cycles, including: an exponentiation operation in dequantization mode for dequantizing the INT8 input, followed by a first multiplication operation, followed by an accumulation operation, followed by a reciprocal operation, followed by a second multiplication operation, and then a quantization operation for quantizing the INT output, the quantization operation being performed by the exponentiation circuit in quantization mode.

[0259] In another exemplary embodiment, one embodiment of this disclosure relates to an operational method for transforming input data in a neural network into output data using multiple interconnected arithmetic logic circuits. This operational method includes generating a first exponent of the fractional part of the input and a second exponent of the integer part of the input via an exponentiation circuit; multiplying the first and second exponents by a multiplier circuit to obtain a first intermediate variable; summing multiple intermediate variables related to the exponents of the multiple inputs via an accumulator circuit; generating the reciprocal of a second intermediate variable input to the reciprocal circuit via a reciprocal circuit; sending a configuration setting sequence via a control circuit to configure multiple interconnected arithmetic logic circuits within one or more cycles; and jointly performing at least one of a softmax operation or a sigmoid linear unit (SiLU) operation on the input data using one or more of the multiple interconnected arithmetic logic circuits.

[0260] In some embodiments, a plurality of interconnected arithmetic logic circuits are interleaved with one or more multiplexer-demultiplexer (mux-demux) units, and the operation method further includes forming one or more data paths among the interconnected arithmetic logic circuits via one or more mux-demux units according to one or more selection signals from a configuration setting. The one or more data paths correspond to a sequence of arithmetic operations performed by the one or more interconnected arithmetic logic circuits.

[0261] In some embodiments, multiple interconnected arithmetic logic circuits are configured with different configuration settings according to different types of input data. The different types of input data include 16-point half-precision floating-point (FP16), 16-point big-point floating-point (BF16), and 8-bit integer (INT8).

[0262] In some embodiments, each of a plurality of interconnected arithmetic logic circuits performs operations on a data format that accommodates the maximum bit width of the sign, exponent, and mantissa in FP16 and BF16.

[0263] In some embodiments, the calculation method further includes configuring the exponent circuit to operate in dequantization mode or in quantization mode according to at least one control signal included in the configuration settings.

[0264] In some embodiments, the operation method further includes configuring the multiplier circuit to multiply two inputs of the multiplier circuit or to multiply one input of the multiplier circuit with another input of a different multiplier circuit, based on at least one control signal included in the configuration settings.

[0265] In some embodiments, the operation method further includes configuring a plurality of interconnected arithmetic logic circuits according to a configuration setting sequence to jointly perform a softmax operation, which is decomposed into a sequence of operations within a plurality of cycles, including: an exponentiation operation, followed by a first multiplication operation, followed by an accumulation operation, followed by a reciprocal operation, followed by a second multiplication operation, both the first multiplication operation and the second multiplication operation being performed by multiplication circuits.

[0266] In some embodiments, the operation method further includes configuring a plurality of interconnected arithmetic logic circuits according to a configuration setting sequence to jointly perform a softmax operation on the INT8 input. The softmax operation is decomposed into a sequence of operations within a plurality of cycles, including: an exponentiation operation in a dequantization mode of dequantizing the INT8 input, followed by a first multiplication operation, followed by an accumulation operation, followed by a reciprocal operation, followed by a second multiplication operation, followed by a quantization operation of quantizing the INT8 output, the quantization operation being performed by the exponentiation circuit in quantization mode.

[0267] In another exemplary embodiment, one embodiment of this disclosure relates to an operational method for constructing a circuit that transforms input data in a neural network into output data. This operational method includes placing a plurality of interconnected arithmetic logic circuits on a circuit. The plurality of interconnected arithmetic logic circuits include an exponentiation circuit that generates a first exponent for the fractional part of the input and a second exponent for the integer part; a multiplier circuit that multiplies the first exponent and the second exponent to obtain a first intermediate variable; an accumulator circuit that sums a plurality of intermediate variables related to the exponents of the plurality of inputs; and a reciprocal circuit that generates the reciprocal of a second intermediate variable input to a reciprocal circuit. A control circuit sends a configuration setting sequence to configure the plurality of interconnected arithmetic logic circuits within one or more cycles, such that one or more of the plurality of interconnected arithmetic logic circuits jointly perform at least one of a softmax operation or a sigmoid linear unit (SiLU) operation on the input data according to the configuration setting sequence.

[0268] In some embodiments, multiple interconnected arithmetic logic circuits are configured with different configuration settings according to different types of input data. The different types of input data include 16-point half-precision floating-point (FP16), 16-point big-point floating-point (BF16), and 8-bit integer (INT8).

[0269] The foregoing outlines the features of several embodiments, enabling those skilled in the art to better understand the nature of one embodiment of this disclosure. Those skilled in the art will understand that one embodiment of this disclosure can be easily used as a basis for designing or modifying other processes and structures for implementing the same purposes and / or achieving the same advantages of the embodiments introduced herein. Those skilled in the art should also recognize that such equivalent constructions do not depart from the spirit and scope of one embodiment of this disclosure, and that such equivalent constructions can be modified, substituted, and replaced herein without departing from the spirit and scope of one embodiment of this disclosure.

Claims

1. A computing device for transforming input data in a neural network into output data, characterized in that, It includes: Multiple interconnected arithmetic logic circuits, including: An exponential circuit for generating a first exponent of a fractional part of an input and a second exponent of an integer part of the input; A multiplier circuit for multiplying the first exponent by the second exponent to obtain a first intermediate variable; An accumulator circuit for summing multiple intermediate variables related to multiple exponents of multiple inputs; and A reciprocal circuit for generating a reciprocal of a second intermediate variable input to the reciprocal circuit; as well as A control circuit is configured to send a sequence of multiple configuration settings to configure the multiple interconnected arithmetic logic circuits within one or more cycles, such that one or more of the multiple interconnected arithmetic logic circuits jointly perform at least one of a normalized exponential operation or a sigmoid linear unit operation on the input data according to the sequence of multiple configuration settings.

2. The computing device as described in claim 1, characterized in that, Further includes: One or more multiplexer-demultiplexer units are interleaved with the plurality of interconnected arithmetic logic circuits. The plurality of configuration settings include one or more selection signals for the one or more multiplexer-demultiplexer units to form one or more data paths between the plurality of interconnected arithmetic logic circuits. The one or more data paths correspond to a sequence of arithmetic operations performed by one or more of the multiple interconnected arithmetic logic circuits.

3. The computing device as described in claim 1, characterized in that, The multiple interconnected arithmetic logic circuits are configured according to the multiple different types of the input data and through multiple different configuration settings. The various types of the input data include 16-point half-precision floating-point, 16-point half-precision floating-point, and 8-bit integer.

4. The computing device as described in claim 3, characterized in that, Each of the plurality of interconnected arithmetic logic circuits operates on a data format with a maximum bit width that accommodates the sign, exponent, and mantissa of half-point floating-point 16-bit and full-point floating-point 16-bit data.

5. The computing device as claimed in claim 1, characterized in that, The plurality of configuration settings include at least one control signal to configure the exponential circuit to operate in an inverse quantization mode or in a quantization mode.

6. The computing device as claimed in claim 1, characterized in that, The plurality of configuration settings include at least one control signal to configure the multiplier circuit to multiply two inputs of the multiplier circuit or to multiply one input of the multiplier circuit with another input of a different multiplier circuit.

7. The computing device as claimed in claim 1, characterized in that, The plurality of configuration settings include at least one control signal to configure the accumulator circuit to sum an input and an output of the same adder, or to sum two outputs of two different adders.

8. The computing device as claimed in claim 1, characterized in that, Multiple interconnected arithmetic logic circuits configured in this sequence are used to jointly execute the normalized exponentiation operation, which is decomposed into a sequence of operations over multiple cycles. This sequence includes: An exponential operation is performed, followed by a first multiplication operation, then an accumulation operation, then a reciprocal operation, and finally a second multiplication operation, wherein both the first and second multiplication operations are performed by a multiplication circuit.

9. A computational method for transforming input data in a neural network into output data using multiple interconnected arithmetic logic circuits, characterized in that, This calculation method includes the following steps: An exponent circuit generates a first exponent of the fractional part of an input and a second exponent of the integer part of the input. The first exponent is multiplied by the second exponent using a multiplier circuit to obtain a first intermediate variable; An accumulator circuit is used to sum multiple intermediate variables that are related to multiple exponents of multiple inputs. A reciprocal circuit is used to generate a reciprocal of a second intermediate variable that is input to the reciprocal circuit. A control circuit sends a sequence of multiple configuration settings to configure the multiple interconnected arithmetic logic circuits within one or more cycles. as well as Through one or more of the plurality of interconnected arithmetic logic circuits, at least one of a normalized exponential operation or a sigmoid linear unit operation is jointly performed on the input data according to the sequence set by the plurality of configurations.

10. A computational method for constructing a circuit that transforms input data in a neural network into output data, characterized in that, This calculation method includes the following steps: Multiple interconnected arithmetic logic circuits are placed on the circuit, wherein the multiple interconnected arithmetic logic circuits include: An exponential circuit that generates a first exponent of a fractional portion of an input and a second exponent of an integer portion of the input; A multiplier circuit that multiplies the first exponent by the second exponent to obtain a first intermediate variable; An accumulator circuit that sums multiple intermediate variables related to multiple exponents of multiple inputs; and A reciprocal circuit that generates a reciprocal of a second intermediate variable input to the reciprocal circuit; as well as A control circuit that sends a sequence of multiple configuration settings in one or more cycles to configure the multiple interconnected arithmetic logic circuits, thereby jointly performing at least one of a normalized exponential operation or a sigmoid unit operation on the input data according to the sequence of multiple configuration settings.

Images