Operational circuit, accelerator, chip and board card

Abstract

The invention relates to an operational circuit, an accelerator, a chip and a board card. The operational circuitry may be implemented as a computing device included in a combined processing device, which may include one or more data processing devices. The combined processing device can also comprise an interface device and other processing devices. And the computing device interacts with other processing devices to jointly complete the computing operation specified by the user. The combined processing device can further comprise a storage device, and the storage device is connected with the equipment and the other processing devices and used for storing data of the equipment and the other processing devices. Through the scheme of the invention, the repeated multiplication operation in the matrix multiplication operation can be simplified into unary operation accumulation based on the counter, so that the dynamic power consumption of hardware can be saved.

Description

Technical Field

[0001] This disclosure generally relates to the field of circuits. More specifically, this disclosure relates to a computing circuit, an accelerator, a chip, and a circuit board. Background Technology

[0002] Deep neural networks (DNNs) have become crucial in various fields, including computer vision, natural language processing (NLP), and speech recognition. Current research shows a strong correlation between model size and performance. This trend is evident in the evolution from early recurrent neural networks (15 million parameters) to larger models like the Transformer (65 million parameters) and BERT (up to 340 million parameters), as well as modern large-scale models such as LLaMA2 (up to 70 billion parameters) and GPT-4. The number of parameters in DNNs has grown from millions to hundreds of billions.

[0003] However, the scale of DNNs also places enormous demands on the computational resources required for inference on various devices, prompting the rise of quantization techniques. Large models with billions of parameters exceed the memory capacity of even the most powerful consumer hardware, making their low-quantized versions increasingly prevalent. In existing technologies, many processors supporting matrix multiplication (such as systolic arrays based on multiply-accumulate units) involve a large number of repetitive multiplication operations and high-bit-width accumulation operations during matrix multiplication. Furthermore, as the data bit width decreases, the efficiency gains also decrease. When the data precision drops to a certain low bit width (e.g., INT4), the actual computational efficiency is no longer significant. Summary of the Invention

[0004] In view of the technical problems mentioned in the background section above, this disclosure proposes an arithmetic circuit. Using the arithmetic circuit of this disclosure, repeated multiplication operations in matrix multiplication can be converted into accumulation operations, effectively reducing hardware power consumption.

[0005] In a first aspect, this disclosure provides a computation circuit comprising: a data storage circuit storing a pre-computation lookup table containing pre-computation results relating to all possible values ​​of the element-wise product of two matrix vectors in a matrix multiplication calculation; a processing unit array comprising a plurality of processing units electrically connected to adjacent processing units, wherein each processing unit contains a plurality of counters, the operation result of each counter being configured to index a pre-computation result in the pre-computation lookup table; and a data conversion circuit electrically connected to each processing unit in the processing unit array and the data storage circuit to calculate a matrix multiplication result between matrix vectors based on all operation results of the counters in each processing unit and the corresponding pre-computation result.

[0006] In a second aspect, this disclosure provides an accelerator whose main computing unit employs the computing circuitry described in the first aspect.

[0007] In a third aspect, this disclosure provides a chip including the arithmetic circuitry described in the first aspect.

[0008] In a fourth aspect, this disclosure provides a board including the chip described in the third aspect.

[0009] The solutions provided in the above-mentioned aspects of this disclosure can effectively reduce the computational complexity and hardware power consumption of matrix multiplication. Specifically, a pre-computation lookup table can be used to store the pre-computation results related to all possible values ​​of the product of elements involved in the matrix multiplication calculation, and the counters and data conversion circuits contained in the processing units in the processing unit array can be used to simplify the repeated multiplication operations in the matrix multiplication operation into unary operation accumulation based on the counter, thereby saving dynamic power consumption of the hardware.

[0010] Furthermore, in some embodiments, the formula is used. Transform all possible values ​​of the product of elements in two matrix vectors for matrix multiplication to reduce the number of counters in the processing unit, thereby saving repetitive arithmetic and reducing accumulation intensity, effectively reducing static power consumption overhead. Attached Figure Description

[0011] The above and other objects, features, and advantages of exemplary embodiments of the present disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the present disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0012] Figure 1 This is a structural diagram of a board according to an embodiment of the present disclosure;

[0013] Figure 2This is a structural diagram illustrating a combined processing apparatus according to an embodiment of the present disclosure;

[0014] Figure 3 This is a schematic diagram showing the internal structure of a computing device according to an embodiment of the present disclosure;

[0015] Figure 4 This is a schematic diagram illustrating the internal structure of a processor core according to an embodiment of the present disclosure;

[0016] Figure 5 This is a schematic diagram illustrating the data writing process between processor cores of different clusters according to embodiments of the present disclosure;

[0017] Figure 6 This is a schematic diagram illustrating the functional implementation and power consumption evaluation of the processing unit in a pulsating array of conventional multiply-accumulate units;

[0018] Figure 7 This schematic diagram illustrates the structure of an operational circuit according to an embodiment of the present disclosure;

[0019] Figure 8 The diagram illustrates the operation process of an arithmetic circuit according to an embodiment of the present disclosure.

[0020] Figure 9 A schematic diagram illustrating a power consumption comparison between an arithmetic circuit according to an embodiment of the present disclosure and a conventional multiply-accumulate unit;

[0021] Figure 10 The diagram illustrates the operation process of an arithmetic circuit according to another embodiment of the present disclosure.

[0022] Figure 11 This schematically illustrates the architecture of a data conversion circuit according to an embodiment of the present disclosure; and

[0023] Figure 12 A schematic diagram of the structure of an accelerator according to an embodiment of the present disclosure is shown. Detailed Implementation

[0024] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0025] It should be understood that the terms "first," "second," "third," and "fourth," etc., in the claims, specification, and drawings of this disclosure are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0026] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0027] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0028] The device side in this embodiment may be an artificial intelligence chip or board, etc. Figure 1 A schematic diagram of the structure of a board 100 according to an embodiment of this disclosure is shown. Figure 1 As shown, the board 100 includes a chip (or "processing chip") 101, which is a system-on-a-chip (SoC) that integrates one or more combined processing devices. These combined processing devices are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. The board 100 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and substantial computing power.

[0029] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.

[0030] The board 100 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 107 and the chip 101 via a bus. The controller 106 in the board 100 is configured to regulate the state of the chip 101. For this purpose, in one application scenario, the controller 106 may include a microcontroller (MCU).

[0031] Figure 2 This is a structural diagram illustrating the combined processing device 200 in chip 101 of this embodiment. (As shown...) Figure 2 As shown, the combined processing device 200 includes a computing device 201, an interface device 202, a processing device 203, and a DRAM 204.

[0032] The computing device 201 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0033] Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them to on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from storage device of computing device 201 and transmit it to processing device 203.

[0034] Processing device 203, as a general-purpose processing device, performs basic control including but not limited to data transfer, and starting and / or stopping computing device 201. Depending on the implementation, processing device 203 may be one or more types of processors, including but not limited to digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, for the purposes of this disclosure only, computing device 201 can be considered to have a single-core structure or a homogeneous multi-core structure. However, when computing device 201 and processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0035] DRAM 204 is used to store data to be processed. It is DDR memory, typically 16G or larger in size, and is used to store data in computing device 201 and / or processing device 203.

[0036] Figure 3 A schematic diagram of the internal structure of computing device 201 is shown. Computing device 201 is used to process input data such as computer vision, speech, natural language processing, and data mining. The computing device 201 in the diagram adopts a multi-core hierarchical architecture design. As a system-on-a-chip, computing device 201 includes multiple clusters, each of which includes multiple processor cores, which can be used to execute the tasks disclosed herein. In other words, computing device 201 is constructed in a hierarchical structure of system-on-a-chip, clusters, and processor cores.

[0037] From the perspective of system-on-a-chip hierarchy, such as Figure 3 As shown, the computing device 201 includes an external storage controller 301, a peripheral communication module 302, an on-chip interconnect module 303, a synchronization module 304, and multiple clusters 305.

[0038] There can be multiple external storage controllers 301; two are shown as an example in the figure. These controllers are used to access external storage devices, such as those issued by the processor core, in response to access requests from the processor core. Figure 2The DRAM 204 in the chip allows the computing device 201 to read data from external sources or write data to external sources. The peripheral communication module 302 receives control signals from the processing device 203 via the interface device 202, initiating the computing device 201 to execute tasks. The on-chip interconnect module 303 connects the external storage controller 301, the peripheral communication module 302, and multiple clusters 305 to transmit data and control signals between the modules. The synchronization module 304 is a Global Barrier Controller (GBC) used to coordinate the working progress of each cluster and ensure information synchronization. The multiple clusters 305 are the computing core of the computing device 201. Four are shown exemplarily in the figure; however, with hardware development, the computing device 201 disclosed herein may also include 8, 16, 64, or even more clusters 305.

[0039] From the perspective of cluster hierarchy, such as Figure 3 As shown, each cluster 305 includes multiple processor cores (IPU cores) 306 and one memory core (MEM core) 307.

[0040] Four processor cores 306 are shown in the figure as an example; this disclosure does not limit the number of processor cores 306. Its internal architecture is as follows: Figure 4 As shown, each processor core 306 includes three main modules: a control module 31, an arithmetic module 32, and a storage module 33.

[0041] The control module 31 coordinates and controls the operation of the computation module 32 and the storage module 33 to complete the deep learning task. It includes an instruction fetch unit (IFU) 411 and an instruction decode unit (IDU) 412. The instruction fetch unit 411 fetches instructions from the processing device 203, and the instruction decode unit 412 decodes the fetched instructions and sends the decoding result as control information to the computation module 32 and the storage module 33.

[0042] The computation module 32 includes a vector operation unit 421 and a matrix operation unit 422. The vector operation unit 421 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit 422 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution.

[0043] Storage module 33 is used to store or move related data, including Neuron RAM (NRAM) 431, Weight RAM (WRAM) 432, Input / Output Direct Memory Access (IODMA) 433, and Move Direct Memory Access (MVDMA) 434. NRAM 431 is used to store input, output data, and intermediate results for the processor core 306 to calculate; WRAM 432 is used to store the weights of the deep learning network; IODMA 433 controls the memory access of NRAM 431 / WRAM 432 and DRAM 204 through broadcast bus 309; MVDMA 434 controls the memory access of NRAM 431 / WRAM 432 and SRAM 308.

[0044] Back Figure 3 The storage core 307 is primarily used for storage and communication, namely storing shared data or intermediate results among processor cores 306, and performing communication between cluster 305 and DRAM 204, communication between clusters 305, and communication between processor cores 306. In other embodiments, the storage core 307 has scalar operation capabilities and is used to perform scalar operations.

[0045] Storage core 307 includes a shared memory unit (SRAM) 308, a broadcast bus 309, a Cluster Direct Memory Access (CDMA) module 310, and a Global Direct Memory Access (GDMA) module 311. SRAM 308 acts as a high-performance data relay station. Data multiplexed between different processor cores 306 within the same cluster 305 does not need to be obtained from DRAM 204 by each processor core 306 individually. Instead, it is relayed between processor cores 306 via SRAM 308. Storage core 307 only needs to quickly distribute the multiplexed data from SRAM 308 to multiple processor cores 306, thereby improving inter-core communication efficiency and significantly reducing on-chip and off-chip I / O access.

[0046] Broadcast bus 309, CDMA 310, and GDMA 311 are used to perform communication between processor cores 306, communication between clusters 305, and data transfer between cluster 305 and DRAM 204, respectively. These will be explained below.

[0047] The broadcast bus 309 is used to complete high-speed communication between the processor cores 306 within the cluster 305. In this embodiment, the broadcast bus 309 supports inter-core communication methods including unicast, multicast, and broadcast. Unicast refers to point-to-point (i.e., data transmission from one processor core to another) data transmission. Multicast is a communication method that transmits a piece of data from SRAM 308 to several specific processor cores 306. Broadcast is a communication method that transmits a piece of data from SRAM 308 to all processor cores 306, and is a special case of multicast.

[0048] CDMA 310 is used to control SRAM 308 access between different clusters 305 within the same computing device 201. Figure 5 This diagram illustrates the operation of CDMA310 when one processor core attempts to write data to another processor core in a different cluster. In this application scenario, the same computing device comprises multiple clusters. For simplicity, only clusters 0 and 1 are shown in the diagram. Both clusters 0 and 1 contain multiple processor cores; similarly, for ease of explanation, only processor core 0 is shown in cluster 0, and only processor core 1 is shown in cluster 1. Processor core 0 intends to write data to processor core 1.

[0049] First, processor core 0 sends a unicast write request to write data into its local SRAM 0. CDMA 0 acts as the master and CDMA 1 acts as the slave. The master pushes the write request to the slave, that is, the master sends the write address AW and the write data W to transmit the data to SRAM 1 of cluster 1. Then, the slave sends a write response B as a response. Finally, processor core 1 of cluster 1 sends a unicast read request to read the data from SRAM 1.

[0050] Back Figure 3The GDMA 311, in conjunction with the external memory controller 301, controls memory access from the SRAM 308 of the cluster 305 to the DRAM 204, or reads data from the DRAM 204 into the SRAM 308. As previously described, communication between the DRAM 204 and the NRAM 431 or WRAM 432 can be achieved through two channels. The first channel is a direct connection between the DRAM 204 and the NRAM 431 or WRAM 432 via the IODAM 433; the second channel involves first transmitting data between the DRAM 804 and SRAM 308 via the GDMA 311, and then transmitting data between the SRAM 308 and the NRAM 431 or WRAM 432 via the MVDMA 434. Although the second channel appears to require more components and has a longer data flow, in some embodiments, the bandwidth of the second channel is significantly greater than that of the first channel. Therefore, communication between the DRAM 204 and the NRAM 431 or WRAM 432 may be more efficient via the second channel. The embodiments of this disclosure can select the data transmission channel according to their own hardware conditions.

[0051] In other embodiments, the functions of GDMA 311 and IODMA 433 can be integrated into the same component. For ease of description, this disclosure treats GDMA 311 and IODMA 433 as different components. For those skilled in the art, any component whose implemented functions and achieved technical effects are similar to this disclosure is within the scope of protection of this disclosure. Furthermore, the functions of GDMA 311, IODMA 433, CDMA 310, and MVDMA 434 can also be implemented by the same component. Similarly, any component whose implemented functions and achieved technical effects are similar to this disclosure is within the scope of protection of this disclosure.

[0052] The above combination Figures 1-5 The hardware architecture and internal structure of this disclosure are described in detail. It is understood that the above description is merely exemplary and not restrictive. Depending on different application scenarios and hardware specifications, those skilled in the art may also make changes to the board (or artificial intelligence device) and its internal structure disclosed herein, and such changes shall still fall within the protection scope of this disclosure.

[0053] As mentioned earlier, the scale of DNNs places enormous demands on the computational resources required for inference across various devices, prompting the rise of quantization techniques. Large models with billions of parameters exceed the memory capacity of even the most powerful consumer hardware, making their underquantized versions increasingly prevalent. For example, for devices with memory budgets of 4GB, 8GB, 24GB, or 40GB, underquantized versions (e.g., 4-bit quantized versions) are best suited for running within these constraints. To meet these requirements, popular inference servers support low-bit-width (e.g., 4-bit) quantization and emphasize the efficiency of low-bit matrix multiplication algorithms and hardware.

[0054] In terms of hardware architecture, there are several typical matrix operation unit processors. Among them, the systolic array (SA) based on multiply-accumulate (MAC) units is a mature microarchitecture paradigm dedicated to matrix multiplication. It achieves efficient parallel computing by having data flow through a series of closely packed process elements (PEs), performing local operations within each processing unit, and transferring data between processing units. These PEs are organized into a network structure through which data flows, enabling efficient parallel computing.

[0055] Quantization and dedicated low-bit matrix processing units can reduce the operating overhead of DNNs. However, the efficiency gains of traditional MAC-based SA decrease as the data bit width decreases. Figure 6 A simplified implementation of MAC-based SA PE is shown, and its power consumption at different bit widths is evaluated. Figure 6 As shown, from INT16 to INT8 and then to INT4, the Boolean complexity of the PE function is expected to decrease by 3.3 to 3.6 times, but the power consumption only decreases by 2.5 times. This means that for every halved quantization bit width, MAC-based SA will suffer a 27.5% loss in performance. When the precision drops to INT4, the actual computational efficiency is no longer significant. There are two reasons for this problem, as follows:

[0056] First, the quantization effect becomes significant and non-negligible at INT4. INT4 can only represent 16 unique values, so there are at most 16 × 16 = 256 unique results in INT4 multiplication. However, in DNN tasks, the matrix dimension processed by SA is usually much greater than 256, which means that most arithmetic operations are repeated, leading to energy waste.

[0057] Secondly, the accumulator's power consumption is the dominant factor. Quantization reduces the width of the multiplier, but the accumulator's width is limited by the maximum supported matrix dimension. The accumulator must guarantee no overflow on the largest matrix. Assuming multiplication of an n-order square matrix, the accumulator must have log₂n bits more than the product. For INT4, the accumulator requires 24 bits or even more. Compared to a multiplier (dual 4-bit inputs, 8-bit output), the accumulator (24-bit storage, 8-bit input) consumes 3.07 times more power. Figure 6 As shown, as the bit width decreases, the power consumption of multiplication drops rapidly, while the power consumption of addition and registers decreases slowly and becomes dominant, which leads to a gradual reduction in the returns from quantization.

[0058] To overcome the shortcomings of the prior art discussed herein and in the background section, the inventors, while researching quantized neural network accelerators, discovered that the power consumption bottleneck of low-bit operation units in existing technologies stems from repetitive multiplication operations and high-width accumulators. Since low-bit (e.g., INT4) operations have very low overhead, existing solutions struggle to achieve net gains in INT4 scenarios. To gain benefits in low-bit operations, it is necessary to reduce repetitive arithmetic and accumulation intensity, and the implementation algorithm must be simple and clear to avoid excessive architectural overhead during implementation. In response, this disclosure proposes an operation circuit that, through the cooperation of the data storage circuit, processing unit array, and data conversion circuit, simplifies the repetitive multiplication operation in matrix multiplication into a counter-based unary accumulation operation, thereby saving dynamic power consumption.

[0059] The specific embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0060] Figure 7 The schematic diagram illustrates the structure of an operational circuit 700 according to an embodiment of the present disclosure.

[0061] like Figure 7 As shown, the arithmetic circuit 700 may include a data storage circuit 701, a processing unit array 702, and a data conversion circuit 703. The data storage circuit 701 stores a pre-computation lookup table, which stores pre-computation results related to all possible values ​​of the element-wise product of two matrix vectors in a matrix multiplication calculation. The number of possible values ​​of the element-wise product is related to the bit width of the elements. For example, if both matrix vectors in the matrix multiplication calculation have elements of INT4, and INT4 can represent 16 unique values, then there are 16 × 16 = 256 possible values ​​for the element-wise product. In some embodiments, the pre-computation results stored in the pre-computation lookup table may be the total number of possible values ​​of these element-wise products (e.g., the 256 results mentioned above) themselves, or they may be intermediate conversion values ​​corresponding to the total number of possible values ​​of the element-wise product.

[0062] The processing unit array 702 may include multiple processing units (such as...) Figure 7 The diagram shows a processing unit (PE) array where adjacent processing units (PEs) are electrically connected. Each processing unit (PE) contains multiple counters, and the operation result of each counter is configured to index a pre-calculated result in the aforementioned pre-calculated lookup table. It should be noted that the scheme disclosed herein does not limit the number of processing units in the processing unit array 702; the number can be set and adjusted according to specific application scenarios. Furthermore, the counters within each processing unit (PE) can be commercially available low-power counters.

[0063] The data conversion circuit 703 can be electrically connected to each processing unit in the processing unit array 702 and the data storage circuit 701 to calculate the matrix multiplication result between matrix vectors based on all operation results of the counters in each processing unit and the corresponding pre-calculated results. It should be noted that... Figure 7 The specific circuit structure of the arithmetic circuit shown is merely illustrative. In practical applications, these circuits can be integrated or combined. For example, the data storage circuit 701 and the data conversion circuit 703 can be integrated into one circuit, and the combined circuit has the functions of storing pre-calculated lookup tables and corresponding data conversion.

[0064] The output value of matrix multiplication is obtained by calculating the inner product. To calculate the output value... All possible values ​​of the product of elements in two matrix vectors w and x can be pre-computed, and a vector m (i.e., a pre-computed lookup table) can be constructed based on these values. This pre-computed lookup table stores the pre-computed results related to all possible values ​​of the product of elements in matrix vectors w and x. y can then be calculated by looking up the pre-computed product in m, instead of performing multiplication on each element. In practical applications, one-hot encoding can be used to implement the lookup of pre-computed results. Where p i It is encoding (x) i ,w i The one-hot vector of ), i.e.

[0065]

[0066] Since vector multiplication satisfies the distributive law with respect to addition, we can first calculate ∑p i Only in the last step will ∑p i With m T Multiplying once, that is The main part of the calculation is ∑p iThis involves summing one-hot vectors. Therefore, binary addition is simplified to an increment operation on one of the pre-calculated results. The increment operation, also known as a unary inheritance function, is a primitive arithmetic operation defined by Peano's axioms, simpler to implement than addition, and can be implemented using a counter. Thus, by storing the pre-calculated results related to all possible values ​​of the product of elements involved in the matrix multiplication in a pre-calculation lookup table, and utilizing the counters and data conversion circuits contained in the processing units within the processing unit array, the repetitive multiplication operations in matrix multiplication are simplified to counter-based unary accumulation, thereby saving dynamic power consumption in the hardware.

[0067] As mentioned above, the pre-computation result in the pre-computation lookup table can include all possible values ​​of the prime product in the matrix vectors themselves. In some embodiments, the pre-computation result in the pre-computation lookup table is all possible values ​​of the element-wise product of the two matrix vectors in the matrix multiplication calculation, and the counter in each processing unit corresponds to the one-hot encoding of the pre-computation result. In this implementation scenario, the arithmetic circuit is used to perform the matrix multiplication calculation as follows: each processing unit in the processing unit array receives elements from the two input vectors to be multiplied, and in each calculation cycle, each processing unit activates the corresponding counter according to the received element to perform an increment operation; and the data conversion circuit is used to multiply all operation results of the counters in each processing unit with the corresponding pre-computation result to obtain the matrix multiplication result of the two input vectors.

[0068] In this embodiment, the aforementioned operational circuit specifically uses the Primitive Matrix Multiplication (PMM) algorithm to reduce the multiplication-addition operation in traditional matrix multiplication to a unary operation (i.e., incremental operation). There are two main differences between PMM and traditional algorithms: First, PMM avoids redundant multiplication operations by pre-compiling a lookup table. The PMM algorithm will be explained in detail using an example where both matrix vectors (w and x) in the matrix multiplication calculation have elements of type INT4.

[0069] It should be noted that the output value of matrix multiplication is obtained by calculating the inner product. All possible values ​​of the product of w and x can be pre-computed and written as a vector m (i.e., the aforementioned pre-computed lookup table). For a 4-bit integer represented in two's complement, m is a 256-dimensional vector:

[0070]

[0071] The pre-computed lookup table m stores all possible products between {-8,-7,…,-1,0,1,…7} and {-8,-7,…,-1,0,1,…7}. Then, y can be computed by looking up the products in the pre-computed m, instead of performing multiplication on each term. Where p i It is encoding (x) i ,w i The one-hot vector of ), i.e.

[0072]

[0073] Second, reduce the strength of addition. Since vector multiplication satisfies the distributive law with respect to addition, we can first calculate ∑p i Only in the last step will ∑p i With m T Multiplying once, that is The main part of the calculation ∑p i This involves summing one-hot vectors. Therefore, binary addition is simplified to an increment operation on one of 256 pre-computed numbers. Increment operations are simpler to implement than addition. For example, binary addition requires a full adder, while increment operations only require a half adder. In this embodiment, the processing units in the processing unit array can replace multiply-adders with counters, using a total of 256 counters to implement the increment calculation ∑p. i It has very low conversion power consumption.

[0074] The following combination Figure 8 The above calculation process will be illustrated with an example. As shown in the figure, Figure 8 middle The overall architecture of the arithmetic circuit is organized as a pulsating array of output stationary (OS) data streams, where each PE executes an inner product and generates a complete output in the event of an interrupt. Internally, each PE contains 256 counters, which can be used to incrementally calculate ∑p through counting. iSince the calculation result of 0 does not need to be recorded, only 15 × 15 = 225 counters are actually deployed inside the PE. These counters can be any commercially available low-power counter. In this embodiment, the counters inside the PE are preferably ripple counters, which are composed of a series of cascaded flip-flops. Each flip-flop stores a 1-bit value, and the Q output of each flip-flop is connected to its input (D port). Whenever a rising edge of the clock arrives, the value of the flip-flop toggles, and the Q output is connected to the clock port of the next flip-flop. Whenever the value of a flip-flop jumps from 1 to 0, it sends a rising edge signal to the next flip-flop. This process continues, with the ripple counter completing an incrementing carry chain through the cascading of flip-flops. Compared to other counters, the ripple counter does not have other combinational circuits; it achieves its counting function through counter cascading, resulting in low power consumption. Furthermore, because it is composed of flip-flops, it is easily customized.

[0075] In actual calculation process, with Figure 8 Taking the PE in the top left corner as an example, this PE receives the row vector [1,2,-2,2] and the column vector [-1,1,0,1]. T The resulting inner product should be 3. In two's complement representation, -2 is represented as E(14), and -1 as F(15). Elements from the two input vectors can be piped to this PE. For example... Figure 8 middle As shown, in the first cycle, the PE receives 1 and F. The PE selects and activates the counter corresponding to (1,F) and increments it, recording the occurrence of 1×F, with the value in the counter incrementing from 0 to 1. In the next three cycles ( Figure 8 middle In the process, three elements are reached, and the corresponding counter values ​​are incremented accordingly. Although each PE contains many counters, only one counter is activated per cycle, and the power consumption of the counters is significantly lower than that of the MAC unit. Therefore, the conversion power consumption is still much lower than that of traditional MAC-based PEs.

[0076] After PE completes all calculations, ∑p is obtained. i The final step ( Figure 4 middle ) is achieved by multiplying by m T The final calculation result is obtained, which can be achieved through a dedicated converter (i.e., the aforementioned data conversion circuit). In some embodiments, the pre-calculated lookup table (m TThe pre-calculated lookup table stores the pre-computed results of the products between the matrix vectors {-8, -7, ..., -1, 0, 1, ..., 7} and {-8, -7, ..., -1, 0, 1, ..., 7}. This pre-computed lookup table can be stored in a data storage circuit (e.g., a memory). The dedicated converter utilizes all the operation results (∑p) of the counters in each processing unit. i The matrix multiplication result is obtained by combining the matrix multiplication result with the corresponding pre-calculated result.

[0077] Due to ∑p i and m T The dimensions are all fixed at 256, which is much shorter than the dimensions of the original matrix to be computed. When dealing with high-dimensional matrix multiplication, the power consumption of this transformation step is almost negligible. See also Figure 9 , Figure 9 Specifically, it demonstrates the traditional MAC PE (such as...) Figure 9 middle (as shown) and the PE structure based on the PMM algorithm of this disclosure (as shown) Figure 9 middle The power consumption of the corresponding logic devices (e.g., multiplier Mul, adder Add, counter Inc) and the number of operations required to process an 11008-dimensional matrix are shown. In this disclosure, the processing unit replaces the multiply-adder with a counter, which has very low conversion power consumption.

[0078] Therefore, by using a pre-computation lookup table to avoid repeated multiplication operations and replacing multiplication-addition operations with unary accumulation operations, dynamic power consumption of the hardware can be saved. It should be noted that in the PMM-based arithmetic circuit, the elements in the two matrix vectors used in the matrix multiplication calculation are not limited to INT4; they can also be other low-bit data types (such as low-bit floating-point numbers, or low-bit integer types such as INT2, INT3, INT5, INT6, etc.). The pre-computation result in the pre-computation lookup table and the number of counters in the processing unit can be adjusted and set according to the specific data type of the elements.

[0079] Furthermore, the arithmetic circuitry in this disclosed scheme can be optimized, specifically by using quarter-square multiplication (QSM) to optimize the number of counters contained in each processing unit in the processing unit array. The formula (1) for this QSM is as follows:

[0080]

[0081] When x and y are integers, xy must also be an integer. Furthermore, x+y and xy are either both even or both odd, therefore the division in the above formula always produces mutually canceling remainders. In other words, the following formula (2) holds true:

[0082] (x+y) 2 ≡(xy) 2 mod 4 (2)

[0083] Therefore, formula (3) holds true:

[0084] Based on the above theoretical analysis, all possible values ​​of the element-wise product of two matrix vectors in matrix multiplication can be transformed into: Where x and y are integer elements in the two matrix vectors used for matrix multiplication, and the pre-computation result in the pre-computation lookup table (Q(n)) is... Where n is the sum of the elements (x+y) of the two matrix vectors being multiplied, or the difference (xy) between the elements of the two matrix vectors being multiplied. The product xy can be calculated by one addition, one subtraction, and two lookups Q(n).

[0085] Therefore, in this embodiment, the pre-computed lookup table does not need to store all possible values ​​of the product of elements in two matrix vectors (e.g., a 256-dimensional pre-computed lookup table m), but only needs to store the intermediate transformation values ​​corresponding to all possible values ​​of the product of elements. That's it. At this point, each processing unit in the processing unit array includes multiple positive counters corresponding to the sum (x+y) of the elements in the two matrix vectors to be multiplied, and multiple negative counters corresponding to the difference (xy) of the elements in the two matrix vectors to be multiplied. The positive and negative counters correspond to the one-hot encoding of the pre-calculated result. In this application scenario, the arithmetic circuit performs the matrix multiplication calculation as follows: each processing unit in the processing unit array receives elements from the two input vectors to be multiplied, and within each calculation cycle, each processing unit activates the corresponding positive counter for incrementing based on the sum of the received elements and the corresponding negative counter for incrementing based on the difference of the received elements. The data conversion circuit multiplies the difference between all operation results of the positive counters and all operation results of the negative counters in each processing unit with the corresponding pre-calculated result to obtain the matrix multiplication result of the two input vectors.

[0086] Obviously, the length of the pre-computed lookup table Q(n) in this embodiment is much shorter than the 256-dimensional pre-computed lookup table m in the PMM-based arithmetic circuit mentioned earlier. For example, for a 4-bit integer (i.e., INT4) represented using two's complement, the range of x+y is [-16, 14], and the range of xy is [-15, 15]. Since Q(n) is symmetric about zero, i.e., Q(n) ≡ Q(-n). Furthermore, since Q(0) = Q(1) = 0, the range of n can be further reduced to [2, 16], a total of 15 items. Specifically, the pre-computed lookup table Q(n) is represented as a vector:

[0087] Q=[1,2,4,6,9,12,16,20,25,30,36,42,49,56,64]

[0088] In this application scenario, the number of counters in each processing unit of the processing unit array can be significantly reduced (e.g., from 256 counters to 29 counters). Specifically, each processing unit can include 15 positive counters and 14 negative counters. Both positive and negative counters can be any commercially available low-power counters; in this embodiment, ripple counters are preferred. It should be noted that in the arithmetic circuit based on the PMM and QSM algorithms, the elements in the two matrix vectors used for the matrix multiplication calculation are not limited to INT4; they can also be low-bit integers such as INT2, INT3, INT5, and INT6. The pre-calculation result in the pre-calculation lookup table and the number of counters in the processing unit can be adjusted and set according to the specific data type of the elements.

[0089] The following combination Figure 10 and Figure 11 The specific implementation of the operational circuit based on the PMM algorithm and the QSM algorithm is explained.

[0090] In this embodiment, the arithmetic circuit may include a data storage circuit storing the pre-computed lookup table Q(n), a processing unit array, and a data conversion circuit. Each processing unit (PE) in the processing unit array includes multiple positive and negative counters. For example, when the element in the input vector is INT4, each processing unit includes 15 positive counters and 14 negative counters. Furthermore, in this embodiment, the processing unit also includes an adder, a subtractor, and a corresponding decoder. When elements 5 and 3 enter a PE, elements 5 and 3 will respectively enter an adder and a subtractor for element-wise addition or subtraction. The results (8 and 2) pass through two decoders (Dec in the diagram), and the decoders output the corresponding one-hot code, which corresponds to the counter to be activated. Figure 10 middle As shown, the seventh counter in the positive counter group and the first counter in the negative counter group are activated and increment. Finally, as... Figure 10 middle As shown, the final calculation result is obtained through a dedicated converter (i.e., a data conversion circuit).

[0091] As an example Figure 11 A specific architecture of the aforementioned data conversion circuit is shown. For example... Figure 11 As shown, this data conversion circuit can be implemented using a converter. This converter is a five-stage pipeline architecture, and the conversion process can be divided into four steps: Step 1 The converter is directly connected to all counters in the processing unit array via a multiplexer. During the conversion process, each cycle selects and reads the values ​​stored in all PEs in one column. Steps The converter subtracts the value of the negative counter from the corresponding value of the positive counter. Steps step The result of the subtraction is multiplied by a preloaded value stored in a data storage circuit (e.g., a quantum register QREG); Step and The multiplication results are accumulated to produce the final output. Regarding power consumption, the dedicated converter does not operate during the accumulation phase of EMP SA, meaning it primarily generates static power. This static power is amortized by all PEs in the entire row of the systolic array, as PEs in different columns share the same converter circuitry through time multiplexing. For example, for multiplication of a 2048-dimensional square matrix, the conversion power accounts for only 1.4% of the total power consumption. Therefore, the computational circuit based on the PMM and QSM algorithms can effectively reduce the number of counters in the processing unit, thereby saving repetitive arithmetic, reducing accumulation intensity, and lowering static power consumption overhead.

[0092] Furthermore, this disclosure also protects an accelerator whose main computing unit can employ the computing circuits described in the above embodiments (e.g., a computing circuit based on the PMM algorithm or a computing circuit based on both the PMM and QSM algorithms). Figure 12 A specific architecture of the accelerator 1200 is shown. For example... Figure 12As shown, Accelerator 1200 is a primitive multiplication-based neural network accelerator (EMP ACC), employing the classic three-buffer neural network accelerator framework. It features a 128KB input neuron buffer (NBIn), a 128KB output neuron buffer (NBOut), and a 256KB synaptic buffer (SB). The buffers and main memory are connected via a direct memory access module (DMA). The main computational unit of the EMP ACC includes an arithmetic circuit based on the PMM and QSM algorithms (EMP SA), which can include multiple physical pairs (PEs) (e.g., 32×32 PEs). In addition to the EMP SA, there is a vector special function unit (SFU) for performing activation, normalization, and quantization operations on the output data. Furthermore, the controller (Ctrl) coordinates all modules in the EMP ACC and is equipped with an 8KB instruction buffer (IB). EMP ACC is a system-on-a-chip capable of running DNNs. The main computing unit in EMPACC adopts an operation circuit design based on the PMM and QSM algorithms, which can solve the problem of low bit width (INT4) power consumption bottleneck in neural network accelerators, realize efficient processing of low bit width matrix multiplication operations, and effectively reduce power consumption.

[0093] It should be noted that the description of the accelerator 1200 structure here is only an illustrative example, and the specific details can be adjusted according to the actual application scenario.

[0094] The solutions disclosed herein have been described in detail above with reference to the accompanying drawings. Depending on the application scenario, the devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare.

[0095] Furthermore, the devices or apparatuses disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminals. In one or more embodiments, the high-power devices or apparatuses according to the disclosed scheme can be applied to cloud devices (e.g., cloud servers), while the low-power devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling, and collaborative work of end-to-cloud or cloud-edge-end integration.

[0096] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0097] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the device or apparatus embodiments described above, this document divides them based on logical functions, but in actual implementation, there may be other division methods. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0098] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0099] In some implementation scenarios, the integrated unit described above can be implemented as a software program module. If implemented as a software program module and sold or used as an independent product, the integrated unit can be stored in a computer-readable storage device (CMSDD). Therefore, when the disclosed solution is embodied in a software product (e.g., a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to cause a computer device (e.g., a personal computer, server, or network device) to execute some or all of the steps of the method described in the embodiments of this disclosure. The aforementioned memory may include, but is not limited to, various media capable of storing program code, such as USB flash drives, flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0100] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the hardware structure of the circuit may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as CPUs, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage unit or storage device can be any suitable storage medium (including magnetic storage medium or magneto-optical storage medium, etc.), such as resistive random access memory ("RRAM"), dynamic random access memory ("DRAM"), static random access memory ("SRAM"), enhanced dynamic random access memory ("EDRAM"), high bandwidth memory ("HBM"), hybrid memory cube ("HMC"), ROM, and RAM, etc.

[0101] While the embodiments of this disclosure are described above, the content is merely an example for the purpose of facilitating understanding of this disclosure and is not intended to limit the scope or application scenarios of this disclosure. Any person skilled in the art can make any modifications and changes in form and detail of the implementation without departing from the spirit and scope disclosed herein; however, the patent protection scope of this disclosure shall still be determined by the scope defined in the appended claims.

Claims

1. An operational circuit, comprising: A data storage circuit that stores a pre-computation lookup table containing pre-computation results related to all possible values ​​of the element-wise product of two matrix vectors in a matrix multiplication calculation. The processing unit array includes multiple processing units, adjacent processing units are electrically connected, wherein each processing unit contains multiple counters, and the operation result of each counter is configured to index a pre-calculated result in the pre-calculated lookup table; as well as A data conversion circuit is electrically connected to each processing unit in the processing unit array and the data storage circuit to calculate the matrix multiplication result between matrix vectors based on all operation results of the counters in each processing unit and the corresponding pre-calculation results.

2. The arithmetic circuit according to claim 1, wherein the pre-calculation result in the pre-calculation lookup table is all possible values ​​of the element-wise product of the two matrix vectors in the matrix multiplication calculation, and the counter in each processing unit corresponds to the one-hot encoding of the pre-calculation result, and the arithmetic circuit is used to perform the matrix multiplication calculation as follows: Each processing unit in the processing unit array receives elements from two input vectors to be multiplied, and within each calculation cycle, each processing unit activates a corresponding counter to perform an increment operation based on the received elements; and The data conversion circuit is used to multiply all the operation results of the counter in each processing unit with the corresponding pre-calculated results to obtain the matrix multiplication result of the two input vectors.

3. The arithmetic circuit according to claim 2, wherein the elements in the two matrix vectors of the matrix multiplication calculation are all of type INT4, the pre-calculation lookup table stores the pre-calculation results of the product between the matrix vectors {-8, -7, ..., -1, 0, 1, ..., 7} and {-8, -7, ..., -1, 0, 1, ..., 7}, and each processing unit in the processing unit array contains 256 or 225 counters.

4. The operational circuit according to claim 1, wherein all possible values ​​of the element-wise product of the two matrix vectors in the matrix multiplication calculation are transformed into: in, x and y are integer elements in the two matrix vectors used for matrix multiplication. The pre-calculation result in the pre-calculation lookup table is... Where n is x+y or xy.

5. The arithmetic circuit according to claim 4, wherein each processing unit includes a plurality of positive counters corresponding to the sum of elements in the two matrix vectors for the matrix multiplication calculation and a plurality of negative counters corresponding to the difference of elements in the two matrix vectors for the matrix multiplication calculation, the positive counters and the negative counters corresponding to the one-hot encoding of the pre-calculation result, and the arithmetic circuit is configured to perform the matrix multiplication calculation as follows: Each processing unit in the processing unit array receives elements from two input vectors to be multiplied. Within each calculation cycle, each processing unit activates a corresponding positive counter for incrementing based on the sum of the received elements and activates a corresponding negative counter for incrementing based on the difference of the received elements. The data conversion circuit is used to multiply the difference between all operation results of the positive counter and all operation results of the negative counter in each processing unit with the corresponding pre-calculated result to obtain the matrix multiplication result of the two input vectors.

6. The arithmetic circuit according to claim 5, wherein the two matrix vectors for the matrix multiplication calculation include low-bit matrix vectors.

7. The arithmetic circuit according to claim 6, wherein the two matrix vector elements in the matrix multiplication calculation are both of type INT4, the value range of n is [2, 16], and each processing unit in the processing unit array contains 15 positive counters and 14 negative counters.

8. The operational circuit according to any one of claims 1 to 7, wherein the counter in the processing unit comprises a ripple counter.

9. An accelerator, wherein the main computing unit of the accelerator employs the computing circuit according to any one of claims 1 to 8.

10. A chip comprising the arithmetic circuit according to any one of claims 1 to 8.

11. A circuit board comprising the chip according to claim 10.

Images