A ternary tensor core and matrix multiply-add array architecture based on dual rail sign-magnitude encoding

Abstract

The application discloses a ternary tensor core and matrix multiplication array architecture based on double-line symbol flag coding, and belongs to the technical field of artificial intelligence chips and tensor processors. The application adopts two standard CMOS binary signal lines to jointly represent a ternary number, and the two signal lines independently identify whether the number is zero and whether the number is negative. The value combination (1, 0) represents +1, the value combination (0, 0) represents 0, and the value combination (1, 1) represents -1. A ternary multiplier is composed of a cross switch array formed by standard CMOS transmission gates. Each multiplication unit realizes multiplication +1, multiplication 0 and multiplication -1 operations through three paths of straight-through, shutdown and cross, and nine input combinations are fully covered, and the hardware overhead is reduced by more than 90% compared with a binary multiplier. Forty multiplication units are arranged in parallel, and 40-way ternary multiplication operations are completed in a single cycle. The operation results are output after being sequentially accumulated by a ternary addition tree and an accumulator array. The application is applied to AI large model reasoning and ternary quantization neural network acceleration scenes, can realize high-density data packaging, extremely simple multiplier array and native sparse acceleration, and is seamlessly compatible with an existing binary ecology through a pipeline codec.

Description

Technical Field

[0001] This invention belongs to the technical fields of artificial intelligence chips, tensor processors, and matrix multiply-accelerate accelerators. Specifically, it relates to a ternary tensor core and matrix multiply-accelerate array architecture based on double-line symbolic encoding, which is suitable for high-performance computing scenarios such as AI large model inference, acceleration of ternary quantized neural networks, and large-scale matrix multiply-accelerate operations. Background Technology

[0002] 2.1 Current Status of Tensor Core Technology Tensor Cores are the core computing units in current AI acceleration chips. Major chip manufacturers such as NVIDIA, AMD, and Huawei integrate Tensor Cores into their GPUs / AI chips to accelerate matrix multiply-accumulate (MAC) operations. The basic architecture of existing Tensor Cores is as follows: (1) Data format: Use binary floating-point number (FP16 / BF16 / FP8) or binary integer (INT8 / INT4) format for storage and operation.

[0003] (2) Calculation mode: Within one clock cycle, multiple data are read from registers or shared memory, and matrix multiplication and addition operations are completed through a multiplier array.

[0004] (3) Core structure: It consists of a multiplier array, an adder tree, and an accumulator.

[0005] (4) Storage access: Data is loaded / stored through shared memory and register stack, relying on multi-level cache hierarchy.

[0006] 2.2 Technical shortcomings of existing tensor cores Existing binary tensor cores have the following fundamental drawbacks when processing ternary neural networks (TNNs): (1) Severe waste of bit width: Three-value weights only need to express three states {-1, 0, +1}. The lower bound of information theory is log23≈1.585bit, but the existing tensor core uses INT8 (8bit) or FP16 (16bit) for storage, resulting in extremely low bit width utilization.

[0007] (2) The hardware overhead of multipliers is large: binary multipliers require a complete multiplier array (hundreds to thousands of transistors), while ternary multiplication (-1, 0, +1) degenerates into signal selection (MUX), which can reduce hardware overhead by more than 90%.

[0008] (3) High memory access bandwidth pressure: Due to bit width redundancy, the amount of data that the binary scheme needs to move under the same matrix size is much greater than the theoretical minimum, which makes memory access a performance bottleneck.

[0009] (4) Unable to utilize ternary sparsity: A large number of elements in a ternary matrix are 0, and the existing tensor core cannot effectively utilize this sparsity to accelerate computation.

[0010] 2.3 Current Research Status of Ternary / Ternary Related Technologies Several solutions for ternary neural networks and ternary logic have emerged in academia and industry: (1) Renesas Electronics Ternary In-Memory Accelerator: A Ternary In-Memory Solution Based on SRAM.

[0011] (2) TENET architecture: a sparse-aware LUT-centric architecture used for reasoning in ternary large language models.

[0012] (3) TiM-DNN: Ternary in-memory accelerator to realize large-scale parallel signed ternary vector-matrix multiplication.

[0013] (4) Huawei ternary logic gate circuit (CN119652311A): Implements ternary logic operation unit, which can reduce the number of transistors and reduce power consumption.

[0014] However, all of the above solutions have the following shortcomings: The ternary / ternary encoding was not deeply integrated with the tensor core architecture. The problem of efficient packing, transmission, and parallel computing of ternary data in tensor kernels remains unsolved. No complete, mass-producible ternary tensor core architecture solution was provided. There is currently no ternary tensor core architecture based on double-line symbol encoding that can simultaneously achieve: high-density data packing, minimal multiplier array, native sparse acceleration, and seamless compatibility with the existing binary ecosystem. Summary of the Invention

[0015] 3.1 Technical problem to be solved by the present invention To overcome the shortcomings of existing binary tensor cores in processing ternary data, such as wasted bit width, high multiplier overhead, high memory access pressure, and inability to utilize sparsity, this invention provides a ternary tensor core and matrix multiply-accumulate array architecture based on double-line sign flag encoding, specifically solving the following technical problems: (1) Solve the problem of high-density storage and transmission of ternary data in tensor kernels, and compress the bit width from 8bit / 16bit to about 1.585bit.

[0016] (2) To solve the problem of excessive hardware overhead of ternary multipliers, the multiplier is degraded from a large-scale array to a signal selection (MUX) plus cross switch.

[0017] (3) Solve the problem that the sparsity of zero values ​​in a ternary matrix cannot be utilized, and implement a hardware acceleration mechanism for zero value skipping.

[0018] (4) Solve the compatibility problem between the ternary tensor core and the existing binary ecosystem, and achieve seamless data interaction through pipelined codec.

[0019] 3.2 Complete Mathematical Principles and Logical Definition of a Ternary Multiplier 3.2.1 Input / output format of a ternary multiplier The ternary multiplier defined in this invention uses a double-line sign-coded format for both its input and output: Multiplicand X: Represented by two standard CMOS binary signal lines (NZ_X, NG_X), where NZ_X=1 indicates non-zero and NG_X=1 indicates negative. The combined encoding is as follows: X = +1: NZ_X=1, NG_X=0 X = 0: NZ_X=0, NG_X=0 X = -1: NZ_X=1, NG_X=1 Multiplier M: It is also represented by two standard CMOS two-state signal lines (NZ_M, NG_M), and the encoding method is the same as that of the multiplicand.

[0020] The output Y = X × M is represented by two standard CMOS two-state signal lines (NZ_OUT, NG_OUT).

[0021] 3.2.2 Nine Combinatorial Coverage Logics of Ternary Multipliers A ternary multiplier needs to process nine combinations of the multiplicand X (3 states) and the multiplier M (3 states). This invention proposes a "multiplier-driven" multiplier architecture, where the operation mode of the signal lines is determined by the state of the multiplier M. The complete mapping of the nine combinations is as follows: Multiplicand X, Multiplier M, Output Result Y, Hardware Operation Corresponding Path +1 (1,0) +1 (1,0) +1 Signal line straight through 0 (0,0) +1 (1,0) 0 Signal line straight through -1 (1,1) +1 (1,0) -1 Signal line straight through +1 (1,0) 0 (0,0) 0 Output off 0 (0,0) 0 (0,0) 0 Turn off output Turn off -1 (1,1) 0 (0,0) 0 Output off +1 (1,0) -1 (1,1) -1 Signal line exchange crossover 0 (0,0) -1 (1,1) 0 Turn off output (prioritizes multiplicand being 0) -1 (1,1) -1 (1,1) +1 Signal line exchange crossover All nine combinations have definite mappings, achieving 100% coverage, and there are no undefined states. This distribution is very reasonable for the following reasons:

[0022] The inherent sparsity of ternary matrices lies here—multiplying any number by 0 results in 0, and 0 itself accounts for one-third of the states. In real AI ternary quantization models, the actual proportion of zero values ​​is typically as high as 50%-90%. Zero values ​​appear most frequently in multiplier arrays, which is the physical basis for the significant role your sparse acceleration mechanism can play—when a large number of multipliers are 0, the clock gating is directly turned off, resulting in significant power savings.

[0023] The product of a positive number and a negative number is positive. Therefore, +1 appears twice (1×1 and -1×-1). The product of a positive number and a negative number is negative. Therefore, -1 also appears twice (-1×1 and 1×-1).

[0024] The symmetrical distribution perfectly conforms to mathematical logic. This 5:2:2 ratio also illustrates one thing: In random ternary data, the probability of a multiplier outputting zero is approximately 56%, meaning that more than one out of every two multiplication operations will result in zero. This is precisely the core value of ternary sparsity acceleration—56% of the computation results are directly zero, eliminating the need for subsequent additions. The hardware can skip these operations directly (clock gating is off), thereby significantly reducing dynamic power consumption.

[0025] 3.2.3 Physical meaning of the three hardware operation paths Based on the above nine combinations of mapping relationships, this invention simplifies the ternary multiplier into the following three hardware operation paths: The output result of the operation multiplier M on the multiplicand signal line. Straight-through +1 NZ_X Straight-through becomes NZ_OUT, NG_X Straight-through becomes NG_OUT NZ_OUT = NZ_X, NG_OUT = NG_X Turning off 0 or when the multiplicand is 0 closes the input path and sets the output to zero: NZ_OUT = 0, NG_OUT = 0 After swapping NZ_X and NG_X, the outputs are NZ_OUT = NG_X and NG_OUT = NZ_X. Special note: When the multiplicand X = 0, the output is 0 regardless of the value of the multiplier M (+1, 0, or -1). In this case, the multiplicand is prioritized to be 0 and a shutdown operation is performed to avoid redundant circuit operation and reduce dynamic power consumption.

[0026] 3.2.4 The essential difference between it and a binary multiplier To facilitate understanding of the inventiveness of this invention, a comparison is made between a ternary multiplier and a binary multiplier: Comparison of dimensional ternary multipliers (this invention) and binary multipliers (prior art) 2 input signal lines per number (4 inputs) Multi-bit binary parallel data bus The number of input states is 3, resulting in 9 combinations, which in turn leads to 2^N combinations. The processing method involves selecting the signal path based on the multiplier state, generating a partial product, and then accumulating the addition tree. Key operations: pass-through / shutdown / exchange (no arithmetic operations required); multi-digit multiplication + partial product summation (arithmetic operations required). Hardware implementation of a transmission gate cross switch (approximately 30-50 transistors) and a multiplier array + adder tree (approximately 800-1200 transistors). The core problem with binary multipliers is that they must handle the dimension of "size". Multiple binary numbers require generating multiple partial products, which are then summed using an addition tree. This is a complete arithmetic operation that cannot be accomplished with simple pass-through / off / switch signal lines.

[0027] The ternary multiplier defined in this invention has only three input states: -1, 0, and +1, without the dimension of "size". Therefore, the multiplication operation degenerates into three signal line operations (pass-through / off / switching), requiring no arithmetic operation unit.

[0028] 3.3 Core Technology Solution This invention provides a ternary tensor core and matrix multiply-accumulate array architecture based on double-line symbolic encoding, which comprises five core units (as shown in Figure 1): (1) Ternary packed data loading unit (shown as number (101) in Figure 1): 64-bit packed ternary data (40 ternary numbers per packet) is read from on-chip storage or shared memory, and split into 40 independent NZ / NG dual-line symbol flag signals by a parallel unpacking circuit, which are then fed into the multiplier array. The loading unit has a built-in dual-port buffer, which supports loading two sets of packed data (matrix A and matrix B) simultaneously, realizing parallel data loading and computation.

[0029] (2) Three-valued multiplier cross switch array (shown as number (102) in Figure 1): Employing the unique cross-switch array structure of the transmission gates (as shown in Figure 2), each multiplication unit requires only 4 standard CMOS transmission gates (TG1-TG4) and 1 2:1 multiplexer (MUX) to achieve multiplication by +1, 0, and -1. Multiply by 1: Signal pass-through (TG1 / TG4 conduct) Multiply by 0: Closes the input path, blocks signal input, and sets the output to a zero state. Multiply by -1: Switch positive and negative signal lines (TG2 / TG3 on) The 40 multiplication units are arranged in parallel, completing 40 three-way multiplication operations in a single cycle, reducing hardware overhead by more than 90% compared to binary multiplier arrays.

[0030] (3) Ternary adder tree and accumulator array (as shown by label (103) in Figure 1): The 40-way multiplication results are fed into a ternary adder tree, using a zero-padded binary tree structure: 40→20→10→8 (padded with 2 zeros)→4→2→1. Each level uses a ternary adder (1 to 2 levels of logic gates), and the final result is stored in a ternary accumulator. The accumulator supports both three-value accumulation and standard binary accumulation modes, which are switched via a mode register.

[0031] (4) Pipeline codec unit (as shown by label (104) in Figure 1): The data input / output path, located within the tensor core, encodes binary two's complement data into NZ / NG double-lined symbol signals in real time at the input direction; and decodes ternary operation results into standard binary data format in real time at the output direction. The conversion latency is completely hidden within the pipeline throughput gaps, enabling seamless data interoperability with the existing binary ecosystem.

[0032] (5) Ternary tensor core controller (shown as number (105) in Figure 1): The complete operation flow of the control tensor core includes: instruction decoding, data loading, multiplication array scheduling, addition tree accumulation, and result writing back. The controller has built-in sparse acceleration logic. When a multiplier of 0 is detected, the dynamic power consumption of that multiplier is turned off through clock gating. The input of the corresponding addition tree is zero, and the accumulator accumulates normally (adding zero operation), effectively utilizing the natural sparsity of the three-value matrix.

[0033] 3.4 Parallel Architecture of Matrix Multiply-Add Array This invention supports two parallel matrix multiplication and addition modes (as shown in Figure 3): Mode 1: Tiled Parallel Mode Large matrices are divided into multiple tiles, each processed in parallel by a set of tensor cores, with results aggregated via an on-chip interconnect bus. This approach is suitable for high-throughput computation scenarios involving large-scale matrices.

[0034] Mode 2: Pipeline Parallel Mode Multiple tensor cores form a deep pipeline, with different cores handling different stages of matrix multiplication and addition (read → multiplication → addition → accumulation → write back), which is suitable for streaming data scenarios (such as Transformer layer-by-layer inference).

[0035] 3.5 Ternary Sparsity Acceleration Mechanism This invention incorporates hardware sparse acceleration logic, which detects when the ternary multiplier is 0: With the clock gate of this multiplier array turned off, dynamic power consumption is reduced to zero. When the input to the addition tree for this path is zero, the addition tree will perform normal accumulation (adding zero). The accumulator continues to accumulate without stopping due to the zero value channel. This mechanism can make full use of the large number of zero values ​​(usually accounting for 50%-90%) in a ternary matrix to achieve a significant improvement in effective computing power.

[0036] 3.6 Coding Physical Layout and Hardware Overhead Based on TSMC N3B technology and a 3.2GHz clock speed, the hardware overhead of a 40-channel ternary multiplier crossbar switch array is as follows: Comparison: Binary INT8 Tensor Core (8×8 Multiplication Array) This Invention: Ternary Tensor Core (40-way Parallel) The multiplier has approximately 800-1200 transistors, or about 30-50 transistors (transmission gate + MUX). Single-cycle throughput: 8 multiplications / 40 multiplications Data width: 64 bits (8 INT8s) 64 bits (40 three-valued numbers) The multiplier latency is approximately 150ps, approximately 45ps. Attached Figure Description

[0037] Figure 1 is a block diagram of the overall architecture of the ternary tensor core and matrix multiply-accumulate array of the present invention.

[0038] Label (101): Ternary Packed Data Loading Unit Label (102): Triple multiplier crossbar switch array (40-way parallel) Label (103): Ternary Adder Tree and Accumulator Array Label (104): Pipeline codec unit Label (105): Ternary Tensor Core Controller Figure 2 is a schematic diagram of the transistor-level circuit structure of the cross-switch array of the ternary multiplier of the present invention.

[0039] Label (201): Transmission gate TG1 (straight-through path, conducts when multiplied by +1) Label (202): Transmission gate TG2 (cross path, conducts when multiplied by -1) Label (203): Transmission gate TG3 (cross path, conducts when multiplied by -1) Label (204): Transmission gate TG4 (straight-through path, conducts when multiplied by +1) Label (205): 2:1 Multiplexer (MUX), selects a straight-through or cross-path. Label (206): Shutdown control switch (under the multiplier 0 condition, shuts down all input transmission gates, blocks valid signals, and outputs a logic zero state flag) Figure 3 is a schematic diagram of two parallel modes of the matrix multiplication and addition array of the present invention.

[0040] Label (301): Tiled Parallel mode Label (302): Pipeline Parallel mode Figure 4 is a timing diagram of the ternary tensor core operation pipeline of the present invention.

[0041] Label (401): Load Stage Label (402): Multiply Stage Label (403): Add Stage Label (404): Accumulate Stage Label (405): Write-back Stage Detailed Implementation

[0042] Example 1: Accelerating Inference for Large Models Using Ternary Quantization Hardware configuration: TSMC N3B process, 3.2GHz main frequency, integrating 128 sets of the present invention's ternary tensor cores (40 parallel paths per set), and 16MB of on-chip shared memory.

[0043] Model to be accelerated: BitNet b1.58 ternary quantization large model (weight values ​​{-1, 0, +1}), model size 3 billion parameters, inference sequence length 2048 tokens.

[0044] Operation process: (1) The ternary weights are stored in on-chip shared memory in a 64-bit packet format (40 ternary values / packet); (2) The tensor core controller receives matrix multiplication and addition instructions and reads packaged data from shared memory through the loading unit (101); (3) The loading unit unpacks the data into 40 NZ / NG dual-line symbol flag signals in parallel and sends them to the multiplier cross switch array (102); (4) The 40-way multiplier completes the three-value multiplication operation in a single cycle, and the result is sent to the addition tree and accumulator (103); (5) The sparse acceleration logic automatically detects channels with a multiplier of 0 and shuts down the dynamic power consumption of that channel through clock gating. After zero-value channels are padded with zeros, the addition tree accumulates normally. (6) The accumulated result is decoded into standard binary format by Codec (104) and written back to shared memory or general-purpose registers.

[0045] Expected results: Compared to the same process and area binary INT8 tensor core, thanks to the combined effects of 40-way parallelism, reduced latency of the simplified multiplier (150ps → 45ps), and ternary sparse acceleration, throughput can be increased by several times to more than ten times under ideal conditions, while dynamic power consumption is significantly reduced. Specific performance data varies depending on the process version, implementation method, and operating conditions.

[0046] Example 2: Large-scale matrix multiplication and addition parallel computing scenario Hardware configuration: 4 sets of ternary tensor cores of this invention, each with 40 parallel paths, connected through an on-chip interconnect bus.

[0047] Matrix to be calculated: 2048×2048 ternary matrix multiplication.

[0048] Operation process: (1) The controller divides the large matrix into multiple sub-blocks (Tile) and distributes them to 4 groups of tensor cores for parallel processing; (2) Each core independently completes the three-value multiplication and addition operations of the sub-blocks, and implements the five-level pipeline of data loading, multiplication, addition, accumulation and write-back through the pipeline parallel mode (302); (3) The core operation results of each group are aggregated through the on-chip interconnect bus to generate a complete result matrix.

[0049] Expected results: The parallel acceleration of 4 cores improves performance by about 3.8 times compared to a single core (affected by interconnect bus overhead), and has significant acceleration and low power consumption advantages compared to binary tensor core solutions of the same size.

[0050] VI. Beneficial Effects (1) Data density is greatly improved: 64 bits pack 40 three-value numbers, the storage density is 5 times that of the binary INT8 scheme, and the memory access bandwidth pressure is reduced to 1 / 5.

[0051] (2) The hardware overhead of the multiplier is reduced by more than 90%: the ternary multiplication is degenerated into a signal selection (MUX) plus a cross switch, and the number of transistors is reduced from 800-1200 to 30-50.

[0052] (3) Native sparsity acceleration: The hardware automatically detects that the multiplier is 0 and turns off the dynamic power consumption through clock gating. After the zero-value channel is filled with zero, the addition tree is accumulated normally, making full use of the natural sparsity of the three-value matrix (zero value accounts for 50%-90%), and the effective computing power increases linearly with the sparsity.

[0053] (4) Seamless compatibility with binary ecosystem: Pipeline codec enables real-time data format conversion, with conversion delay hidden in pipeline throughput gaps.

[0054] (5) Multi-mode parallel adaptation: Supports two modes: flat parallel and pipeline parallel, adapting to two scenarios: large matrix high throughput and streaming data low latency.

[0055] (6) The process is mass-producible: all are implemented by standard CMOS standard cell library (transmission gate, MUX, AND gate, OR gate, NOT gate), without the need for customized analog circuits or special devices, and is 100% compatible with advanced processes.

Claims

1. A ternary multiplier cross switch array, characterized in that, It is composed of standard CMOS transmission gates, and each multiplication unit implements the operations of multiplying by +1, multiplying by 0, and multiplying by -1 through three paths: pass-through, turn-off, and cross-through, respectively. The multiplication by +1 indicates a direct signal line output; Multiplying by 0 closes the input path, blocks signal input, and outputs a zero-state flag. The multiplication by -1 represents the output path of the two signal lines being switched.

2. The ternary multiplier cross switch array according to claim 1, characterized in that, Each multiplication unit consists of four standard CMOS transmission gates (TG1-TG4) and one 2:1 multiplexer (MUX). When multiplying by +1, TG1 / TG4 are turned on; when multiplying by 0, all transmission gates are turned off; when multiplying by -1, TG2 / TG3 are turned on to achieve signal exchange. The overall multiplication unit has no arithmetic operation circuit; it only completes the three-value multiplication mapping by switching the transmission gate paths.

3. A ternary tensor core, characterized in that, include: (1) The ternary multiplier cross switch array as described in claim 1 or 2; (2) A ternary packed data loading unit reads 64-bit packed ternary data from on-chip storage and splits it into multiple NZ / NG dual-line symbol flag signals through a parallel unpacking circuit and sends them into the multiplier array; (3) A ternary adder tree and accumulator array are used to accumulate the results of multiplication step by step and output the results in three-value or binary format.

4. The ternary tensor core according to claim 3, characterized in that, It also includes a pipelined encoding / decoding unit that encodes binary data into NZ / NG double-line symbol flags or decodes the results of ternary operations into standard binary format.

5. The ternary tensor core according to claim 3, characterized in that, It also includes a tensor core controller, which controls data loading, multiplication array scheduling, addition tree accumulation and result writing back, and has built-in sparse acceleration logic.

6. The ternary tensor core according to claim 5, characterized in that: The sparse acceleration logic configures the channel mask register; when a multiplier of 0 is detected, the clock gate and input path of the corresponding channel multiplier are closed, and the corresponding adder tree channel is automatically padded with zeros; By marking the zero-value channel position through the mask register, the data stream alignment is automatically completed, ensuring that the timing and dimension of the entire 40-channel output are fully matched, and the accumulator continues to accumulate normally throughout the process.

7. A ternary matrix multiply-add array architecture, characterized in that, It comprises multiple sets of ternary tensor cores as described in any one of claims 3 to 6, connected via an on-chip interconnect bus, and supports both tiled parallel and pipelined parallel operating modes. (1) In the flat parallel mode, the large matrix is ​​divided into multiple sub-blocks and distributed to multiple tensor cores for parallel processing; (2) In the pipeline parallel mode, multiple tensor cores constitute a deep pipeline to handle the five stages of data loading, multiplication, addition, accumulation and write-back respectively.

8. An AI acceleration chip, characterized in that, Includes the ternary tensor core of any one of claims 3 to 6, or the ternary matrix multiply-accumulate array architecture of claim 7.

Images