Matrix multiplication calculation device for AI accelerator integrated circuit and AI accelerator device

By using data compression and decompression technology and block floating point numerical format in the matrix computing device, the problems of dense computing and high memory requirements are solved, efficient matrix multiplication calculation is realized, and the performance and efficiency of AI accelerators are improved.

CN221960554UActive Publication Date: 2024-11-05D-MATRIX CORP
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202322826485.7
Authority / Receiving Office
CN · China
Patent Type
Utility models(China)
Current Assignee / Owner
Priority Date
2022-10-21
Filing Date
2023-10-20
Publication Date
2024-11-05
Estimated Expiration
2033-10-20

AI Technical Summary

Technical Problem

Existing transformer models have high computational density and memory requirements for computing workloads in natural language processing and other machine learning applications, resulting in time and inefficiency, especially in the training and inference process of large-scale models.

Method used

A matrix computing device was designed, through data compression and decompression technology, using the combination of memory devices and computing devices, including registers, converters and computing devices, to optimize the storage and processing of weight matrix, realize efficient calculation of matrix multiplication, and support blocks. Floating point numerical format to improve computational efficiency.

Benefits of technology

It realizes that without increasing the cost of silicon area, improves computing efficiency and memory utilization, reduces memory operation latency, and improves the training and inference performance of large-scale models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN221960554U_ABST
    Figure CN221960554U_ABST
Patent Text Reader

Abstract

The utility model relates to a matrix multiplication calculation device for an AI accelerator integrated circuit and an AI accelerator device. The utility model relates to a device for compressing a neural network weight block in a computing accelerator, and provides a matrix multiplication computing device. The apparatus includes a memory configured to store weight matrix elements in a first format, the first format including a scaling factor and columns of mantissa blocks. A first register is configured to receive scale factors for each column of weight matrices through a crossbar device coupled to a memory, a converter is configured to determine a maximum index using the scale factors for the columns, and a second register is configured to store the maximum index. Further, the first register is configured to receive mantissa blocks of the column, and the converter is configured to determine a conversion block using all scaling factors and mantissa blocks. A weight buffer receives the conversion block and the maximum exponent as elements of a second format, which is used by the computing device to determine matrix multiplication outputs, which are stored in an output buffer.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application generally relates to integrated circuit (IC) devices and artificial intelligence (AI). More specifically, the present application relates to a device structure for accelerating computing workloads, such as computing workloads in transformer-based models (also known as transformers). Background Art

[0002] Transformers have been the dominant neural network architecture in natural language processing (NLP) and their use is expanding to other machine learning applications. The original Transformer was introduced in “Attention is all you need” (Vaswani et al., 2017), which led to the development of many Transformer model variants, such as the Generative Pretrained Transformer (GPT) model and the Bidirectional Encoder Representations from Transformers (BERT) model. These Transformers significantly outperform other models in inference tasks because they use a self-attention mechanism that avoids recurrence and allows for easy parallelization. On the other hand, Transformer workloads are very computationally intensive and memory-intensive, and have historically suffered from being time-intensive and inefficient.

[0003] Recently, both the model size and computational requirements of NLP models have grown by a thousand-fold. For example, it can take about 4 months for 1024 graphics processing units (GPUs) to train a model like GPT-3 with 175 billion parameters. Currently, new NLP models with a trillion parameters are under development, and models with trillions of parameters are on the horizon. Such rapid growth makes it increasingly difficult to serve large-scale NLP models.

[0004] As can be seen above, improved devices for accelerating computational workloads for AI are highly desirable. Summary of the invention

[0005] The present application generally relates to integrated circuit (IC) devices and artificial intelligence (AI) systems. More specifically, the present application relates to device structures for accelerating computing workloads, such as device structures in transformer-based neural network models (also known as transformers), etc. These structures can be used for machine / deep learning applications such as natural language processing (NLP), computer vision (CV), etc. As an example only, the present application has been applied to AI accelerator devices and core grain devices configured in PCIe cards.

[0006] According to an example, the present application relates to data compression and decompression in a matrix computing device. In some applications, it is necessary to improve the processing of large amounts of data. For example, a transformer-based modeling network typically involves a large number of elements (e.g., weights, activations, etc.), which cannot all be stored in on-chip memory. Therefore, access to these elements requires frequent transfers from a memory storage device (e.g., DDR), which may cause the processing of these elements to be memory-constrained due to the large latency of such memory operations.

[0007] In this example, the present application provides a matrix multiplication computing device for an AI accelerator integrated circuit, which is configured as an integrated circuit, and the device includes: a memory device, the memory device is configured to store a plurality of weight matrix elements in a first format, the first format includes a plurality of weight matrix columns, each weight matrix column includes a plurality of scaling factors and a plurality of mantissa blocks; a crossbar device coupled to the memory device; a first register coupled to the crossbar device, the first register is configured to receive a plurality of scaling factors and a plurality of mantissa blocks for each weight matrix column; a converter device coupled to the crossbar device, the converter device is configured to determine a maximum exponent of each weight matrix column using the plurality of scaling factors of the weight matrix column, thereby generating a plurality of maximum exponents; and a converter device configured to determine a plurality of conversion mantissa blocks using a plurality of scale factors and a plurality of mantissa blocks for a plurality of matrix weight columns; a second register coupled to the crossbar device, the second register configured to store the plurality of maximum exponents; a weight buffer device coupled to the crossbar device, the weight buffer device configured to receive a plurality of weight matrix elements in a second format, the second format comprising a plurality of conversion mantissa blocks and a plurality of maximum exponents; a computing device coupled to the weight buffer device, the computing device configured to determine a plurality of matrix multiplication outputs using the plurality of weight matrix elements in the second format; and an output buffer device coupled to the computing device, the output buffer device configured to store the plurality of matrix multiplication outputs.

[0008] In this example, the first register, the second register, and the converter device are configured in a crossbar device. Each mantissa block includes one or more mantissas, and wherein each of the plurality of scale factors is associated with one of the plurality of mantissa blocks; wherein the converter device is configured to determine the plurality of converted mantissa blocks by multiplying each mantissa of each mantissa block by an associated scale factor to produce a scaled mantissa, shifting each scaled mantissa of each mantissa block to produce a shifted mantissa, and rounding each shifted mantissa of each mantissa block to produce a rounded mantissa. The plurality of matrix inputs of the first format are represented by a 36×64 byte storage configuration, and wherein the plurality of matrix inputs of the second format are represented by a 65×64 byte storage configuration. Each of the first format and the second format includes a block floating point format; wherein each of the plurality of scale factors is represented by an unsigned 8-bit floating point scale factor, the unsigned 8-bit floating point scale factor including a 4-bit exponent field and a 4-bit fraction field; and wherein each of the plurality of mantissa blocks is represented by an 8-byte block, wherein each byte of the 8-byte block stores two 4-bit integer mantissa values.

[0009] In an example, the present application provides an AI accelerator device, which includes: a memory storage device, the memory storage device is used to store a plurality of weight matrix elements in a first format, the first format includes a plurality of matrix weight columns, each weight matrix column includes a plurality of scaling factors and a plurality of mantissa blocks; at least one chip device coupled to the memory storage device, the chip device includes: a central processing unit CPU; a first register coupled to the CPU, the first register is configured to receive a plurality of scaling factors and a plurality of mantissa blocks for each weight matrix column; a converter device coupled to the CPU, the converter is configured to use the plurality of scaling factors of the weight matrix column to determine each weight a maximum exponent of a matrix column, thereby generating a plurality of maximum exponents; and a converter device configured to determine a plurality of conversion mantissa blocks using a plurality of scale factors and a plurality of mantissa blocks for the plurality of matrix weight columns; a second register coupled to the CPU, the second register configured to store the plurality of maximum exponents; and a plurality of slices coupled to the CPU, each slice having a memory device and a computing device; wherein the plurality of memory devices are configured to receive a plurality of weight matrix elements in a second format, the second format comprising a plurality of conversion mantissa blocks and a plurality of maximum exponents for each matrix input; and wherein the plurality of computing devices are configured to determine a plurality of matrix multiplication outputs using the plurality of weight matrix elements in the second format.

[0010] In this example, the first register, the second register, and the converter device are configured in a scheduling device, and the scheduling device is coupled to the CPU. Each mantissa block includes one or more mantissas, and wherein each of the plurality of scale factors is associated with one of the plurality of mantissa blocks; wherein the converter device is configured to determine the plurality of converted mantissa blocks by multiplying each mantissa of each mantissa block by the associated scale factor to produce a scaled mantissa, shifting each scaled mantissa of each mantissa block to produce a shifted mantissa, and rounding each shifted mantissa of each mantissa block to produce a rounded mantissa. The plurality of matrix inputs in the first format are characterized by a 36×64 byte storage configuration, and wherein the plurality of matrix inputs in the second format are characterized by a 65×64 byte storage configuration. Each of the first format and the second format comprises a block floating point format; wherein each of the plurality of scale factors is represented by an unsigned 8-bit floating point scale factor, the unsigned 8-bit floating point scale factor comprising a 4-bit exponent field and a 4-bit fraction field; and wherein each of the plurality of mantissa blocks is represented by an 8-byte block, wherein each byte of the 8-byte block stores two 4-bit integer mantissa values.

[0011] In an example, the present application provides a matrix multiplication computing device and an operating method thereof. The device includes a memory device, which is configured to store multiple weight matrices in a first format including multiple matrix weight columns, wherein each matrix weight column includes multiple scale factors and multiple mantissa blocks. A crossbar device is coupled to the memory device and one or more computing paths, which include a weight buffer (WB) device, a computing device, and an output buffer device. A first register device coupled to the crossbar device is configured to receive multiple scale factors for each weight matrix column, and a converter device is configured to determine the maximum index of each column using multiple scale factors of the column. A second register device coupled to the crossbar device is configured to store the obtained multiple maximum indexes. In addition, the first register device is configured to receive multiple mantissa blocks for each column, and the converter device is configured to determine multiple converted mantissa blocks using all multiple scale factors and multiple mantissa blocks. The WB device is configured to receive multiple converted mantissa blocks and multiple maximum indexes, thereby generating multiple weight matrix elements in a second format. Then, the computing device determines multiple matrix multiplication outputs using multiple matrix weight elements in the second format, and the OB device stores multiple matrix multiplication outputs.

[0012] In an example, the first register device, the second register device, and the converter device can be configured separately or together, and can be configured in a crossbar device. Each mantissa block can include one or more mantissas, and each of the multiple scale factors is associated with one of the mantissa blocks. The first format and the second format can include a block floating point format (e.g., 36×64 bytes, 65×64 bytes, etc.), and the scale factor can be characterized by a floating point (FP) scale factor (e.g., an unsigned 8-bit FP scale factor with a 4-bit exponent field and a 4-bit fraction field). Further, the converter device can be configured to determine multiple converted mantissa blocks by multiplying each mantissa by its associated scale factor, shifting each scaled mantissa, and rounding each shifted mantissa. Those of ordinary skill in the art will recognize other variations, modifications, and substitutions.

[0013] The compression / decompression techniques described above may also be implemented in one or more core devices coupled to a memory device (e.g., a DDR memory) in an AI accelerator apparatus. In this case, each core device includes a CPU coupled to a plurality of slice devices, and each slice device includes at least a memory device and a computing device. Similar to the previous example, a first register coupled to the CPU is configured to receive a plurality of scale factors and a plurality of mantissa blocks. The converter device coupled to the CPU is configured to determine a plurality of maximum exponents from the scale factors of each column, and to determine a plurality of converted mantissa blocks from the scale factors and mantissa blocks from the memory. A second register device coupled to the CPU stores a plurality of maximum exponents, which are sent together with a plurality of converted mantissa blocks to form a plurality of matrix weight elements in a second format in the memory devices of the plurality of slices. The computing devices of the slices are then configured to determine a plurality of matrix multiplication outputs using these weight matrix elements in the second format. As with the previous example, there may be variations, modifications, and substitutions.

[0014] Although the previous examples discuss weight matrix elements, the present compression / decompression implementation can also be applied to other matrix elements, such as matrix activations. In this case, the crossbar converter device is coupled to the crossbar device and the IB device, and the decompression method can be applied to multiple activation matrix elements or input matrix elements stored in the memory device. Those skilled in the art will recognize other variations, modifications and alternatives.

[0015] Embodiments of such a matrix computing device can provide a number of benefits. The device can store a large number of matrix elements in a compressed format that can be decompressed upon retrieval to perform matrix calculations. In addition, this compression-decompression capability can be accomplished without the need for completely separate hardware and computational paths. Further, these advantages can be implemented in IC chips and chiplet devices with minimal additional silicon area costs.

[0016] A further understanding of the nature and advantages of the present application may be realized by reference to the following portions of the specification and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] For a more complete understanding of the present application, reference is made to the accompanying drawings. Understanding that these drawings are not to be considered limiting of the scope of the present application, additional details are provided through the use of the accompanying drawings to describe the presently described embodiments and the best mode of the present application currently understood.

[0018] Figure 1A to Figure 1B is a simplified block diagram illustrating an AI accelerator device according to an example of the present application.

[0019] FIG. 2A to FIG. 2B is a simplified block diagram illustrating a 16-slice die device according to an example of the present application.

[0020] FIG. 3A to FIG. 3B is a simplified block diagram illustrating a slicing device according to an example of the present application.

[0021] Figure 4 is a simplified block diagram illustrating an in-memory computing (IMC) module according to an example of the present application.

[0022] Figure 5A is a simplified flow chart illustrating the numerical format of data processed in a slicing device according to examples of the present application.

[0023] Figure 5B is a simplified diagram illustrating an example numerical format.

[0024] Figure 6 is a simplified block diagram of the transformer architecture.

[0025] Fig. 7A is a simplified block diagram illustrating a column blocking converter apparatus according to an example of the present application.

[0026] Figure 7B is a simplified block diagram illustrating a column blocking converter apparatus according to an example of the present application.

[0027] Fig. 8A is a simplified flow chart illustrating a method of operating a column partitioning apparatus according to an example of the present application.

[0028] Figure 8B is a simplified flow chart illustrating a method of operating a column partitioning apparatus according to an example of the present application.

[0029] Fig. 9 is a simplified block flow diagram illustrating a mapping process between a transformer and an AI accelerator device according to an example of the present application.

[0030] Fig. 10Ais a simplified diagram illustrating a matrix computing device according to an example of the present application.

[0031] Fig. 10B is a simplified diagram illustrating a method of operating a matrix computing device according to an example of the present application.

[0032] Fig.11A is a simplified diagram illustrating a matrix computing device according to an example of the present application.

[0033] Fig. 11B is a simplified diagram illustrating a matrix computing device according to an example of the present application.

[0034] Fig. 11C is a simplified diagram illustrating a data format according to an example of the present application. DETAILED DESCRIPTION

[0035] The present application generally relates to integrated circuit (IC) devices and artificial intelligence (AI) systems. More specifically, the present application relates to methods and devices for accelerating computational workloads in transformer-based neural network models (also known as transformers). These methods and structures can be used for machine / deep learning applications such as natural language processing (NLP), computer vision (CV), etc. By way of example only, the present application has been applied to AI accelerator devices and core-grain devices configured to perform high-throughput operations of NLP.

[0036] At present, the vast majority of NLP models are based on transformer models, such as the Bidirectional Encoder Representations from Transformers (BERT) model, the BERT Large model, and the Generative Pre-trained Transformer (GPT) model, such as GPT-2 and GPT-3. However, these transformers have very high requirements for computing and memory. According to an example, the present application provides an apparatus using a core particle device, which is configured to accelerate transformer calculations for AI applications. Figure 1A and Figure 1B An example of an AI accelerator device is shown.

[0037] Figure 1AA simplified AI accelerator apparatus 101 with two chip devices 110 is illustrated. As shown, the chip devices 110 are coupled to each other via one or more chip-to-chip (D2D) interconnects 120. In addition, each chip device 110 is coupled to a memory interface 130 (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic RAM (SDRAM), etc.). The apparatus 101 also includes a substrate member 140 that provides mechanical support for the chip devices 110 configured on a surface area of ​​the substrate member 140. The substrate may include an interposer, such as a silicon interposer, a glass interposer, an organic interposer, etc. The chip can be coupled to one or more interposers that can be configured to enable communication between the chip and other components (e.g., acting as a bridge or conduit to allow electrical signals to pass between internal and external components).

[0038] Figure 1B A simplified AI accelerator apparatus 102 is illustrated in which eight chiplet devices 110 are configured in two groups of four chiplets on a substrate member 140. Here, each chiplet device 110 in a group is coupled to other chiplet devices via one or more D2D interconnects 120. The apparatus 102 also shows a DRAM memory interface 130 coupled to each chiplet device 110. The DRAM memory interface 130 may be coupled to one or more memory modules, represented by "Mem" blocks.

[0039] As shown, AI accelerator devices 101 and 102 are embodied in a peripheral component interconnect express (PCIe) card form factor, but the AI ​​accelerator device can also be configured in other form factors. These PCIe card form factors can be configured in various sizes (e.g., full height, full length (FHFL); half height, half length (HHHL), etc.) and mechanical sizes (e.g., 1×, 2×, 4×, 16×, etc.). In the example, one or more substrate components 140 are coupled to the PCIe card, each substrate component 140 having one or more core particles. Those of ordinary skill in the art will recognize other variations, modifications, and alternatives to these elements and configurations of the AI ​​accelerator device.

[0040] Embodiments of AI accelerator devices may implement several techniques to improve performance (e.g., computational efficiency) in various AI applications. AI accelerator devices may include digital in-memory computing (DIMC) to integrate computing functions and memory structures. Mappers, numerical and sparsity algorithms may be optimized in the computing structure. Moreover, the use of core particles and interconnects configured on an organic interposer may provide modularity and scalability.

[0041] According to an example, the present application implements a core particle with in-memory computing (IMC) capabilities that can be used to accelerate the calculations required for the workload of the transformer. The calculations used to train these models can include performing a scaled dot product attention function to determine the probability distribution associated with the desired outcome in a specific AI application. In the case of training an NLP model, the desired outcome can include predicting subsequent words, determining contextual word meaning, translating into another language, etc.

[0042] The coregrain architecture may include multiple slice devices (or slices) controlled by a central processing unit (CPU) to perform transformer calculations in parallel. Each slice is a modular IC device that can handle a portion of these calculations. The multiple slices may be divided into tiles / groups (i.e., subsets) of one or more slices, with a CPU coupled to each slice in a tile. Such tile CPUs may be configured to perform transformer calculations in parallel by each slice in the tile. A global CPU may be coupled to each of these tile CPUs to be configured to perform transformer calculations in parallel by all slices in one or more coregrains using the tile CPUs. Further details of the coregrains are described in detail in the accompanying drawings. FIG. 2A to FIG. 5B for discussion, and the transformer is referenced Figures 6 to 9 for discussion.

[0043] Figure 2A 2 is a simplified block diagram illustrating an example configuration of a 16-slice device 201. In this case, the core 201 includes four tile devices 210, each of which includes four slice devices 220, a CPU 221, and a hardware scheduling (HW DS) device 222. In a specific example, the tiles 210 are arranged in a symmetrical manner. As previously described, the CPU 221 of the tile 210 can coordinate operations performed by all slices within the tile. The HW DS 222 is coupled to the CPU 221 and can be configured to coordinate control of the slices 220 in the tile 210 (e.g., to determine which slice in the tile processes the target portion of the transformer calculation). In a specific example, the CPU 221 can be a reduced instruction set computer (RISC) CPU, etc. Further, the CPU 221 can be coupled to a scheduling engine configured to coordinate control of the CPU 221 (e.g., to determine which parts of the transformer calculation are processed by a specific CPU).

[0044] The CPU 221 of each tile 210 can be coupled to a global CPU via a global CPU interface 230 (e.g., a bus, a connector, a socket, etc.). Such a global CPU can be configured to coordinate the AI ​​accelerator devices (e.g., Figure 1A and Figure 1BThe core grain 201 also includes a D2D interconnect 240 and a memory interface 250, both of which are coupled to each CPU 221 in each tile. In an example, the D2D interconnect can be configured with single-ended signaling. The memory interface 250 may include one or more memory buses coupled to one or more memory devices (e.g., DRAM, SRAM, SDRAM, etc.).

[0045] Further, the core 201 includes a PCIe interface / bus 260 coupled to each CPU 221 in each tile. The PCIe interface 260 can be configured to communicate with a server or other communication system. In the case of multiple core devices, the main bus device is coupled to the PCIe bus 260 of each core device by using a main core device (for example, the main bus device is also coupled to the main core device). Such a main core device is coupled to each other core device by using at least a D2D interconnect 240. The main core device and the main bus device can be configured on a substrate member (for example, the same substrate as the core or a separate substrate). The device integrating one or more cores can also be coupled to a power source (for example, configured on-chip, configured in the system, or coupled externally), and can be configured and can operate a server, a network switch, or a host system by using a main bus device. The server device can also be one of a plurality of server devices configured for a server cluster or other similar configuration within a data center.

[0046] In a specific example, an AI accelerator device configured for GPT-3 may incorporate eight cores (with Figure 1B The chiplet can be configured with a D2D 16×16Gb / s interconnect, a 32-bit LPDDR5 6.4Gb / s memory module, and a 16-lane PCIe Gen 5 PHY NRZ 32Gb / s / lane interface. LPDDR5 (16×16Gb) can provide the necessary capacity, bandwidth, and low power for large-scale NLP models (e.g., quantized GPT-3). Of course, there can be other variations, modifications, and substitutions.

[0047] Figure 2Bis a simplified block diagram illustrating an example configuration of a 16-slice core device 202. Similar to core 201, core 202 includes four groups 210 (or tiles), each group including four slice devices 220 and a CPU 221. As shown, the CPU 221 of each group / tile 210 is coupled to each slice 220 and each other CPU 221 in the other groups / tiles 210. In the example, the tiles / groups act as neural cores and the slices act as compute cores. Using this multi-core configuration, the core device can be configured to process and run multiple calculations in parallel. The CPU 221 is also coupled to a global CPU interface 230, a D2D interconnect 240, a memory interface 250, and a PCIe interface 260. As shown in FIG. Figure 2A As described, the global CPU interface 230 is connected to the global CPU that controls all the CPUs 221 of each group 210 .

[0048] Figure 3A is a simplified block diagram of an example slice device 301 illustrating corelets. For the 16-slice corelet example, the slice device 301 includes a compute core 310 having four compute paths 312, each of which includes an input buffer (IB) device 320, a digital in-memory compute (DIMC) device 330, an output buffer (OB) device 340, and a single instruction multiple data (SIMD) device 350, which are coupled together. Each of these paths 312 is coupled to a slice crossbar / controller 360, which is controlled by the tile CPU to coordinate the computations performed by each path 312.

[0049] In an example, the DIMC is coupled to a clock and configured in one or more portions of each of a plurality of slices of a core particle to allow high throughput of one or more matrix calculations provided in the DIMC, such that the high throughput is characterized by 512 multiplication-accumulations per clock cycle. In a specific example, the clock coupled to the DIMC is a second clock derived from a first clock (e.g., a core particle clock generator, an AI accelerator device clock generator, etc.), the second clock being configured to output a clock signal of approximately 0.5 GHz to 4 GHz; the second clock may be configured to have an output rate of approximately half the rate of the first clock. The DIMC may also be configured to support block structure sparsity (e.g., to impose structural constraints on weight patterns of a neural network such as a transformer).

[0050] In an example, SIMD device 350 is a SIMD processor coupled to the output of DIMC. SIMD 350 can be configured to process one or more non-linear operations and one or more linear operations on a vector process. SIMD 350 can be a programmable vector unit, etc. SIMD 350 can also include one or more random access memory (RAM) modules, such as a data RAM module, an instruction RAM module, etc.

[0051] In an example, a slice controller 360 is coupled to all blocks in each compute path 312 and also includes a control / status register (CSR) 362 coupled to each compute path. The slice controller 360 is also coupled to a memory bank 370 and a data reorganization engine (DRE) 380. The slice controller 360 can be configured to feed data from the memory bank 370 to the blocks in each compute path 312 and coordinate these compute paths 312 through a processor interface (PIF) 364. In a specific example, the PIF 364 is coupled to the SIMD 350 of each compute path 312.

[0052] Figure 3B Further details of the computation core 310 are shown. The simplified block diagram of the slice device 302 includes an input buffer 320, a DIMC matrix vector unit 330, an output buffer 340, a network-on-chip (NoC) device 342, and a SIMD vector unit 350. The DIMC unit 330 includes a plurality of in-memory computation (IMC) modules 332 configured to compute a scaled dot product attention function on the input data to determine a probability distribution, which requires a high throughput matrix multiply-accumulate operation.

[0053] These IMC modules 332 may also be coupled to the block floating point alignment module 334 and the partial product reduction module 336 for further processing before outputting the DIMC results to the output buffer 540. In the example, the input buffer 320 receives the data from the memory bank 370 (eg, Figure 3A The IMC module 332 may receive input data (eg, data vectors) of the memory bank 370 and send the data to the IMC module 332. The IMC module 332 may also receive instructions from the memory bank 370.

[0054] In addition to the details discussed above, SIMD 350 can also be configured as an element-pointing vector unit. SIMD 350 can include a computation unit 352 (e.g., add, subtract, multiply, maximum, etc.), a lookup table (LUT) 354, and a state machine (SM) module 356 configured to receive one or more outputs from output buffer 340.

[0055] The NoC device 342 is coupled to the output buffer 340 configured in the feed-forward loop via a shortcut connection 344. In addition, the NoC device 342 is coupled to each slice and is configured for multicast and unicast processes. More specifically, the NoC device 342 can be configured to connect all slices and all tiles, connect multicast input activations to all slices / tiles, and collect partial computations that are unicast for accumulation of a particular distribution.

[0056] Considering the previous example of an 8-chip AI accelerator device, the input buffer may have a capacity of 64KB with 16 groups, and the output buffer may have a capacity of 128KB with 16 groups. The DIMC may be an 8-bit block with size 64×64 (eight 64×64 IMC modules), and the NoC may have a size of 512 bits. The computation blocks in the SIMD may be configured to perform 8-bit and 32-bit integer (int) and unsigned integer (uint) computations as well as floating point computations such as IEEE854float16 or float32. These slice components may vary depending on the transformer that the AI ​​accelerator device will serve.

[0057] Figure 4 is a simplified block diagram illustrating an example IMC module 700. As shown, the module 700 includes one or more computation tree blocks 410 configured to perform desired computations on input data from one or more read-write blocks 420. Each of these read-write blocks 420 includes one or more first memory selection units 422 (also denoted as "W"), one or more second memory selection units 424 ("also denoted as I"), an activation multiplexer 426, and an operator unit 428. The first memory selection unit 422 provides input to the operator unit 428, while the second memory selection unit 424 controls the activation multiplexer 426, which is also coupled to the operator unit 428. In a multiply-accumulate operation, the operator unit 428 is a multiplier unit, and the computation tree block 410 is a multiplier adder tree block (i.e., ∑xw).

[0058] As shown in close-up 401, each of the memory selection units 422, 424 includes a memory cell 430 (e.g., an SRAM cell, etc.) and a selection multiplexer 432. Each of the memory selection units 422, 424 is coupled to a read-write controller 440, which is also coupled to a memory bank / driver block 442. In an example, the read-write controller 440 can be configured with a column write driver and a column read sense amplifier, while the memory bank / driver block 432 can be configured with a sequential row select driver.

[0059] An input activation controller 450 may be coupled to the activation multiplexer 426 of each read / write block 420. The input activation controller 450 may include precision and sparsity aware input activation registers and drivers. An operator unit 428 receives the output of the first memory selection unit 422 and receives the output of the block 450 through an activation multiplexer 426, which is controlled by the output of the second memory selection unit 424. The output of the operator unit 428 is then fed into the computation tree block 410.

[0060] The input activation block 450 is also coupled to a clock source / generator 460. As previously described, the clock generator 460 can generate a second clock derived from a first clock configured to output a clock signal of approximately 0.5 GHz to 4 GHz; the second clock can be configured at an output rate of approximately half the rate of the first clock. The clock generator 460 is coupled to one or more sign and precision aware accumulators 470 configured to receive the output of the computation tree block 410. In an example, the accumulator 470 is configured to receive the output of two computation tree blocks 410.

[0061] Referring back to the 8-core AI accelerator device example, the memory cells may be dual-group 2×6 TSRAM cells, and the selection multiplexer may be an 8T group selection multiplexer. In this case, the memory group / driver block 442 includes dual-group SRAM groups. In addition, the read / write controller may include a 64-byte write driver and a 64-byte read sense amplifier. A person of ordinary skill in the art will recognize other variations, modifications, and alternatives to these IMC module components and their configurations.

[0062] Figure 5A is a simplified block flow diagram illustrating an example numerical format of data processed in a slice. Figure 5A 501 in FIG. 5 shows a loop with data formats of GM / input buffer 510, IMC 520, output buffer 530, SIMD 540, and NoC 550, which feeds back to GM / input buffer 510. IMC block 520 shows a multiply-accumulate operation (∑xw). Additionally, the format of data from IMC 532 also flows to output buffer 530. In this example, the numeric formats include integers (int), floating point (float), and block floating point (BFP) of different lengths.

[0063] Figure 5B The diagram includes Figure 5ASimplified diagram of some formats of some numeric formats shown. Block floating point numerics can be used to address certain performance barriers. Transformers are typically trained in floating point, either 32-bit or 16-bit, and inference is typically performed using 8-bit integers ("int8"). When using block floating point arithmetic, a group of mantissas share a common exponent (see Figure 5B The bottom int8 vector has a diagonal padding block), while for floating point operations, each mantissa has a separate exponent (see Figure 5A 32-bit floating point operations and 16-bit floating point operation formats at the top). Methods using block floating point numeric formats for inference can exhibit the efficiency of fixed point without the precision and deployment issues of integer operations, and can also use smaller mantissas, such as 4-bit integers ("int4") while maintaining precision. In addition, by using block floating point formats (e.g., for activations, weights, etc.) and sparsity, inference of trained models can be accelerated, resulting in better performance. Those of ordinary skill in the art will recognize other variations, modifications, and alternatives to these numeric formats for processing transformer workloads.

[0064] Figure 6 A simplified transformer architecture 600 is illustrated. A typical transformer can be described as having an encoder stack configured with a decoder stack, and each such stack can have one or more layers. In the encoder layer 610, a self-attention layer 612 determines contextual information while encoding the data, and feeds the encoded data to a feedforward neural network 616. The encoder layer 610 processes the input sequence from bottom to top, converting the output into a set of attention vectors K and V. The decoder layer 620 also includes a corresponding self-attention layer 622 and a feedforward neural network 626, and further includes an encoder-decoder attention layer 624, which uses the attention vectors from the encoder stack to assist the decoder in further context processing. The decoder stack outputs a floating point vector (such as Figure 5B ), which is fed to a linear and softmax layer 630 to project the output as the final desired result (e.g., desired word prediction, interpretation, or translation). The linear layer is a fully connected neural network that projects the decoder output vector into a larger vector (i.e., a logits vector) containing scores associated with all potential results (e.g., all potential words), and the softmax layer turns these scores into probabilities. Based on the probabilistic output, the projected word meaning can be selected based on the highest probability or according to other derived criteria applied.

[0065] Transformer model variations include variations based only on the decoder stack (e.g., transformer language models, such as GPT-2, GPT-3, etc.) and variations based only on the encoder stack (e.g., masked language models, such as BERT, BERT Large, etc.). The transformer is based on four parameters: sequence length (S) (i.e., number of tokens), number of attention heads (A), number of layers (L), and embedding length (H). Variations on these parameters are now used to build almost all transformer-based models. Embodiments of the present application can be configured for any similar model type.

[0066] Transformers start out untrained and are pre-trained by exposure to a desired dataset of the desired learning application. Transformer-based language models are exposed to a large amount of text (e.g., Wikipedia) to train language processing functions such as predicting the next word in a text sequence, translating text into another language, etc. This training process involves converting text (e.g., words or parts of words) into token IDs, evaluating the context of the tokens through a self-attention layer, and predicting the outcome through a feed-forward neural network.

[0067] The self-attention process involves (1) determining the query (Q), key (K), and value (V) embeddings for each word in the input sentence, (2) computing a score from the dot product Q and K of each word in the input sentence with the target word, (3) dividing the score by the square root of the dimension K, (4) passing the result through a softmax operation to normalize the score, (5) multiplying each V by the softmax score, and (6) summing the weighted V vectors to produce the output. Note that the value matrix V becomes the weight matrix that is matrix-multiplied with the softmax attention matrix; in the context of block floating-point values, this requires a column-blocking transformer for V, as described below.

[0068] Many things can affect the performance of such a transformer architecture. The Softmax function tends to be the critical path of the transformer layer (and is already difficult to accelerate in hardware). The need to overlap computation, SIMD operations, and NoC transfers can also affect performance. Further, the efficiency of NoC, SIMD, and memory bandwidth utilization is also important.

[0069] Different techniques can be used in conjunction with the AI ​​accelerator device and coregrain device examples to improve performance, such as quantization, sparsity, knowledge distillation, efficient tokenization, and software optimization. Supporting variable sequence lengths (i.e., no padding to a maximum sequence length) can also reduce memory requirements. Other techniques can include optimizations on how to split self-attention between slices and chips, moving layers and tensors between slices and chips, and data movement between layers and FC matrices.

[0070] According to an example, the present application provides an AI accelerator device (such as Figure 1A and Figure 1B ), the apparatus is coupled to a transformer device set (e.g., BERT, BERT Large, GPT-2, GPT-3, etc.). In a specific example, such a transformer device set may include a plurality of transformers configured in a stack ranging from three layers to N layers, where N is an integer not exceeding 128.

[0071] In an example, each transformer is configured in one or more DIMCs such that each transformer includes a plurality of matrix multipliers including a QKV matrix configured for the attention layer of the transformer preceding three fully connected matrices (FC). In this configuration, the DIMC is configured as a speed-up transformer and further includes a QK T The dot product, followed by softmax(QK T / square root(d k ))V. In an example, the AI ​​accelerator device also includes a SIMD device (such as Figure 3A and Figure 3B As shown), it is configured to accelerate the calculation process of the softmax function.

[0072] Using transformers (such as BERT Large), NLP requires very high computation (e.g., 5 orders of magnitude higher than CV). For example, BERT Large requires 5.6 giga-multiply-accumulate operations (“GMACs”) per second per transformer layer. Therefore, the NLP inference challenge is to achieve this performance with the lowest energy consumption.

[0073] Although the present application is discussed in the context of BERT Large transformers for NLP applications, those skilled in the art will recognize variations, modifications, and substitutions. The specific embodiments shown may also be applicable to other transformer-based models and other AI / machine learning applications.

[0074] As mentioned earlier, the block floating point (BFP) format is important for efficient hardware acceleration of matrix multiplication operations in deep neural network inference. Matrix weights are typically blocked along columns, while activations are typically blocked along rows. Therefore, BFP values ​​enable efficient integer arithmetic for matrix multiplication while maintaining a large dynamic range. After the matrix multiplication, the activation row vector with the weight column vector is accumulated in floating point (FP) format (e.g., FP32, FP16, etc.) and stored in the output buffer as matrix tiles (e.g., 64×64 tiles for FP16). We can also use BFP32-1, which has a 24-bit mantissa in 2's complement and an 8-bit exponent in 2's complement, as an equivalent format to FP32 for accumulation of partial products.

[0075] Output buffer memory loads / stores are typically implemented row-by-row, which is convenient for the typical case of row-by-row BFP chunking to generate activations for the next matrix multiplication. However, there are cases where the output of a matrix multiplication is used as a weight matrix for a subsequent matrix multiplication (e.g., matrix multiplication of the value matrix with the attention function in the BERT encoder model), which requires storing data from the output buffer in a column-chunked configuration. In these cases, when the memory load / store is characterized by a row-by-row storage configuration, chunking across columns poses a challenge because the output converter can only read one row of data at a time.

[0076] According to an example, the present application provides a column-blocking converter device and method for converting data from a first format in a row-blocking configuration to a second format in a column-blocking configuration. The column-blocking device can be configured as an IC of an AI accelerator device, such as the aforementioned example AI accelerator IC.

[0077] Fig. 7A and Figure 7B is a simplified block diagram illustrating a column-blocking converter device 701 / 702 according to an example of the present application. As shown in the figure, the device 701 / 702 and Figure 3A The slicing device 301 shown is similar. Any shared reference values ​​between these figures refer to the same elements described above. Here, Fig. 7A and Figure 7B Only two computation paths 313 in computation core 310 are shown, however, there may be additional computation paths 312 depending on the application.

[0078] Fig. 7A The apparatus 701 may include a computation path 312 having an input buffer (IB) device 320, a computation device 330, and an output buffer (OB) 340 device. The IB device 320 is coupled to the computation device 330 and is configured to receive a plurality of matrix inputs. The computation device 330 is coupled to the OB device 340 and is configured to perform a plurality of matrix calculations on the plurality of matrix inputs. In a specific example, the computation device 330 may be a digital in-memory computation (DIMC) device 330 that performs a softmax function, as described above. In this case, the OB device 340 may be characterized by a row-by-row storage configuration and may be configured to store a plurality of matrix outputs generated by a plurality of matrix calculations in a first format.

[0079] The OB converter device 710 may be coupled between the computing device 330 and the OB device 340. Such an OB converter device 710 may be configured to store the plurality of matrix outputs in the first format in the OB device 340. As shown, the OB converter device 710 is configured separately from the OB device 340, however, the OB converter device 710 may also be configured in the OB device 340. These configurations may be implemented as an inline column blocking converter arrangement.

[0080] The crossbar device 360 ​​is coupled to the IB device 320, the computing device 330, and the OB device 340. The crossbar converter device 720 is also coupled to the OB device 340 and is configured to convert the plurality of matrix outputs from a first format to a second format using the maximum exponent value and the mantissa value determined for each of the plurality of matrix outputs, thereby generating a plurality of converted matrix outputs. As shown, the crossbar converter device 720 is configured in the computing path 312, however, the crossbar converter device 720 may also be configured in the crossbar device 360.

[0081] Further, a memory device 370 is coupled to the crossbar device 360. This memory device is configured to use the maximum exponent value and the mantissa value to store multiple converted matrix outputs in a second format and a column-blocked configuration. The first format can be a floating point (FP) format, and the second format can be a block floating point (BFP) format. In a specific example, the first format is an FP16 format, and the second format is a BFP format (BFP16-64 format) with a block size of 64 elements, a mantissa bit width of 8 bits, and a shared exponent of 8 bits. In this case, multiple matrix outputs can be characterized by 64×64 byte tiles of mantissas and 64 byte rows of shared exponents. This embodiment of the present application includes an efficient algorithm and hardware architecture for converting FP16 tiles of 64×64 elements stored in an output buffer into a column-blocked BFP16-64 tile.

[0082] In an example, the crossbar converter device 720 includes a maximum exponent register 722 configured to store a maximum exponent value for each of the plurality of matrix outputs. The OB device 340 and the converter device may be configured together to determine a maximum exponent value for each of the plurality of matrix outputs in a first row-by-row process, determine a mantissa value for each of the plurality of matrix outputs in a second row-by-row process, and store the maximum exponent value and the mantissa value in a memory device. The maximum exponent register 722 may be used to store the maximum exponent value in the first row-by-row process.

[0083] In a specific example, the crossbar converter device 720 is configured to perform a shift process and a rounding process on the mantissa value of each of the plurality of matrix outputs in the second row-by-row process. The crossbar device 360 ​​can be configured to write the mantissa value to the memory device 370 after each row in the second row-by-row process. Further, the crossbar device 360 ​​can be configured to write the maximum exponent value to the memory device 370 after the second row-by-row process. The crossbar converter device 720 can be coupled to the OB device 340 in a feedback configuration to perform the first row-by-row process and the second row-by-row process.

[0084] and Fig. 7A compared to, Figure 7B An alternative device architecture of a column blocking converter apparatus 702 is shown, wherein the OB converter device 710 also includes a maximum exponent register 712 coupled to a crossbar converter device 720. Instead of the column blocking converter device 720, the OB converter device 710 may be configured to determine a maximum exponent value for each of a plurality of matrix outputs in a first row-by-row process and store the maximum exponent value in the first maximum exponent register 712. The crossbar converter device 720 may then be configured to store the maximum exponent value from the first maximum exponent register 712 in its second maximum exponent register 722 and determine mantissa values ​​for each of a plurality of matrix outputs from the OB device in a second row-by-row process.

[0085] Similar to the first architecture, the crossbar converter device 720 can be configured to perform a shift process and a rounding process on the mantissa value of each of the plurality of matrix outputs. Also, the crossbar device 360 ​​can be configured to write the maximum exponent value to the memory device 370 after the second row-by-row process. Further details of the processes performed by the OB converter device 710 and the crossbar converter device 720 are described in detail in Fig. 8A and Figure 8B Have a discussion.

[0086] Fig. 8A is a simplified flow chart illustrating a method 801 of operating a column-blocking converter device according to an example of the present application. This method corresponds to Fig. 7AThe apparatus 701 shown, wherein the crossbar converter device 720 is configured to perform a first row-by-row process and a second row-by-row process. As shown, method 801 first receives a plurality of matrix outputs (tiles of N×M matrix outputs, where N and M are integers) at the OB converter 710. In an example, these matrix outputs are the result of a matrix multiplication (e.g., for a softmax function) (each row is represented by "Data1" to "DataN"), and the OB converter 710 converts the result to a first format (represented by "D1-F1" to "DN-F1") and writes it to the OB device / group 340 (represented by "DN,M-F1"). Considering a 64×64-byte example, each of the 64 elements in a row is in the BFP32-1 format, and the OB converter 710 converts this format to the FP16 format.

[0087] Here, the crossbar converter device 720 reads the OB group 340 to perform the first row-by-row process and the second row-by-row process to respectively determine the maximum exponent value and the mantissa value. The crossbar converter device 720 reads each row of the data stored in the OB group 340 one row at a time to determine the maximum exponent value of each entry and update the maximum exponent register 722 (e.g., if exp i <reg_exp[i], then reg_exp[i]=exp i ). In a 64×64-byte example, the converter device 720 reads one row of 64 FP16 elements at a time. After all rows have been processed, the maximum exponent register 722 includes the maximum exponent of each column of the data tile stored in the OB group 340 (represented by "Exp1" to "ExpM").

[0088] Then, the converter device 720 reads each row again from the OB group 340 in the second row-by-row process to determine the mantissa value. For each entry in the OB group entry, the converter device 720 can perform a shift process and a rounding process to convert the mantissa value to a desired format (e.g., integer format or other numerical format). In a 64×64-byte example, the shift process and the rounding process can result in converting the mantissa value to an 8-bit integer (int8) format. After processing one row of mantissas (represented by "Man1" to "ManN"), the processed data is sent to be written to the memory device 370. Once all rows have been processed, the conversion of the mantissa to the second format (represented by "DN,M-F2") in the column chunking configuration is complete. Regarding the maximum exponent register data sent later, the memory device 370 will contain consecutive data blocks, where each column is in the second format. In a 64×64-byte matrix data example, the consecutive blocks are characterized by 65×64 bytes, and each column is in the BFP16-64 format.

[0089] Figure 8Bis a simplified flowchart illustrating a method 802 of operating a column-blocking converter device according to an example of the present application. This method corresponds to Figure 7B the illustrated device 702, where the OB converter device 710 is configured to perform a first row-by-row process using its own maximum exponent register 712, and the bar converter device 720 is configured to perform a second row-by-row process. Using the same representation as method 801, method 802 first receives a plurality of matrix outputs (tiles of matrix outputs) at the OB converter 710. In the example, the OB converter device 710 converts the output to a first format. After converting each row of data to the first format, the OB converter device 710 also determines the maximum exponent value of each entry and updates the maximum exponent register 712 (e.g., if exp i <reg_exp[i], then reg_exp[i] = exp i ). After all rows of the output have been processed by the OB converter device 710, the maximum exponent register 712 contains the maximum exponent of each column of the tile. Considering the 64×64 byte example, each of the 64 elements in a row is in the BFP32-1 format (32-bit floating-point format), and the OB converter 710 converts this format to the FP16 format (16-bit floating-point format).

[0090] Then, the bar converter device 720 reads the maximum exponent data from the OB converter register 712 into its own maximum exponent register 722. Similar to method 801, the bar converter device 720 reads each row from the OB group 340 during the second row-by-row process to determine the tail values. The converter device 720 also performs a shift process and a rounding process to convert the tail values to the desired format (e.g., integer format or other numerical formats). After processing a row of mantissas, the processed data is sent to be written to the memory device 370. Once all rows have been processed, the conversion of the mantissas to the second format in the column-blocking configuration is complete. Regarding the maximum exponent register data sent later, the memory device 370 will contain a contiguous block of data, where each column is in the second format. In the 64×64 byte matrix data example, the contiguous block is characterized by 65×64 bytes, and each column is in the BFP16-64 format.

[0091] Although these examples are discussed with respect to FP and BFP numerical formats, the column block converter apparatus and methods thereof can be applied to convert data from any first format to any second format that can be determined by corresponding exponent and mantissa values. There are also variations in calculating the shared block index, for example, instead of the maximum index, percentile values ​​can be used. In addition, in the case where the buffer memory load / store is implemented column by column, the same techniques described herein can be used to convert the column by column storage configuration to the row by row storage configuration. Those of ordinary skill in the art will recognize other variations, modifications and substitutions of these block conversion methods and structures.

[0092] Fig. 9 is a simplified block flow diagram illustrating the mapping process between a transformer and an example AI accelerator device. As shown, the transformer 901 includes multiple transformer layers 910, each of which has an attention layer 902. In this case, there are 16 attention heads 920 (e.g., BERT Large) that calculate the aforementioned attention function. These 16 attention heads are mapped to 16 slices 930 of the AI ​​accelerator device 903 (similar to devices 201 and 202) through the global CPU 932 to be passed to the tile CPU 934.

[0093] According to an example, the present application provides a method and apparatus for data conversion in a matrix computing device. In a specific example, the matrix computing device can be configured as a multiplication and accumulation (MAC) unit, which acts as a key building block of dot product and matrix multiplication hardware for accelerating deep neural network reasoning applications, including the aforementioned NLP workloads. In such applications, more than one data format may need to be processed. For example, efficient MAC implementations are typically based on integer algorithms that support fixed point or block floating point (BFP) numerical formats. However, in some applications, a MAC unit or other matrix computing device is required to have the ability to process floating point (FP) or brain floating point (Bfloat) numerical formats.

[0094] Therefore, the present application provides a method and apparatus capable of implementing a matrix computing device, which is configured to process matrix data in a target format by segmenting the data and processing the segmented data portions in parallel in a native format in the matrix computing device. As an example only, the present application discusses a native format as an 8-bit integer (int8) format and a target format as a 16-bit floating point (FP16) format. An embodiment of the present matrix computing device can be configured as an IC of an AI accelerator IC, such as the aforementioned AI accelerator system. Referring to Fig. 10A and Fig. 10B Discuss further details.

[0095] Fig. 10A1 is a simplified diagram illustrating a matrix computing device 1001 according to an example of the present application. As shown in the figure, similar to Figure 3B In the example slice device 302 of FIG. 1 , the apparatus 1001 may be configured with an input buffer (IB) 1010 device, a computing device 1020 (e.g., a DIMC device) coupled to the IB device 1010, and an output buffer (OB) device 1030 coupled to the computing device 1020. In addition, a single instruction multiple data (SIMD) device 1040 may be coupled to the OB device 1030. Similar to the slice device 302, such an apparatus 1001 may be configured in a chiplet device (see Figure 2A and Figure 2B ), the chiplet device is an AI accelerator system (see Figure 1A and Figure 1B ) section.

[0096] In an example, an input buffer (IB) device 1010 is configured to receive one or more matrix inputs (eg, from a memory device, etc.). Such an IB device 1010 may be similar to the IB devices shown above (eg, Figure 3A and Figure 3B ) to be configured. Each such matrix input may be characterized by a first format and have at least a first input portion and a second input portion. These input portions are segmented portions of the matrix input to be processed in parallel by the computing device 1020. According to an embodiment, the matrix input may have multiple input portions, including a matrix weight and an activation portion (see Fig. 10B ).

[0097] IB device 1010 may receive a first matrix input or a plurality of matrix inputs in a first format from an input converter device configured to convert (a plurality of) matrix inputs into a first format. Such an input converter device, such as a CPU (e.g., Figure 2B The first format may be a BFP format, a fixed point format, etc. As long as the first format allows for conversion and division of matrix data in the original format, other formats may also be used.

[0098] As an example only, the matrix computing device can be configured to perform matrix calculations in an integer numerical format. In such a case, the computing device can be configured to process matrix inputs in a data portion that can conform to an integer format. For example, each of a plurality of computing units can be configured to perform matrix calculations in an int8 format, and the matrix input can be in an FP16 format in a 64×64 byte tile configuration. In this case, an input converter device (e.g., a tile CPU, an embedded input converter 1012, etc.) converts the FP16 matrix input to a 24-bit block floating point (BFP24) format with a 16-bit mantissa and an 8-bit exponent. The mantissa can then be divided into two 8-bit parts, the most significant byte (MSB) part and the least significant byte (LSB) part, to be processed in parallel by the computing device 1020.

[0099] In an example, the computing device 1020 includes a plurality of computing units 1022, at least having a first computing unit 1022 and a second computing unit 1022. The pair of computing units may be configured to perform matrix calculations on matrix inputs in a non-native format. More specifically, the first computing unit 1022 may be configured to determine a first matrix output using at least a first input portion, and the second computing unit 1022 may be configured to determine a second matrix output using at least a second input portion. The computing device 1020 may then be configured to determine a combined matrix output in a second format using the first matrix output and the second matrix output. In a specific example, the computing device 1020 determines the combined matrix output by shifting the first matrix output and adding the shifted first matrix output to the second matrix output.

[0100] In the example, each of the matrix inputs includes a matrix weight and a matrix activation. Each matrix weight input may include a matrix weight exponent and a matrix weight mantissa. Referring back to the FP16 example, the matrix weight exponent includes 8 bits and the matrix weight mantissa includes 16 bits, which may be divided into an 8-bit MSB portion and an 8-bit LSB portion. Similarly, the matrix activation exponent also includes 8 bits and the matrix activation mantissa also includes 16 bits, which may be divided into an 8-bit MSB portion and an 8-bit LSB portion. In this case, the computing device determines the first matrix output by performing a dot product process using the MSB portion of the matrix activation and the matrix weight mantissa. Similarly, the computing device determines the second matrix output by performing a dot product process using the LSB portion of the matrix activation and the matrix weight mantissa.

[0101] Although the example above has only discussed that matrix input data is divided into two parts, other examples can be divided into multiple data parts by multiple computing unit parallel processing.In such case, computing device 1020 will use similar shift and add process to determine multiple matrix outputs, to combine these matrix outputs into combined matrix output, wherein, each data part is all located in appropriate order.These parts also can be stored in the segmented part that matches with the native format of computing device.Those of ordinary skill in the art will recognize other changes, modifications and substitutions to the selection of data format and data segmentation.

[0102] Considering the FP16 example, the first input portion is the MSB weight portion, and the second input portion is the LSB weight portion. The first calculation unit 1022 is configured to determine the first matrix output using the MSB portion, and the second calculation unit 1022 is configured to determine the second matrix output using the LSB portion. By shifting the MSB portion by 8 bits and adding the LSB portion, the matrix output is combined, as shown in FIG. Fig. 10B The resulting combined matrix output has a 38-bit mantissa (for a 64×64 matrix) and an 8-bit exponent, which can be represented in BFP46-1 format.

[0103] In an example, the computing device includes an alignment device 1024 coupled to a plurality of computing units 1022. The alignment device 1024 may be configured to determine a rounding matrix output of a third format using the combined output. This rounding process may be used to prepare a matrix output for a subsequent partial product reduction (PPR) process. In the FP16 example, the combined matrix output of the BFP46-1 format may be rounded to a matrix output of the BFP32-1 format. In another example, the BFP46-1 combined matrix output may be converted to a FP32 matrix output by the alignment device 1024 or a data converter coupled to the alignment device 1024.

[0104] In an example, the PPR device 1026 is coupled to the alignment device 1024. The PPR device 1026 can be configured to determine the compact matrix output using the rounded matrix output. The PPR process can be used to prepare the matrix output for subsequent conversion to the original data format (e.g., FP16) for storage in the OB device 1030.

[0105] In the example, the computing device 1020 also includes a computing converter 1028 configured to determine a first conversion matrix output of a conversion output format using the previous matrix output. In the FP16 example, the computing converter 1028 converts the compact matrix output in the BFP32-1 format to the FP16 matrix output. In the case of converting the combined matrix output to the FP32 format, the computing converter 1028 converts the compact matrix output in the FP32 format to the FP16 matrix output.

[0106] In an example, the OB device 1030 is configured to store the resulting conversion matrix output. Such an OB device 1030 may be similar to the OB devices shown above (e.g., Figure 3A and Figure 3B ) is configured. As discussed for IB device 1010 in the FP16 example, OB device 1030 may be configured to store the matrix output in a 64×64 byte tile configuration. Additional details of the matrix data conversion and calculation process are discussed with reference to FIG. 10 .

[0107] Embodiments of such matrix computing devices and related methods thereof can provide many benefits. The present method and device can perform computational processing on matrix inputs of different data formats that can be segmented into portions compatible with native formats. In addition, such multi-format capabilities can be accomplished without requiring completely independent hardware and computational paths. Further, these benefits can be implemented in IC chips and core-grain devices with minimal cost increase in silicon area.

[0108] Fig. 10B 1 is a simplified diagram illustrating a data format conversion method using data segmentation and parallel processing in a matrix computing apparatus 1002 according to an example of the present application. As shown in the figure, the apparatus 1002 includes an IB device 1010 and a computing device 1020, the computing device 1020 having a plurality of computing units 1022 numbered 0 to N, an alignment device 1024, a PPR device 1026, and a computing converter 1028.

[0109] As discussed in the previous examples, each matrix input can include a matrix weight and a matrix activation. Each matrix weight input can include a matrix weight exponent and a matrix weight mantissa. Referring back to the FP16 example, the matrix weight exponent includes 8 bits and the matrix weight mantissa includes 16 bits, which can be divided into an 8-bit MSB portion and an 8-bit LSB portion. In this case, the matrix activation exponent also includes 8 bits and the matrix activation mantissa also includes 16 bits, which can be divided into an 8-bit MSB portion and an 8-bit LSB portion.

[0110] In this case, the first portion (e.g., MSB) of the matrix weights is stored in the first computing unit 1022-0 (shown as IMC0), and the second portion (e.g., LSB) of the matrix weights is stored in the second computing unit 1022-4 (shown as IMC4). The computing device 1020 uses the first portion of the matrix activations and the matrix weights to determine the first matrix output by performing a dot product process, and uses the second portion of the matrix activations and the matrix weights to determine the second matrix output by performing a dot product process. The first matrix output is then shifted (by 8 bits in the FP16 example) and added to the second matrix output to determine the combined matrix output.

[0111] Subsequently, the alignment device 1024 may determine the rounding matrix output from the combined matrix output, and the PPR device 1026 may determine the compaction matrix output from the rounding matrix output. Further, the computational converter 1028 may determine the conversion matrix output from the compaction matrix output. The flow chart of the matrix output is in Fig. 10B 1020 are shown within the dashed lines relative to the components of computing device 1020.

[0112] As mentioned above, other examples can divide data into multiple data parts, which are processed in parallel by multiple computing units. In such a case, computing device 1020 will use a similar shift and add process to determine multiple matrix outputs to combine these matrix outputs into a combined matrix output, wherein each data part is positioned in an appropriate order. These parts can also be stored in a segmented part that matches the native format of a computing device (e.g., an int8 computing unit configured to process FP16 matrix inputs). Further, the steps of processing matrix outputs and their respective hardware components can be added, deleted or rearranged according to the application. Those of ordinary skill in the art will recognize other changes, modifications and substitutions to the selection of data format and data segmentation.

[0113] According to an example, the present application provides a method and apparatus for data compression and decompression in a matrix computing device. In a specific example, the matrix computing device can be configured as a matrix multiplication computing device (e.g., a MAC unit) configured to accelerate deep neural network reasoning applications, including the aforementioned NLP workloads. In such applications, it is necessary to improve the handling of large data sizes. For example, transformer-based modeling networks typically involve a large number of elements (e.g., weights, activations, etc.), which cannot all be stored in on-chip memory. Therefore, access to these elements requires frequent transmission from a memory storage device (e.g., DDR), which may cause the processing of these elements to become memory-constrained due to the large delay of such memory operations.

[0114] Therefore, the present application provides a method and apparatus capable of implementing a matrix computing device, which is configured to process matrix data stored in a compressed format and decompress the data for matrix calculation. By way of example only, the present application discusses data blocks in a 36×64 byte compressed block floating point (BFP) format (expressed as SBFP-12-16), which are decompressed into a 65×64 byte BFP format (expressed as BFP16-64). An embodiment of the present matrix computing device can be configured as an IC of an AI accelerator IC, such as the aforementioned AI accelerator system. Referring to Fig.11A and Fig. 11B Discuss further details.

[0115] Fig.11A 1 is a simplified diagram illustrating a matrix multiplication computing device 1101 according to an example of the present application. As shown in the figure, such a device can be similar to Figure 3A The exemplary slicing device 301 is configured (see the previous description of the same reference numerals as those denoted by the same reference numerals). Figure 3A 101). In contrast, the apparatus 1101 includes a crossbar converter device 1110 coupled to the crossbar device 360 ​​and a weight buffer (WB) device 1120 coupled to the computing device 330. Here, the converter device 1110 includes at least a first register device 1112 and a second register device 1114. The converter device 1110 may be configured to decompress data from the memory device 370 via the crossbar device 360 ​​and send the data to the WB device 1120 in preparation for processing by the computing device 330.

[0116] In an example, the WB device 1120 may be configured as a buffer device together with the input buffer (IB) device 320. In addition, the crossbar converter device 1110, the first register device 1112, the second register device 1114, and any other register devices may be configured together or separately in each computation path 312. Alternatively, any register of the crossbar converter device 1110 may also be configured in the crossbar device 360 ​​and coupled to each computation path 312. Further details of the method of operating the apparatus are discussed below.

[0117] According to an example, the present application provides a method for operating a matrix multiplication computing device using block compression / decompression. Such a device includes at least a memory device configured to store a plurality of weight matrix elements in a first format, the first format including a plurality of weight matrix columns. Each such column includes a plurality of scale factors and a plurality of mantissa blocks. In this case, the device is similar to Fig.11A The method can be briefly summarized as follows:

[0118] 1. Receiving, by a first register device, a plurality of scaling factors for each weight matrix column from a memory device;

[0119] 2. determining, by the converter device, a maximum index for each weight matrix column using a plurality of scale factors for the weight matrix column, thereby generating a plurality of maximum indices;

[0120] 3. storing the plurality of maximum exponents in a second register device;

[0121] 4. Receiving, by the first register device, a plurality of mantissa blocks for each weight matrix column from the memory device;

[0122] 5. determining, by the converter device, a plurality of conversion mantissa blocks using a plurality of scale factors of a plurality of weight matrix columns and a plurality of mantissa blocks;

[0123] 6. Receiving, by the WB device, a plurality of weight matrix elements in a second format, the second format comprising a plurality of conversion mantissa blocks and a plurality of maximum exponents;

[0124] 7. Determining, by the computing device, a plurality of matrix multiplication outputs using the plurality of weight matrix elements in the second format;

[0125] 8. Storing multiple matrix multiplication outputs by the OB device; and

[0126] 9. Perform other steps as needed.

[0127] The above sequence of steps is used to operate a matrix multiplication computing device configured for an AI accelerator device according to one or more embodiments of the present application. Depending on the embodiment, one or more of these steps may be combined or deleted, or other steps may be added without departing from the scope of the claims herein. A person of ordinary skill in the art will recognize other variations, modifications, and substitutions. Further details of this method are provided throughout this specification and are provided in more detail below.

[0128] In an example, each mantissa block includes one or more mantissa values, and each of the plurality of scale factors is associated with one of the plurality of mantissa blocks. The step of determining a plurality of converted mantissa blocks may include multiplying each mantissa of each mantissa block by the associated scale factor, thereby generating a scaled mantissa. This step may also include shifting each scaled mantissa of each mantissa block, thereby generating a shifted mantissa. Further, this step may include rounding each shifted mantissa of each mantissa block, thereby generating a rounded mantissa.

[0129] In an example, multiple weight matrix elements in a first format can be represented by a 36×64 byte storage configuration (e.g., SBFP12-16 format), and multiple weight matrix elements in a second format can be represented by a 65×64 byte storage configuration (e.g., BFP16-64). In the case of SBFP12-16, multiple weight matrix elements are configured into 64 weight matrix columns so that each weight matrix column includes four 8-byte mantissa blocks and four 1-byte scale factors (36 bytes in total). In a specific example, each of the multiple scale factors (always positive in the SBFP12-16 format) is represented by an unsigned 8-bit floating point (FP8) scale factor, which includes a 4-bit exponent field and a 4-bit fraction field. An optional programmable exponent bias can be used to optimize the scale factor dynamic range. For each FP8 scale factor, there is a corresponding 8-byte mantissa, each byte stores two 4-bit integer (int4) mantissa values ​​(so each FP8 scale factor has a total of 16 4-bit mantissa elements). Therefore, a 65-byte BFP16-64 block can be compressed into four SBFP12-16 blocks, totaling 36 bytes, with a compression ratio of 1.8056.

[0130] Referring again to SBFP12-16, the foregoing method may include reading in 4×64 byte scale factors into a first register device, which requires the first register device to have a capacity of at least 256 bytes. The converter device may then calculate the maximum exponent on the exponent fields of the four scale factors and save the result in a second register device, which requires the second register device to have a capacity of at least 64 bytes. Each 64-byte mantissa block row (each with two 4-bit mantissas) is then read into the first register device to determine the converted mantissa block. The conversion process includes multiplying each 4-bit mantissa by its respective FP8 scale factor mantissa value (5-bit integer, including 4 fraction bits and implicit 1), and then shifting and rounding the result to 8 bits. A 9-bit mantissa is obtained after multiplication. The shift and rounding process is characterized in that each 9-bit mantissa is first shifted by a certain amount, which is calculated by subtracting the block exponent of the mantissa from the maximum exponent over the 4 blocks, and then rounding the result to 8 bits. One or more rows of converted mantissa blocks can be sent to the WB device, and after all rows of mantissa blocks are processed, a 64-byte exponent can be sent to the WB device to complete the decompression of the BFP16-64 format. Here, we assume that the matrix multiplication unit is designed for the BFP16-64 numerical format and therefore needs to be decompressed from SBFP12-16. Of course, there can be other variations, modifications, and alternatives. For example, the SBFP block size can be increased or decreased from 16 to trade off compression ratio versus compression accuracy. Similarly, we can consider different bit widths of the SBFP mantissa and different FP formats of the scale factor.

[0131] Fig. 11B1 is a simplified diagram illustrating a matrix multiplication computing device 1102 according to an example of the present application. As shown in the figure, such a device 1102 can be similar to Figure 3B The exemplary slicing device 302 is configured (see the previous description of the same reference numerals as those shown in FIG. Figure 3B 102). In contrast, apparatus 1102 includes a WB device 1120 coupled to an in-memory computing (IMC) module 332. Similar to IB device 320, WB device 1120 is also coupled to a network-on-chip (NOC) device 342 and a memory device (represented by input from "GM"). As previously described, WB device 1120 can be configured together with IB device 320.

[0132] While the previous examples discussed weight matrix elements, the current compression / decompression implementation can also be applied to other matrix elements, such as matrix activations. In this case (see Fig.11A ) under the condition that the crossbar converter device 1110 is coupled to the crossbar device 360 ​​and the IB device 320, and the decompression method can be applied to multiple activation matrix elements or input matrix elements stored in the memory device 370.

[0133] Fig. 11C 1 is a simplified diagram illustrating a data format 1103 according to an example of the present application. As shown, the data format 1103 includes a plurality of mantissa blocks 1132 and a plurality of scaling factors 1134. As previously described, these mantissa blocks and scaling factors can be configured as columns, wherein each column has a portion of a mantissa block 1132 and a portion of a scaling factor 1134. In other applications, the blocks 1132 and factors 1134 can be configured as rows.

[0134] In the example, blocks 1132 are arranged in an N×M array (composed of B N,M ), and the factors 1134 are arranged in an N×M array (denoted by S N,M Here, the entire array includes an array of blocks 1132 arranged on an array of factors 1134, but this order can be reversed. In a row-by-row configuration, blocks 1132 and factors 1134 can be arranged on the right and left sides of the entire array. Other configurations can also be used depending on the application.

[0135] Referring back to the SBFP12-16 format, each column is configured as a weight matrix column that includes four 8-byte mantissa blocks and four 1-byte scale factors. In this case, each mantissa block row includes 64 mantissa blocks, and each scale factor row includes 64 scale factors, which means that the matrix data format includes a total of 4×64 mantissa blocks and 4×64 scale factors. As previously described, each mantissa block includes 2 int4 mantissas, and each scale factor is configured as an FP8 scale factor. Of course, there may be other variations, modifications, and alternatives.

[0136] Embodiments of such matrix computing devices and related methods thereof can provide many benefits. The present method and device can store a large number of matrix elements in a compressed format that can be decompressed upon retrieval to perform matrix computing. In addition, such compression-decompression capabilities can be accomplished without requiring fully independent hardware and computing paths. Further, these benefits can be implemented in IC chips and core-grain devices with minimal added cost for silicon area.

[0137] According to an example, the present application provides an AI accelerator device configured to perform block compression / decompression. Such a device includes at least a memory device (e.g., a DDR memory) configured to store multiple weight matrix elements in a first format, the first format including multiple weight matrix columns. Each such column includes multiple scale factors and multiple mantissa blocks. The device is also configured with one or more core devices coupled to the memory device, and each core device has at least a CPU coupled to multiple slice devices.

[0138] In this case, the device is similar to Figure 2A The device 201 shown is configured. Each core grain device may include at least a first register device, a second register device, and a converter device, all of which are coupled to the CPU. Similar to the previous matrix computing device, the converter device and the register device may be configured together or separately. In a specific example, the converter device and the register device may be configured in the scheduling device 222. According to the example, the method of operating the AI ​​accelerator device using block compression / decompression can be simply summarized as follows:

[0139] 1. Receiving, by a first register device, a plurality of scaling factors for each weight matrix column from a memory device;

[0140] 2. determining, by the converter device, a maximum index for each weight matrix column using a plurality of scale factors for the weight matrix column, thereby generating a plurality of maximum indices;

[0141] 3. storing the plurality of maximum exponents in a second register device;

[0142] 4. Receiving, by the first register device, a plurality of mantissa blocks for each weight matrix column from the memory device;

[0143] 5. determining, by the converter device, a plurality of conversion mantissa blocks using a plurality of scale factors of a plurality of weight matrix columns and a plurality of mantissa blocks;

[0144] 6. Receiving, by a plurality of memory devices configured in a plurality of slicing devices, a plurality of weight matrix elements in a second format, the second format comprising a plurality of conversion mantissa blocks and a plurality of maximum exponents;

[0145] 7. Determine, by a plurality of computing devices coupled to the plurality of slicing devices, a plurality of matrix multiplication outputs using the plurality of weight matrix elements in the second format; and

[0146] 8. Perform other steps as needed.

[0147] The above sequence of steps is used to operate an AI accelerator device configured to perform block compression / decompression according to one or more embodiments of the present application. Depending on the embodiment, one or more of these steps may be combined or deleted, or other steps may be added without departing from the scope of the claims herein. A person of ordinary skill in the art will recognize other variations, modifications, and substitutions. Further details of this method are provided throughout this specification.

[0148] Although the above is a complete description of a specific embodiment, various modifications, alternative constructions, and equivalents may be used. As an example, the AI ​​accelerator device and the core particle device may include any combination of the above elements and elements outside of this specification. Therefore, the above description and illustration should not be taken as limiting the scope of the present application, which is defined by the appended claims.

Claims

1. A matrix multiplication computing device for an AI accelerator integrated circuit, configured as an integrated circuit, the device comprising: a memory device configured to store a plurality of weight matrix elements in a first format, the first format comprising a plurality of weight matrix columns, each weight matrix column comprising a plurality of scale factors and a plurality of mantissa blocks; a crossbar device coupled to the memory device; a first register coupled to the crossbar device, the first register configured to receive the plurality of scale factors and the plurality of mantissa blocks for each weight matrix column; a converter device coupled to the crossbar device, the converter device configured to determine a maximum index for each weight matrix column using the plurality of scale factors for the weight matrix columns, thereby producing a plurality of maximum indexes; and the converter device is configured to determine a plurality of converted mantissa blocks using the plurality of scale factors and the plurality of mantissa blocks for the plurality of matrix weight columns; a second register coupled to the crossbar device, the second register configured to store the plurality of maximum exponents; a weight buffer device coupled to the crossbar device, the weight buffer device configured to receive the plurality of weight matrix elements in a second format, the second format comprising the plurality of conversion mantissa blocks and the plurality of maximum exponents; a computing device coupled to the weight buffer device, the computing device configured to determine a plurality of matrix multiplication outputs using the plurality of weight matrix elements in the second format; as well as An output buffer device is coupled to the computing device, the output buffer device being configured to store the plurality of matrix multiplication outputs.

2. The matrix multiplication computing device according to claim 1, wherein: The first register, the second register and the converter device are arranged in the crossbar device.

3. The matrix multiplication computing device according to claim 1, wherein: Each of the mantissa blocks includes one or more mantissas, and wherein each scale factor of the plurality of scale factors is associated with one of the plurality of mantissa blocks; The converter device is configured to determine the plurality of converted mantissa blocks by multiplying each mantissa of each mantissa block by the associated scaling factor to produce a scaled mantissa, shifting each scaled mantissa of each mantissa block to produce a shifted mantissa, and rounding each shifted mantissa of each mantissa block to produce a rounded mantissa.

4. The matrix multiplication computing device according to claim 1, wherein: The plurality of matrix inputs in the first format are characterized by a 36×64 byte storage configuration, and wherein the plurality of matrix inputs in the second format are characterized by a 65×64 byte storage configuration.

5. The matrix multiplication computing device according to claim 1, wherein: Each of the first format and the second format comprises a block floating point format; wherein each of the plurality of scale factors is represented by an unsigned 8-bit floating point scale factor, wherein the unsigned 8-bit floating point scale factor includes a 4-bit exponent field and a 4-bit fraction field; and Wherein, each of the plurality of mantissa blocks is represented by an 8-byte block, wherein each byte of the 8-byte block stores two 4-bit integer mantissa values.

6. An AI accelerator device, comprising: a memory storage device for storing a plurality of weight matrix elements in a first format, the first format comprising a plurality of matrix weight columns, each weight matrix column comprising a plurality of scale factors and a plurality of mantissa blocks; At least one chiplet device coupled to a memory storage device, the chiplet device comprising: Central Processing Unit CPU; a first register coupled to the CPU, the first register configured to receive the plurality of scale factors and the plurality of mantissa blocks for each weight matrix column; a converter device coupled to the CPU, the converter configured to determine a maximum exponent for each weight matrix column using the plurality of scale factors for the weight matrix columns, thereby producing a plurality of maximum exponents; and the converter device configured to determine a plurality of conversion mantissa blocks using the plurality of scale factors for the plurality of matrix weight columns and the plurality of mantissa blocks; a second register coupled to the CPU, the second register configured to store the plurality of maximum exponents; and a plurality of slices coupled to the CPU, each of the slices having a memory device and a computation device; wherein the plurality of memory devices are configured to receive the plurality of weight matrix elements in a second format, the second format comprising the plurality of conversion mantissa blocks and the plurality of maximum exponents for each matrix input; and Wherein, the plurality of computing devices are configured to determine a plurality of matrix multiplication outputs using the plurality of weight matrix elements in the second format.

7. The AI ​​accelerator device according to claim 6, wherein: The first register, the second register and the converter device are configured in a scheduling device, which is coupled to the CPU.

8. The AI ​​accelerator device according to claim 6, wherein: Each of the mantissa blocks includes one or more mantissas, and wherein each scale factor of the plurality of scale factors is associated with one of the plurality of mantissa blocks; The converter device is configured to determine the plurality of converted mantissa blocks by multiplying each mantissa of each mantissa block by the associated scaling factor to produce a scaled mantissa, shifting each scaled mantissa of each mantissa block to produce a shifted mantissa, and rounding each shifted mantissa of each mantissa block to produce a rounded mantissa.

9. The AI ​​accelerator device according to claim 6, wherein: The plurality of matrix inputs in the first format are characterized by a 36×64 byte storage configuration, and wherein the plurality of matrix inputs in the second format are characterized by a 65×64 byte storage configuration.

10. The AI ​​accelerator device according to claim 6, wherein: Each of the first format and the second format comprises a block floating point format; wherein each of the plurality of scale factors is represented by an unsigned 8-bit floating point scale factor, wherein the unsigned 8-bit floating point scale factor includes a 4-bit exponent field and a 4-bit fraction field; and Wherein, each of the plurality of mantissa blocks is represented by an 8-byte block, wherein each byte of the 8-byte block stores two 4-bit integer mantissa values.

Citation Information

Cited By

  • Tensor kernel operation component

    CN119576274A

  • Memory device and memory operation method

    TWI907264B