Accelerated chip, data processing method and application

Abstract

The application discloses an acceleration chip, comprising: an off-chip storage unit for storing input activation data and weight matrix data of a block floating-point data structure; an on-chip cache unit for caching block floating-point data to be calculated; a block floating-point number direct transfer unit for transferring block floating-point data between the off-chip storage unit and the on-chip cache unit; a block floating-point number matrix multiplication and addition unit for performing a mantissa fixed-point multiplication and addition operation based on a block floating-point data structure and generating a floating-point result; a data format conversion unit for converting between a floating-point format and a block floating-point data structure; and a scheduling control unit for coordinating the work of the block floating-point number direct transfer unit, the block floating-point number matrix multiplication and addition unit and the data format conversion unit. The application also discloses a data processing method, which has a wide application prospect.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and relates to an acceleration chip, a data processing method, and its application. Background Technology

[0002] Matrix multiplication is a core computational component in large models (such as Transformer, GPT, etc.), and its computational efficiency directly impacts the model's training and inference performance. Current accelerator chips based on floating-point operations (such as FP16, BF16) face the following challenges when implementing large-scale matrix multiplication:

[0003] High data transmission bandwidth pressure: The weight matrix and activation matrix have huge data volumes, making memory bandwidth a bottleneck;

[0004] High storage capacity requirements: Model parameters typically reach hundreds of billions, requiring large-capacity off-chip storage;

[0005] High power consumption of computing units: floating-point multipliers have high hardware overhead and low energy efficiency.

[0006] Limited caching efficiency: Traditional floating-point data formats have low utilization rates in on-chip caches. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the present invention aims to provide an acceleration chip, a data processing method, and applications. This invention utilizes the acceleration chip to achieve hardware acceleration of matrix multiplication based on the Block Floating Point (BFP) data format, optimizing matrix multiplication and addition operations in large models. The BFP data format is a compressed representation method that allows multiple data points to share the same exponent, significantly reducing data storage and transmission overhead and allowing the use of fixed-point multipliers for computation, thereby improving energy efficiency while maintaining computational accuracy.

[0008] The computational power of matrix multiplication in large models is mainly constrained by four factors: the number of multiply-accumulate units, data bandwidth, cache efficiency, and off-chip storage capacity. To address this, this invention proposes an acceleration chip and a matrix data processing method based on a block floating-point data format utilizing the acceleration chip. By enabling multiple data sets to share the same exponent information, data bandwidth and storage requirements are effectively reduced while maintaining computational accuracy. Simultaneously, fixed-point multipliers can replace floating-point multipliers in multiplication operations, thereby reducing hardware resource overhead and improving energy efficiency. Based on this data format, a data flow architecture for the acceleration chip is constructed: the host converts the input activation data and weight matrix data into block floating-point format and transmits it to the off-chip storage unit via an interface; the block floating-point direct transport unit schedules data blocks to the on-chip cache unit; the block floating-point matrix multiply-accumulate unit reads data from the on-chip cache unit, performs fixed-point multiply-accumulate operations, and outputs floating-point results; the floating-point results are converted by the data format conversion unit and temporarily stored in the on-chip cache unit or off-chip storage unit, and finally read by the host and converted back to floating-point format for use by the upper-layer model, thereby achieving a systematic improvement in computational throughput and energy efficiency.

[0009] In a first aspect, the present invention provides an acceleration chip that performs matrix calculations based on a block floating-point data structure, comprising:

[0010] Off-chip storage unit is used to store the input activation data and weight matrix data of the block floating-point data structure, where multiple data share the same exponent information and store the corresponding mantissa information respectively.

[0011] On-chip cache unit, used to cache block floating-point data to be computed;

[0012] A block floating-point direct transport unit is used to transport block floating-point data between the off-chip storage unit and the on-chip cache unit;

[0013] The block floating-point matrix multiply-add unit is used to perform mantissa fixed-point multiply-add operations based on the block floating-point data structure and generate floating-point results.

[0014] The data format conversion unit is used to convert between floating-point format and block floating-point data structure;

[0015] The scheduling and control unit is used to coordinate the work of the block floating-point number direct transport unit, the block floating-point number matrix multiplication and addition unit, and the data format conversion unit.

[0016] By setting up collaborative units to implement block floating-point data structures and functions such as data transport, calculation, and format conversion, a dedicated data flow architecture is constructed, which reduces data bandwidth pressure and improves matrix calculation efficiency as a whole.

[0017] In one possible implementation, in the block floating-point data structure, every 8 or 16 floating-point data points share an exponent information. By using a block floating-point structure where multiple data points share an exponent, the data storage volume can be reduced while maintaining computational accuracy, thereby reducing storage and transmission bandwidth requirements.

[0018] In one possible implementation, the on-chip cache unit adopts a multi-bank structure and is physically divided into an activation data area, a weight data area, and an intermediate result data area to support parallel read and write operations. Through the multi-bank partitioned cache structure, parallel access to activation data, weight data, and intermediate results is achieved, thereby improving on-chip data reuse rate and computing throughput.

[0019] In one possible implementation, the on-chip cache unit includes a mantissa storage area and an exponent storage area, which are used to store mantissa information and exponent information in the block floating-point data, respectively. By separating the exponent area and the mantissa area into separate storage structures, the continuity of data access and storage utilization can be improved, and cache access conflicts can be reduced.

[0020] In one possible implementation, the block floating-point direct transport unit may further include a block floating-point direct read unit and a block floating-point direct write unit;

[0021] The block floating-point direct read unit reads data from the off-chip memory unit and converts it into streaming data based on the read enable, read base address, and read length parameters.

[0022] The block floating-point direct write unit writes streaming data back to the off-chip storage unit based on write enable, write base address, and write length parameters.

[0023] Automatic data transfer via block-level read / write units can reduce memory access control overhead and improve data transfer efficiency between off-chip and on-chip environments.

[0024] In one possible implementation, the block floating-point matrix multiply-add unit includes a fixed-point multiplier and a multi-level adder structure, and employs a multi-level pipelined structure to perform fixed-point multiply-add operations, fixed-point to floating-point conversion, and multiply-add accumulation. The multi-level adder structure may further include a fixed-point multiplier, a fixed-point adder tree, a floating-point converter, a floating-point accumulator, and a floating-point mantissa converter.

[0025] By replacing traditional floating-point multiply-accumulate units with fixed-point multiplication arrays and multi-stage pipeline structures, computing density can be increased and power consumption reduced within the same hardware area.

[0026] In one possible implementation, the acceleration chip is configured as a PCIe expansion card and connected to the host server via a high-speed bus interface.

[0027] Secondly, the present invention provides a data processing method applied to the aforementioned acceleration chip, the method being based on a block floating-point data structure, comprising:

[0028] Convert the input activation data and weight data into a block floating-point data structure;

[0029] The exponent and mantissa information in the block floating-point data structure are stored in an off-chip storage unit;

[0030] Block floating-point data is moved to the on-chip cache unit via the block floating-point direct transfer unit;

[0031] In the block floating-point matrix multiplication and addition unit, fixed-point multiplication and addition operations are performed based on the mantissa information, and alignment processing is performed based on the exponent information to generate floating-point format calculation results;

[0032] The floating-point calculation result is converted into a new block floating-point data structure and written back to the off-chip storage unit.

[0033] By using a block floating-point data structure-driven transport and fixed-point multiplication-addition calculation process, memory access bandwidth requirements can be reduced and overall matrix calculation efficiency can be improved.

[0034] In one possible implementation, the matrix multiplication and addition operation in this invention adopts a multi-stage pipeline structure, wherein the pipeline sequentially includes a fixed-point multiplication stage, a fixed-point addition tree stage, a fixed-point to floating-point conversion stage, a floating-point accumulation stage, and a floating-point output stage.

[0035] Performing matrix multiplication and addition operations through a multi-stage pipeline structure can reduce the waiting time between computation stages and improve processing throughput.

[0036] In one possible implementation, the scheduling and control unit coordinates the work of the block floating-point number direct transport unit, the block floating-point number matrix multiplication and addition unit, and the data format conversion unit to form a three-level pipeline execution structure of transport-computation-writeback. This allows the data transport and computation processes to overlap, enabling memory access and computation processes to overlap, thereby improving hardware utilization.

[0037] Thirdly, the present invention also provides the application of the above-mentioned acceleration chip or the above-mentioned data processing method in large-scale matrix computing scenarios, which can support matrix operations with higher throughput under the same bandwidth conditions.

[0038] The beneficial effects of this invention include: Compared to existing matrix computation schemes based on general-purpose processors, this invention introduces a block floating-point data structure and builds a dedicated data transport, on-chip cache, and matrix multiplication-addition computation architecture around this data structure, achieving coordinated optimization from data storage and transmission to computation execution. Since multiple data points share the same exponent information, the data volume is significantly reduced while ensuring computational accuracy, thereby alleviating off-chip storage bandwidth pressure and improving data transmission efficiency. Simultaneously, the mantissa portion uses a fixed-point multiplier array for computation, which can replace traditional floating-point multiplication-addition units, integrating more computational units within the same chip area, increasing computational density and reducing power consumption. Furthermore, through a block-level data transport mechanism, a multi-bank on-chip cache structure, and a pipelined execution method of transport-computation-writeback, the overlap between memory access and computation processes is achieved, reducing the time computational units wait for data, improving overall hardware utilization and matrix computation throughput, thereby achieving a comprehensive improvement in energy efficiency and performance at the system level. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is the system architecture diagram of the present invention.

[0041] Figure 2 This is a schematic diagram of the block floating-point data storage format of the present invention.

[0042] Figure 3 This is a schematic diagram illustrating the arrangement of eight BF16 format floating-point data after conversion into block floating-point format (BFP) in one possible implementation of the present invention.

[0043] Figure 4 This is a schematic diagram of the BDMA function of the present invention.

[0044] Figure 5 This is a schematic diagram of the BMMA structure of the present invention. Detailed Implementation

[0045] The present invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the present invention are all common knowledge and general knowledge in the art, and the present invention does not have any particular limitations.

[0046] This invention aims to solve the following key technical problems encountered when performing matrix multiplication on large models such as large language models on general-purpose computing platforms (CPU / GPU):

[0047] 1. Low energy efficiency and computational density: The general-purpose floating-point computing units in CPUs / GPUs have high hardware overhead. When performing intensive matrix multiplication and addition operations, which are mainly of medium to low precision in large models, there is significant power consumption and area waste, resulting in insufficient effective computing power density. 2. "Memory wall" and bandwidth bottleneck: Large models have a huge number of parameters, and frequent data transfer leads to a serious "memory wall" problem. The general-purpose storage architecture of CPUs / GPUs (such as multi-level cache and high-bandwidth memory) is not optimized for large matrix contiguous block operations, resulting in low data reuse efficiency and limited effective computing power utilization. 3. Lack of native support for efficient data formats: CPUs / GPUs mainly support standard floating-point formats and cannot efficiently support specialized formats such as block floating-point, which can significantly compress data and reduce bandwidth, making it difficult to fundamentally alleviate storage and transmission pressure. 4. Control and scheduling overhead caused by the general architecture: The complex control logic and thread scheduling mechanism of CPUs / GPUs generate unnecessary performance overhead when handling large model matrix operations with regular computational patterns.

[0048] The core problem lies in the fact that the trade-offs made by general-purpose processor architectures between flexibility, accuracy and energy efficiency, and specialization are no longer suitable for the extreme demands of large-scale model computing for high energy efficiency, high data throughput, and customized data streams.

[0049] To address this, the present invention proposes a dedicated matrix multiplication acceleration chip based on block floating-point data format. By customizing the computing unit, storage level, and data path, the above problems are systematically solved, achieving higher energy efficiency and computing efficiency.

[0050] The block floating-point data format used in this invention is an efficient data representation method that allows all values ​​within a data block to share the same exponent, while each value independently stores its fixed-point mantissa. The core of this design lies in transforming the traditional floating-point concept of "each number having its own exponent" into "the entire group of numbers sharing a single scale." This significantly reduces the space overhead required to store the exponent while maintaining a large dynamic range. Furthermore, when performing operations such as addition and subtraction within a block, only integer operations need to be performed on the mantissa and the shared exponent needs to be adjusted uniformly. The calculation process is simplified to efficient shifting and addition, avoiding complex floating-point multiplication. Therefore, it is particularly suitable for hardware optimization and power-sensitive applications, such as digital signal processing and edge computing.

[0051] This invention provides a matrix multiplication acceleration chip specifically designed for large model inference. Its core lies in its efficient support for block floating-point data formats through a fully customized hardware architecture, thereby systematically improving computing energy efficiency and throughput.

[0052] The acceleration chip mainly includes the following core hardware units:

[0053] 1. Data Interface and Off-Chip Storage Unit: The system receives input activation data and weight matrix data from the host via a PCIe interface. The host pre-converts the input activation data and weight matrix data into block floating-point format using the CPU. The converted block floating-point format data is temporarily stored in an off-chip storage unit, which can be a high-capacity DRAM or HBM.

[0054] In the block floating-point data structure, typically every 8 or 16 floating-point numbers share one exponent information to reduce data storage volume and transmission bandwidth requirements.

[0055] 2. Block Floating-Point Direct Transfer Unit and On-Chip Cache Unit: A block floating-point direct transfer unit is set up to efficiently schedule data from off-chip storage units to a dedicated on-chip cache unit for block floating-point data. The on-chip cache unit is optimized for continuous access mode for matrix blocks. In DRAM / HBM, the exponents of multiple data blocks are arranged consecutively, and the mantissas are arranged consecutively. DMA can read and write block floating-point data through consecutive address bursts, maximizing data reuse and alleviating the "memory wall" problem.

[0056] The read / write address and data length of the block floating-point direct transfer unit are specified by instructions, and the read / write process is autonomously scheduled by the block floating-point direct transfer unit.

[0057] 3. Core Computation: The block floating-point matrix multiply-accumulate unit is the core of the computation. It reads the input activation data and weight matrix data in block floating-point format from the on-chip cache unit. Thanks to the characteristics of the block floating-point format, the block floating-point matrix multiply-accumulate unit uses a simplified and efficient fixed-point multiplier array to perform parallel multiply-accumulate operations, achieving higher computing unit density and energy efficiency, and finally outputting standard floating-point calculation results.

[0058] 4. Data Backflow and Output: A data format conversion unit is set up to convert between floating-point format and block floating-point data structure. Specifically, it performs the conversion from floating-point format to block floating-point data structure. The floating-point result obtained in the previous step is converted back to block floating-point format in real time and written back to the on-chip cache unit for use in subsequent layer calculations or for output preparation. The final result is returned to the host through the PCIe interface.

[0059] In one possible implementation, the acceleration chip further includes a scheduling and control unit for coordinating the work of the block floating-point number direct transport unit, the block floating-point number matrix multiplication and addition unit, and the data format conversion unit.

[0060] The entire solution utilizes native hardware support for block floating-point format to construct a customized data flow from storage and transfer to computation and reflow, significantly improving computational parallelism and energy efficiency while reducing data bandwidth and storage pressure.

[0061] Figure 1 This diagram illustrates the system architecture of the present invention, showing the main processing units and data flow.

[0062] BFP: Block Float Point data format;

[0063] BDMA: Block Floating-Point Direct Memory Access, which is divided into Block Floating-Point Direct Read (RDMA) and Block Floating-Point Direct Write (WDMA).

[0064] BRAM: BFP SRAM Block Floating-Point Static Storage Unit, is an on-chip cache unit used in a possible implementation, which is divided into Mantissa Storage Unit MRAM and Exponent Storage Unit ERAM;

[0065] BMMA: Block Floating-Point Matrix Multiply Accumulate (BFP)

[0066] F2B: Float to BFP (Floating-point to Block Floating-point) unit, also known as the data format conversion unit.

[0067] Figure 2 This describes the storage format of the block floating-point data in the on-chip cache unit of the present invention. The storage space of the on-chip cache unit is divided into two parts: the mantissa segment (MAN SEGMENT) and the exponent segment (EXP SEGMENT). Each part is further divided into three sub-segments: input activation data, output activation data, and weight matrix data. All exponents are stored contiguously together, and all mantissas are also stored contiguously together. This storage structure facilitates efficient data reading and writing in memory by BDMA.

[0068] Figure 3 This indicates the arrangement of 8 BF16 format floating-point data after conversion to block floating-point format (BFP) in a possible implementation.

[0069] Figure 4 The diagram illustrates the function of BDMA. RDMA is controlled by three parameters: read enable, read base address, and read length. It automatically issues an AXI read command, reads AXI data from memory and converts it into streaming data, and issues a read end flag when the read is complete.

[0070] WDMA is controlled by three parameters: write enable, write base address, and write length. It automatically issues AXI (Advanced Dextensible Interface) write commands, converts streaming data into AXI write data, writes the data into memory, and issues a write end flag when the write is complete.

[0071] Figure 5 The structure of BMMA is described below. BMMA takes block-float formatted input activation data and block-float formatted weight matrix data as input, and outputs floating-point formatted activation data. It consists of a five-stage pipeline: the first stage is a fixed-point multiplier, the number of which is determined by the shared number of block-float numbers; the second stage is a multi-stage fixed-point addition tree that performs pairwise additions; the third stage is a floating-point converter, used to convert fixed-point numbers to floating-point numbers by multiplying fixed-point data by floating-point coefficients; the fourth stage is a floating-point accumulator, which accumulates the results of multiple multiplication and addition operations; and the fifth stage is a floating-point mantissa converter, which outputs floating-point numbers according to the required floating-point format.

[0072] The embodiment of the present invention is a collaborative system encompassing both hardware and software. Its specific execution process can be divided into the following four stages, as illustrated in the accompanying drawings (…). Figures 1-3 Corresponding to:

[0073] Phase 1: Data Preprocessing and Unloading

[0074] Model loading and format conversion: On the host server, the driver and runtime library are used to batch convert the large model weight matrix data and input activation data (INPUT) to be calculated from standard floating-point format (such as BF16) to the block floating-point (BFP) format defined in this invention. This process can be performed offline, and the converted weights can be persistently stored, enabling one-time conversion and multiple uses.

[0075] Data transfer: The converted BFP format input activation data and weight matrix data are efficiently transferred to the off-chip storage unit of the accelerator card, such as mass storage (DRAM or HBM), via the PCIe DMA engine.

[0076] Phase Two: On-Chip Computing and Data Scheduling

[0077] Block Data Transfer: A dedicated Block Floating-Point Direct Transfer (BDMA) unit, following instructions from the scheduling and control unit, prefetches the activation and weights of specific data blocks (tiles) from off-chip memory to the Block Floating-Point On-Chip Cache (BRAM) unit. The BRAM employs a multi-bank design, physically divided into an activation data area, a weight data area, and an intermediate result data area, supporting parallel read and write operations for activation, weights, and intermediate results.

[0078] Matrix multiplication and addition: The Block Floating-Point Matrix Multiplication and Addition Unit (BMMA) reads the activation block (IA_MAN, IA_EXP) and weight block (WGT_MAN, WGT_EXP) from the BRAM. The computation core adopts a multi-stage pipelined architecture, internally using a fixed-point multiplier to process the mantissa (MAN) multiplication, and performing shift alignment and accumulation according to the shared exponent (EXP), ultimately producing an intermediate output (OUTPUT) in floating-point format.

[0079] In some embodiments, the first-stage pipeline consists of 8 or 16 fixed-point multipliers, forming a fixed-point multiplication stage; the second-stage pipeline is a fixed-point addition tree stage, employing a fixed-point addition tree; the third-stage pipeline is a fixed-point to floating-point stage, employing a floating-point converter; the fourth-stage pipeline is a floating-point accumulation stage, employing a floating-point accumulator; and the fifth-stage pipeline is a floating-point output stage (including mantissa truncation and rounding), employing a floating-point mantissa converter.

[0080] In some embodiments, the block floating-point matrix multiply-accumulate unit calculates a fixed-point multiply-accumulate operation between an active data block and a weighted data block each time, and completes the final floating-point accumulation.

[0081] Result format write-back: The output floating-point result is dynamically reorganized into a new BFP data block (generating a new OA_EXP and OA_MAN) through the floating-point to block floating-point unit (F2B), and written back to BRAM as input for the next layer of calculation or waiting for output.

[0082] Phase 3: Computational Control and Automation

[0083] A centralized instruction scheduling and control unit coordinates the work of BDMA, BMMA, and F2B units, forming a three-stage pipeline of "transfer-computation-writeback". The control information of the scheduling and control unit is pre-written into the instruction queue, and the control information is retrieved from the instruction queue one by one during actual operation.

[0084] While BMMA is computing the current data block, BDMA is moving the next data block needed for computing, and F2B is processing the result of the previous computing block, thus achieving complete overlap between computing and data moving, hiding memory access latency, and maximizing hardware utilization.

[0085] Phase 4: Results Collection and Post-processing

[0086] After the calculation is completed, the output data of the final layer (BFP format) is written back from the BRAM to the off-chip memory via the output WDMA.

[0087] Data is transmitted to the host server via the PCIe interface.

[0088] The host-side driver quickly converts the BFP format results back to the standard floating-point format and delivers them to the upper-layer AI application. The entire process is transparent to the application.

[0089] Compared to existing general-purpose computing architectures based on CPU / GPU, the matrix multiplication acceleration chip based on block floating-point (BFP) proposed in this invention achieves multi-dimensional and systematic performance and efficiency breakthroughs in core computing tasks involving large models. Its beneficial effects are specifically manifested as follows:

[0090] First, it achieves a fundamental improvement in the energy efficiency and density of computing cores. Existing CPUs / GPUs rely on highly complex floating-point multiply-accumulate units (FMA), resulting in huge hardware overhead and power consumption. This invention adopts a block floating-point data format, whose multiplication operations can be converted into fixed-point multipliers with far lower hardware resource consumption than floating-point units. This allows for the integration of more computing units within the same chip area, achieving higher computing density and better energy efficiency.

[0091] Secondly, it significantly alleviates the "memory wall" bottleneck that restricts the release of computing power. In CPU / GPU architectures, massive model parameters and activation values ​​are transmitted in standard floating-point format, putting enormous pressure on memory bandwidth and causing computing units to frequently be in a state of data starvation. This invention compresses data in block floating-point format (multiple data points share a single exponent), enabling the transmission of more effective data within the same physical bandwidth, improving data transmission efficiency by up to 50%. This allows computing units to work continuously at full capacity, fully releasing theoretical computing power.

[0092] Third, it significantly reduces storage costs and capacity requirements. Large models with hundreds of billions of parameters require massive off-chip storage (such as HBM). Storing raw floating-point data is costly. This invention uses a block floating-point format to store weights and activations, which can effectively compress data volume while maintaining accuracy. This allows for the deployment of larger-scale models with the same hardware storage capacity, or saves expensive storage hardware costs for the same model.

[0093] Fourth, optimize data movement and cache utilization efficiency. General-purpose cache architectures have limited utilization of the data locality of independent floating-point numbers. The block floating-point format adopted in this invention has an inherent regular block structure, which is perfectly matched with the dedicated Block Data Moving Unit (BDMA) and on-chip cache (BRAM) strategy, realizing more efficient data block (tile) scheduling and reuse, reducing redundant access, and improving the overall storage hierarchy efficiency.

[0094] This invention can bring the following quantifiable or clearly perceptible benefits:

[0095] 1. Performance benefits: The effective utilization rate of computing power is doubled.

[0096] Benefits: In typical large-scale matrix multiplications (such as forward propagation in GPT-3), the time spent by computing units waiting for data is significantly reduced due to a 30-50% decrease in data bandwidth requirements. It is expected that, under the same process technology, the chip's measured computing power utilization (i.e., the percentage of peak computing power reached) can be improved by more than 40% compared to general-purpose GPUs, allowing nominal computing power to be more fully converted into actual computing speed.

[0097] 2. Energy efficiency benefits: Significantly reduced power consumption per unit of computing power.

[0098] Benefits: By replacing general-purpose floating-point multipliers with custom fixed-point multipliers, combined with the power savings from off-chip memory access due to data compression, the core computing module's energy efficiency ratio (TOPS / W) is expected to be 2-3 times that of comparable GPUs when performing the same computational tasks. This has direct value for reducing operating costs (electricity bills) and meeting carbon reduction targets in data centers.

[0099] 3. Cost-benefit: The total cost of ownership (TCO) of the system decreases.

[0100] Benefits are reflected in:

[0101] Storage Costs: For models with hundreds of billions of parameters, using BFP format storage weights can save approximately 35% of HBM or DRAM capacity requirements. In high-end AI accelerator cards, HBM costs account for a very high percentage, so this directly reduces hardware material costs.

[0102] Bandwidth cost: Lower physical memory bandwidth is required to achieve the same computing power, allowing the use of memory solutions with lower bandwidth specifications and lower costs, or supporting larger models with the same bandwidth.

[0103] Deployment density: A higher energy efficiency ratio means that more computing cards can be deployed within a fixed rack power limit, thereby improving the overall AI computing power output of the data center.

[0104] 4. Applicability benefits: Supports larger models and more complex applications.

[0105] Benefits: Within the constraints of given hardware resources (storage capacity, interconnect bandwidth), this invention can support full-parameter training or inference for models with a 30%-50% increase in parameter size. This enables a single accelerator card or server to handle more complex multimodal models and language models with longer contexts, expanding the application boundaries of a single device.

[0106] In summary, this invention provides a highly efficient and dedicated hardware solution for addressing the computing power and energy efficiency bottlenecks in large-scale model inference through collaborative design of data format, computing units, storage architecture, and control logic.

[0107] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0108] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0109] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0110] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0111] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0112] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

[0113] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of this invention are included in this invention and are protected by the appended claims.

Claims

1. An accelerator chip, characterized in that, The acceleration chip natively supports block floating-point format through end-to-end customized hardware, constructing a customized data stream from storage and transfer to computation and backflow; The acceleration chip includes: Off-chip storage units are used to store the input activation data and weight matrix data of the block floating-point data structure; An on-chip cache unit is used to cache block floating-point data to be calculated. The on-chip cache unit includes a mantissa storage area and an exponent storage area. The mantissa information and exponent information in the block floating-point data are stored separately through a separate storage structure for the exponent area and the mantissa area. The on-chip cache unit adopts a multi-bank structure and is physically divided into an activation data area, a weight data area and an intermediate result data area, supporting parallel read and write. The block floating-point direct transfer unit is used to transfer block floating-point data between the off-chip storage unit and the on-chip cache unit; the block floating-point direct transfer unit also includes a block floating-point direct read unit and a block floating-point direct write unit; The block floating-point direct read unit reads data from the off-chip memory unit and converts it into streaming data based on the read enable, read base address, and read length parameters. The block floating-point direct write unit writes streaming data back to the off-chip storage unit based on write enable, write base address and write length parameters; The block floating-point matrix multiply-add unit is used to perform mantissa fixed-point multiplication and addition operations based on the block floating-point data structure, and to perform alignment processing according to the exponent information to generate floating-point format calculation results. The block floating-point matrix multiply-add unit includes a fixed-point multiplier and a multi-level addition structure. The multi-level addition structure includes a fixed-point multiplier, a fixed-point addition tree, a floating-point converter, a floating-point accumulator, and a floating-point mantissa converter. In the block floating-point data structure, every 8 or 16 floating-point data shares one exponent information. When performing intra-block addition and subtraction operations, only integer operations need to be performed on the mantissa and the shared exponent needs to be adjusted uniformly. The calculation process is simplified to shifting and addition. The data format conversion unit is used to convert between floating-point format and block floating-point data structure; The scheduling and control unit is used to coordinate the work of the block floating-point number direct transport unit, the block floating-point number matrix multiplication and addition unit, and the data format conversion unit.

2. The acceleration chip as described in claim 1, characterized in that, The acceleration chip is configured as an expansion card and is connected to the host server via a high-speed bus interface.

3. A data processing method, applied to the acceleration chip as described in claim 1 or 2, characterized in that, The method includes: Convert the input activation data and weight data into a block floating-point data structure; The exponent and mantissa information in the block floating-point data structure are stored in an off-chip storage unit; Block floating-point data is moved to the on-chip cache unit via the block floating-point direct transfer unit; In the block floating-point matrix multiplication and addition unit, fixed-point multiplication and addition operations are performed based on the mantissa information, and alignment processing is performed based on the exponent information to generate floating-point format calculation results; The floating-point calculation result is converted into a new block floating-point data structure and written back to the off-chip storage unit.

4. The method as described in claim 3, characterized in that, The matrix multiplication and addition operation adopts a multi-stage pipeline structure, which includes a fixed-point multiplication stage, a fixed-point addition tree stage, a fixed-point to floating-point conversion stage, a floating-point accumulation stage, and a floating-point output stage.

5. The method as described in claim 3, characterized in that, The scheduling and control unit coordinates the work of the block floating-point number direct transport unit, the block floating-point number matrix multiplication and addition unit, and the data format conversion unit to form a three-level pipeline execution structure of transport-computation-writeback, so that the data transport and computation processes overlap.

6. The application of the acceleration chip as described in claim 1 or 2 in large-scale matrix computing scenarios.

7. The application of the data processing method as described in any one of claims 3-5 in large-scale matrix computation scenarios.

Images