Method for improving resource utilization rate of AI chip, design architecture and AI chip

Abstract

The invention provides a method for improving the resource utilization rate of an AI chip, a design architecture and the AI chip, and the method for improving the resource utilization rate of the AI chip comprises the steps: obtaining input data; determining a matrix multiplication operator type adopted for calculation according to sparsity and dynamics of the input data; and selecting a corresponding calculation engine for adaptation according to the matrix multiplication operator type. According to the method, the corresponding calculation array can be selected according to the characteristics of the input data, and the AI chip resource utilization rate is improved.

Description

Technical Field

[0001] This invention relates to the field of integrated circuit technology, and in particular to a method, design architecture, and AI chip for improving the resource utilization of AI chips. Background Technology

[0002] The rapid development and widespread application of Artificial Intelligence (AI) technology have placed higher demands on computer hardware. AI chips play a crucial role in AI applications, providing powerful computing capabilities and high operating speeds.

[0003] Research shows that the biggest pain point in the engineering implementation of AI is the low utilization rate of AI chip resources. This is because AI algorithm tasks are diverse in type and parameters, while chip parameters are fixed. Therefore, different AI algorithm tasks have different utilization rates of chip resources, directly affecting the actual performance of the chip.

[0004] To improve the utilization rate of hardware acceleration chips for AI algorithms, matrix multiplication operators that appear frequently in AI algorithms are typically adapted separately. However, depending on the characteristics of the input data, the behavior of matrix algorithm operators varies, thus the problem of low utilization rate still exists. Summary of the Invention

[0005] This invention provides a method, design architecture, and AI chip for improving the utilization rate of AI chips, which can further improve the resource utilization rate of AI chips.

[0006] To achieve the above objectives, the present invention provides a method for improving the resource utilization of AI chips, comprising:

[0007] Get the input data;

[0008] Based on the sparsity and dynamism of the input data, determine the type of matrix multiplication operator to be used in the calculation;

[0009] Select the appropriate computing engine based on the type of matrix multiplication operator.

[0010] This invention also provides a design architecture for improving the resource utilization of AI chips, comprising:

[0011] Four computing engines are used to adapt to the sparsity and dynamism of the input data and perform matrix multiplication operations;

[0012] The global controller is used to extract instructions and send control signals to the four computing engines, as well as receive their status signals.

[0013] The present invention also provides an AI chip, including the above-described design architecture for improving the resource utilization of AI chips.

[0014] This invention improves the overall resource utilization of an AI chip by selecting the type of matrix multiplication operator to be used based on the sparsity and dynamism of the input data, in order to match a computing array with high resource utilization. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart of a method for improving the resource utilization of AI chips is provided in an embodiment of the present invention;

[0017] Figure 2 This is a schematic diagram illustrating the mapping of different network operators by the AI ​​algorithm in an embodiment of the present invention;

[0018] Figure 3 This is a schematic diagram of the four types of matrix multiplication operators classified by the AI ​​algorithm in this embodiment of the invention;

[0019] Figure 4 This is a schematic diagram of the computational architecture corresponding to the four types of matrix multiplication operators provided in the embodiments of the present invention;

[0020] Figure 5 This is a schematic diagram of a design architecture for improving the resource utilization of AI chips, provided by an embodiment of the present invention.

[0021] Figure 6 This is a schematic diagram of a specific design architecture provided for an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] In existing technologies, the lack of consideration for the characteristics of input data leads to computational engines that often achieve high resource utilization only when performing calculations on specific types of input data. However, when data characteristics change, computational and storage resources are wasted significantly. Therefore, this invention proposes a method that can automatically configure the corresponding computational engine based on the sparsity and dynamic changes of input data, thereby improving resource utilization.

[0024] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0025] Figure 1 A flowchart of a method for improving the resource utilization of AI chips is provided as an embodiment of the present invention, as follows: Figure 1 As shown, the method in this embodiment includes:

[0026] Step 101: Obtain input data;

[0027] The input data in this step can be feature data or weight data from the AI ​​chip to be used for computation.

[0028] Step 102: Determine the type of matrix multiplication operator to be used for the calculation based on the sparsity and dynamism of the input data.

[0029] Since feature data can be either dense or sparse, sparse data, due to the presence of multiple zero features, can be skipped during computation, eliminating the need for continuous data input. Therefore, computational methods for dense data would result in significant resource waste. To address this, this embodiment of the invention differentiates data based on sparsity to facilitate the use of different computation engines. Furthermore, data can be static or dynamic. Static data is more suitable for in-memory computation (CIM), as it eliminates the need for repeated interactions with external components during computation. Therefore, this embodiment of the invention differentiates data based on its dynamic nature to reduce the interaction frequency of static data, thereby improving resource utilization and reducing power consumption.

[0030] Based on the above analysis, the matrix multiplication operators in this embodiment of the invention are divided into four types: dense static matrix multiplication (FWMM), dense dynamic matrix multiplication (FFMM), sparse static matrix multiplication (SpFWMM), and sparse dynamic matrix multiplication (SpFFMM).

[0031] Step 103: Select the corresponding computing engine for adaptation based on the matrix multiplication operator type.

[0032] After determining the matrix multiplication operator type based on the input data, select the corresponding computing engine for adaptation. The computing engine can be an engine for calculating tensors or an engine for calculating vectors. Regardless of the type, its main function is to complete the corresponding matrix multiplication operation.

[0033] The aforementioned computing engines include: single-address digital CIM arrays, multi-address digital CIM arrays, systolic arrays, and SIMD (Single Instruction Multiple Data) arrays. Dense static matrix multiplication is adapted to single-address digital CIM arrays, dense dynamic matrix multiplication to systolic arrays, sparse static matrix multiplication to multi-address digital CIM arrays, and sparse dynamic matrix multiplication to SIMD arrays. Single-address digital CIM arrays use a single address for multiplexing all subarrays, making them more suitable for computation with continuous input of dense data; therefore, dense static matrix multiplication is adapted to single-address digital CIM arrays. For sparse static feature data, since non-zero data is discontinuous, its corresponding weight data requires different addresses for location; therefore, sparse static matrix multiplication is adapted to multi-address digital CIM arrays. The PE (processing unit) in the systolic array computing engine uses a two-dimensional multiply-accumulate computing array, suitable for dense dynamic data computation, while the SIMD array computing engine uses a one-dimensional multiply-accumulate computing array, suitable for sparse dynamic data computation. In this embodiment of the invention, the resource utilization of the entire chip system is improved by adaptively matching the corresponding computing engine with higher resource utilization based on the sparsity and dynamism of the input data.

[0034] Figure 2 This is a schematic diagram illustrating the mapping of different network operators by the AI ​​algorithm in an embodiment of the present invention. Figure 3 This is a schematic diagram of the four types of matrix multiplication operators classified by the AI ​​algorithm in this embodiment of the invention; Figure 4 This is a schematic diagram of the computational architecture corresponding to the four types of matrix multiplication operators provided in the embodiments of the present invention, such as... Figures 2-4 As shown, in the algorithms of CNN (Convolutional Neural Network), ViT (Visual Transformer), ViG (Visual GNN), and GCN (Graph Convolutional Network), different matrix multiplication operators can be mapped to the sparsity and dynamics of the input data. According to the above embodiments, different matrix multiplication operators are adapted to different computing engines.

[0035] Figure 5 This is a schematic diagram of a design architecture for improving the resource utilization of AI chips, provided by an embodiment of the present invention. Figure 5 As shown, the design architecture of this embodiment includes: four computing engines and a global controller. The four computing engines are used to adapt to the sparsity and dynamism of the input data and perform matrix multiplication operations. The global controller is used to extract instructions and send control signals to the four computing engines and receive their status signals. Under the control of the global controller, this design architecture can execute the methods in the above embodiments, selecting the appropriate computing engine from the four computing engines based on the sparsity and dynamism of the input data, thereby achieving the highest resource utilization.

[0036] In practical applications, the design architecture also includes an on-chip network, which enables data interaction between the four computing engines to reduce interaction with external components. At the same time, it can enable the sharing of cache (buffer) resources within the four computing engines on the chip, improve storage resource utilization, and reduce the number of external memory accesses.

[0037] In some embodiments, the design architecture may further include: an instruction memory for storing instructions extracted by the global controller, which facilitates the execution of AI chip operations by storing pre-compiled instructions.

[0038] Figure 6 A specific design architecture diagram is provided for an embodiment of the present invention, such as... Figure 6 As shown, the four computation engines are FWMM tensor engine, FFMM tensor engine, SpFWMM tensor engine and SpFFMM tensor engine.

[0039] This invention also provides an AI chip, including the design architecture for improving AI chip resource utilization described in the above embodiments.

[0040] This invention improves the overall resource utilization of an AI chip by selecting the appropriate matrix multiplication operator based on the sparsity and dynamism of the input data, thereby matching it with a computing architecture engine that has high resource utilization.

[0041] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0042] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for improving the resource utilization rate of AI chips, characterized in that, include: Get the input data; Based on the sparsity and dynamism of the input data, determine the type of matrix multiplication operator to be used in the calculation; Select the appropriate computing engine based on the type of matrix multiplication operator.

2. The method according to claim 1, characterized in that, The matrix multiplication operators include four types: dense static matrix multiplication, dense dynamic matrix multiplication, sparse static matrix multiplication, and sparse dynamic matrix multiplication.

3. The method according to claim 2, characterized in that, The computing engine includes: a single-address digital CIM array, a multi-address digital CIM array, a systolic array, and a SIMD array; the dense static matrix multiplication is adapted to the single-address digital CIM array, the dense dynamic matrix multiplication is adapted to the systolic array, the sparse static matrix multiplication is adapted to the multi-address digital CIM array, and the sparse dynamic matrix multiplication is adapted to the SIMD array.

4. A design architecture for improving the resource utilization of AI chips, characterized in that, include: Four computing engines are used to adapt to the sparsity and dynamism of the input data and perform matrix multiplication operations; The global controller is used to extract instructions and send control signals to the four computing engines, as well as receive their status signals.

5. The design architecture according to claim 4, characterized in that, Also includes: The on-chip network is used to enable data interaction between the four computing engines.

6. The design architecture according to claim 4 or 5, characterized in that, Also includes: Instruction memory is used to store instructions retrieved by the global controller.

7. An AI chip, characterized in that, Including the design architecture for improving AI chip resource utilization as described in any one of claims 4-6.

Images