A design method for a novel compute-memory fusion architecture for neural network accelerators

Abstract

This invention discloses a design method for a novel compute-memory fusion architecture for neural network accelerators, belonging to the field of neural network processor technology. To address the issues of reducing data movement and improving computational efficiency, the invention includes: obtaining a quantized neural network through weight and activation quantization; designing a dedicated single-input multiplier; where the quantized NN accelerator uses two multipliers with fixed-point integer inputs for weight and input multiplication operations, fixing one input of the multiplier to a specific multiplier to obtain a dedicated single-input multiplier; performing weight analysis on the quantized NN obtained in step S1; designing the hardware for the data path; and performing matrix multiplication operations on the compute-memory fusion architecture NN accelerator: the input data of each layer in the NN is transmitted to the designated dedicated single-input multiplier through the data path, and the results of the multiplication operations are accumulated by an adder according to the rules of matrix multiplication, ultimately obtaining the matrix multiplication results of the compute-memory fusion architecture NN accelerator.

Description

Technical Field

[0001] This invention belongs to the field of neural network processor technology, and specifically relates to a design method for a novel compute-memory fusion architecture for neural network accelerators. Background Technology

[0002] Neural Network (NN) technology has demonstrated superior capabilities in tasks such as natural language processing, machine vision, and image and video generation. It not only brings great convenience to human life but also assists humans in performing complex tasks, accelerating the development of various industries. However, under current technological levels, the massive parameter scale, the large amount of computing resources required, and the need for data access pose significant challenges in terms of energy consumption and computing power, further hindering the development and application of neural network technology. Therefore, the industry urgently needs to seek new, energy-efficient computing architectures that can significantly improve computing energy efficiency while ensuring the performance of intelligent tasks as much as possible.

[0003] To improve the energy efficiency of intelligent computing systems, the field has opted to replace general-purpose central processing units (CPUs) and graphics processing units (GPUs) with dedicated hardware accelerators to handle intelligent computing tasks. Intelligent computing hardware accelerators typically employ application-specific integrated circuit (ASIC) optimization techniques such as multi-core processing, pipelined computation processes, and approximate circuit design. These are combined with pruning and quantization techniques to compress models, and further enhanced through optimized storage strategies and hardware-software co-design to improve energy efficiency. Compared to general-purpose CPU and GPU computing platforms, these accelerators offer significant performance and power consumption optimizations.

[0004] However, these optimization methods cannot effectively reduce the energy consumption caused by large-scale data movement. In contrast, a great deal of work is being done on in-memory computing and near-memory computing techniques, aiming to reduce the energy costs and performance losses caused by data movement by performing computational operations inside or near memory or storage cells. However, in-memory computing is limited by the technology and scale of memory, and its computational performance and accuracy are very limited; near-memory computing only shortens the distance of movement, but still cannot avoid data movement altogether. Summary of the Invention

[0005] The problem this invention aims to solve is to reduce data movement and improve computational efficiency while ensuring the computational performance of neural network accelerators. It proposes a design method for a novel compute-memory fusion architecture for neural network accelerators.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A design method for a novel compute-memory fusion architecture for neural network accelerators includes the following steps:

[0008] S1. Weight and Activation Quantization: Using an open-source weight quantization method, the weights of the neural network and the input of each layer are quantized on a software platform built on the PyTorch open-source framework to obtain a quantized neural network.

[0009] S2. Design a dedicated single-input multiplier: Based on the quantized neural network obtained in step S1, design a dedicated single-input multiplier for the quantized neural network accelerator. When the quantized NN accelerator performs the multiplication operation between the weights and the input, it uses a multiplier with two fixed-point integer inputs. Fix one input of the multiplier to a specific multiplier to obtain a dedicated single-input multiplier.

[0010] S3. Weight Analysis: The quantized NN obtained in step S1 is further analyzed on the software platform used in step S1. The number of different elements in the weight matrix is ​​counted to determine the number of different single-input dedicated multipliers used in a single calculation operation.

[0011] S4. Hardware design of the data path: On the software platform used in step S1, first complete the declaration of the module name, input and output ports in the hardware description file; then, use the software platform used in step S1 to re-traverse the weight matrix, generate the corresponding connection statements according to the arrangement of weights in the weight matrix and the correspondence during matrix multiplication, and write them into the hardware description file; after the weight traversal is complete, add the statement to end the module establishment in the hardware description file to complete the hardware design of the data path.

[0012] S5. Matrix multiplication operation of the compute-memory NN accelerator: The input data of each layer in the NN is transmitted to the designated single-input dedicated multiplier through the data path constructed in step S4. The result of the multiplication operation is then accumulated by the adder according to the operation rules of matrix multiplication, and finally the result of the matrix multiplication operation of the compute-memory NN accelerator is obtained.

[0013] Furthermore, the matrix multiplication operation of the weights and inputs in the quantized neural network in step S1 internally includes multiplication operations between a low number of specific points.

[0014] Furthermore, the range of multipliers in the single-input dedicated multiplier in step S2 depends on the number of bits used to quantize the weights in step S1.

[0015] Furthermore, in step S3, the method for counting the number of different elements in the weight matrix is ​​to traverse all elements in the weight matrix and increment the corresponding count variable by 1 according to the value of the element.

[0016] Furthermore, in step S3, when the weights of the NN are the weights of the fully connected layer, a large weight matrix is ​​used for the entire layer, and the weight matrix is ​​analyzed column by column; in step S3, when the weights of the NN are in the form of a convolutional layer, each input channel corresponds to a convolutional kernel, and the weights of the convolutional layer are analyzed channel by channel.

[0017] Furthermore, in step S4, the hardware design of the data path implements the use of wiring to represent the correspondence from input data to a single-input dedicated multiplier.

[0018] The beneficial effects of this invention are:

[0019] The present invention describes a novel computation-memory fusion architecture design method for neural network accelerators. This method stores neural network weights in the form of a single-input specific multiplier and data path into the logic computing unit. By placing the weights within the structure of the logic computing circuit, it is a hardware design method that avoids loading the weight matrix from external memory, eliminates weight-related memory access operations, and significantly reduces data movement in the neural network accelerator.

[0020] The present invention discloses a design method for a novel compute-memory fusion architecture for neural network accelerators, which proposes to store the weight matrix in the computing unit. This avoids a large amount of data movement while ensuring that the computing efficiency of the compute-memory fusion computing unit is close to that of commonly used logic operation units. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the design method of a novel compute-memory fusion architecture for neural network accelerators as described in this invention.

[0022] Figure 2 This is a schematic diagram of the single-input dedicated multiplier of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention; that is, the described specific embodiments are merely a part of the embodiments of the invention, and not all of them. The components of the specific embodiments of the invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations, and the invention may also have other embodiments.

[0024] Therefore, the following detailed description of specific embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected specific embodiments of the invention. All other specific embodiments obtained by those skilled in the art based on these specific embodiments without inventive effort are within the scope of protection of this invention.

[0025] To further understand the invention's content, features, and effects, the following specific embodiments are provided, along with accompanying drawings. Figure 1 and attached Figure 2 The detailed instructions are as follows:

[0026] Example 1:

[0027] A design method for a novel compute-memory fusion architecture for neural network accelerators includes the following steps:

[0028] S1. Weight and Activation Quantization: Using an open-source weight quantization method, the weights of the neural network and the input of each layer are quantized on a software platform built on the PyTorch open-source framework to obtain a quantized neural network.

[0029] Furthermore, the matrix multiplication operation of the weights and inputs in the quantized neural network in step S1 internally contains a low-ratio multiplication operation between specific points.

[0030] Furthermore, this implementation requires a computational architecture design for a quantized neural network;

[0031] S2. Design a dedicated single-input multiplier: Based on the quantized neural network obtained in step S1, design a dedicated single-input multiplier for the quantized neural network accelerator. When the quantized NN accelerator performs the multiplication operation between the weights and the input, it uses a multiplier with two fixed-point integer inputs. Fix one input of the multiplier to a specific multiplier to obtain a dedicated single-input multiplier.

[0032] Furthermore, the range of multipliers in the single-input dedicated multiplier in step S2 depends on the number of bits used to quantize the weights in step S1;

[0033] Furthermore, a single-input dedicated multiplier is specifically designed to handle multiplication by 2, 3, ..., etc. Thus, single-input dedicated multipliers with fixed multipliers can be obtained. The range of multipliers depends on the number of bits used to quantize the weights in S1. For example, for weights quantized by 3 bits, the numerical range is limited to integers within the interval (-8, 8]. Therefore, integers between -7 and 8 are fixed as one input to the multiplier, while 0 and 1 are not fixed. This results in 14 possible multipliers. Similarly, for weights quantized by n bits, the range of -(2...) is determined by... n -1) to 2 nThe integer is fixed as one input to the multiplier, excluding 0 and 1, thus obtaining (2 n+1 -2) types of multipliers.

[0034] S3. Weight Analysis: The quantized NN obtained in step S1 is further analyzed on the software platform used in step S1. The number of different elements in the weight matrix is ​​counted to determine the number of different single-input dedicated multipliers used in a single calculation operation.

[0035] Furthermore, the method for counting the number of different elements in the weight matrix in step S3 is to traverse all elements in the weight matrix and increment the corresponding count variable by 1 according to the value of the element.

[0036] Furthermore, taking 3 as the quantization level as an example, a total of 14 counting variables n are set. -7 Up to n8 (excluding n0 and n1), during the traversal of the weight matrix, if an element in the matrix is ​​2, perform (n2+1), and so on. Since the range of the number of weights in a quantized neural network is limited, the scale of this statistical method is also limited.

[0037] Furthermore, in step S3, when the weights of the NN are the weights of the fully connected layer, a large weight matrix is ​​used for the entire layer, and the weight matrix is ​​analyzed column by column; in step S3, when the weights of the NN are in the form of a convolutional layer, each input channel corresponds to a convolutional kernel, and the weights of the convolutional layer are analyzed channel by channel.

[0038] S4. Hardware design of the data path: On the software platform used in step S1, first complete the declaration of the module name, input and output ports in the hardware description file; then, use the software platform used in step S1 to re-traverse the weight matrix, generate the corresponding connection statements according to the arrangement of weights in the weight matrix and the correspondence during matrix multiplication, and write them into the hardware description file; after the weight traversal is complete, add the statement to end the module establishment in the hardware description file to complete the hardware design of the data path.

[0039] Furthermore, in step S4, the hardware design of the data path implements the use of wiring to represent the correspondence from the input data to the single-input dedicated multiplier;

[0040] S5. Matrix multiplication operation of the compute-memory NN accelerator: The input data of each layer in the NN is transmitted to the designated single-input dedicated multiplier through the data path constructed in step S4. The result of the multiplication operation is then accumulated by the adder according to the operation rules of matrix multiplication, and finally the result of the matrix multiplication operation of the compute-memory NN accelerator is obtained.

[0041] Furthermore, compared to the traditional NN accelerator method of loading weights and input data from memory and then passing these two parts of data to the matrix multiplication unit, the novel computing architecture proposed in this embodiment only needs to pass in the input data, without needing to pass in the weight matrix data, thus avoiding large-scale data movement.

[0042] This embodiment describes a novel compute-memory fusion architecture design method for neural network accelerators. By designing a compute-memory fusion NN accelerator using a 28nm process library and evaluating it with Cadence Genus 15.0 software, the compute-memory fusion NN accelerator achieves a 43.52%-89.20% reduction in data movement energy consumption compared to a conventional NN accelerator that loads weights from external memory. Compared to similar methods (in-memory computation, near-memory computation), the comparison results are shown in Table 1. This method effectively eliminates some data movement while avoiding the computational performance and accuracy issues associated with using novel storage devices.

[0043] Table 1 Comparison of Data Movement Reduction

[0044]

[0045] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0046] Although this application has been described above with reference to specific embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of this application. In particular, as long as there is no structural conflict, the features in the specific embodiments disclosed in this application can be combined with each other in any way. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, this application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A design method for a novel compute-memory fusion architecture for neural network accelerators, characterized in that, Includes the following steps: S1. Weight and Activation Quantization: Using an open-source weight quantization method, the weights of the neural network and the input of each layer are quantized on a software platform built on the PyTorch open-source framework to obtain a quantized neural network. S2. Design a dedicated single-input multiplier: Based on the quantized neural network obtained in step S1, design a dedicated single-input multiplier for the quantized neural network accelerator. When the quantized NN accelerator performs the multiplication operation between the weights and the input, it uses a multiplier with two fixed-point integer inputs. Fix one input of the multiplier to a specific multiplier to obtain a dedicated single-input multiplier. S3. Weight Analysis: The quantized NN obtained in step S1 is further analyzed on the software platform used in step S1. The number of different elements in the weight matrix is ​​counted to determine the number of different single-input dedicated multipliers used in a single calculation operation. S4. Hardware design of the data path: On the software platform used in step S1, first complete the declaration of the module name, input and output ports in the hardware description file; then use the software platform used in step S1 to re-traverse the weight matrix, generate the corresponding connection statement according to the arrangement of the weights in the weight matrix and the correspondence during matrix multiplication, and write it into the hardware description file. After the weights have been fully traversed, add the statement to end the module creation in the hardware description file to complete the hardware design of the data path. S5. Matrix multiplication operation of the compute-memory NN accelerator: The input data of each layer in the NN is transmitted to the designated single-input dedicated multiplier through the data path constructed in step S4. The result of the multiplication operation is then accumulated by the adder according to the operation rules of matrix multiplication, and finally the result of the matrix multiplication operation of the compute-memory NN accelerator is obtained.

2. The design method of a novel compute-memory fusion architecture for neural network accelerators according to claim 1, characterized in that, The matrix multiplication operation of weights and inputs in the quantized neural network in step S1 internally involves multiplication operations between a low number of specific points.

3. The design method for a novel compute-memory fusion architecture for neural network accelerators according to claim 2, characterized in that, The range of multipliers in the single-input dedicated multiplier in step S2 depends on the number of bits used to quantize the weights in step S1.

4. The design method of a novel compute-memory fusion architecture for neural network accelerators according to claim 3, characterized in that, The method for counting the number of different elements in the weight matrix in step S3 is to traverse all elements in the weight matrix and increment the corresponding count variable by 1 according to the value of the element.

5. The design method of a novel compute-memory fusion architecture for neural network accelerators according to claim 4, characterized in that, In step S3, when the weights of the NN are the weights of the fully connected layer, a large weight matrix is ​​used for the entire layer, and the weight matrix is ​​analyzed column by column. In step S3, when the weights of the NN are in the form of a convolutional layer, each input channel corresponds to a convolutional kernel, and the weights of the convolutional layer are analyzed channel by channel.

6. The design method of a novel compute-memory fusion architecture for neural network accelerators according to claim 5, characterized in that, In step S4, the hardware design of the data path implements the use of connections to represent the correspondence from input data to a single-input dedicated multiplier.

Images