Neural network acceleration device and method based on streaming computation

Abstract

The invention provides a neural network acceleration device and method based on streaming computing, and the device is configured as follows: according to the structure of a multi-layer neural network, a layer module used for processing a task of the layer is constructed for each layer of the multi-layer neural network, and each layer module processes data through vector granularity; a vector inter-module assembly line is established between every two layer modules, and the assembly line is configured as follows: after the upstream layer module completes correlation calculation of a part of vectors and the downstream layer module is prepared to receive an intermediate result corresponding to the part of vectors, the intermediate result is immediately transmitted between the two layer modules; according to the device, on the premise that the calculation precision is guaranteed, calculation delay, data transmission delay and resource waste are remarkably reduced through innovative optimization of a hardware architecture and a calculation process, so that the overall calculation performance and the resource utilization rate are greatly improved, and finally the energy efficiency ratio of the acceleration device is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, specifically to the field of neural network hardware acceleration technology. More specifically, it relates to a neural network acceleration device and method based on streaming computing. Background Art

[0002] Current hardware accelerators mainly fall into two implementation methods:

[0003] (1) Single Instruction Multiple Data (SIMD) computing mode based on GPU

[0004] This mode uses a single instruction to parallel process multiple data units, which is suitable for large-scale parallel computing tasks and has high flexibility. However, due to the need for complex control logic to coordinate each computing unit, the GPU architecture often faces problems of large on-chip resource consumption and high power consumption. Especially when performing compute-intensive tasks such as matrix multiplication, the complexity of the control logic and the computing latency will significantly affect the overall performance of the system.

[0005] (2) Systolic Array computing mode based on TPU

[0006] The systolic array computing mode relies on the systolic array architecture to complete. It has been widely used in deep learning acceleration with its efficient matrix multiply-accumulate ability, especially suitable for large-scale matrix operations. However, when performing neural network calculations with unbatched data input, large matrix dimensions, and a large aspect ratio difference, the systolic array faces the following two major challenges:

[0007] Low computing performance and high latency: When performing matrix operations, the systolic array requires a large amount of time to load the weight matrix and empty the calculation results. This process usually occupies a large part of the entire computing process (more than 50%), resulting in the computing unit being idle during the waiting period, which greatly affects the efficient use of computing resources.

[0008] Low resource utilization rate: Due to the long injection and emptying time during the matrix calculation process, the systolic array does not run at full load continuously during the calculation process, and the utilization efficiency of computing resources is significantly reduced.

[0009] Matrix Multiply-Accumulate (MAC) is one of the most core computing processes in neural networks, especially widely used in operations such as fully connected layers and convolutional layers. During the training and inference processes of deep neural networks, matrix multiply-accumulate is used to calculate the weighted sum between neurons. It not only determines the output of each layer of the neural network but also largely determines the computing complexity and resource consumption of the network.

[0010] In large-scale neural networks, especially deep learning models, matrix multiplication and accumulation operations often involve multiplication and addition of high-dimensional matrices. These operations are computationally intensive and typically require a large number of computing cycles. Traditional hardware architectures, such as GPUs and TPUs, can accelerate this process, but due to bottlenecks they face when performing matrix multiplications of specific sizes (such as the loading and draining time of the systolic array), overall computing performance falls short of ideal levels.

[0011] In particular, in applications such as deep learning-based molecular dynamics simulation models (Deep Potential Molecular Dynamics, or DeePMD), the matrices involved in neural networks are large in dimension and have widely varying aspect ratios. This makes traditional matrix multiplication operations more resource-intensive and latency-intensive. These issues prevent traditional architectures from providing efficient computing support in such applications.

[0012] Therefore, existing hardware architectures have deficiencies in computing performance and resource utilization.

[0013] It should be noted that this background information is intended solely to introduce relevant information related to the present invention to facilitate understanding of the present invention's technical solution. It does not necessarily constitute prior art. Relevant information submitted and disclosed together with the present invention's solution should not be considered prior art unless there is evidence that the relevant information was disclosed prior to the filing date of the present invention. Summary of the Invention

[0014] Therefore, the purpose of the present invention is to overcome the above-mentioned defects of the prior art and provide a neural network acceleration device and method based on streaming computing.

[0015] The purpose of the present invention is achieved through the following technical solutions:

[0016] According to a first aspect of the present invention, a neural network acceleration device based on streaming computing is provided, which is configured to: construct a layer module for processing the tasks of each layer of the multi-layer neural network according to the structure of the multi-layer neural network, wherein each layer module processes data at a vector granularity; establish a vector-level inter-module pipeline between every two layer modules, and the pipeline is configured to: when the upstream layer module completes the relevant calculation of part of the vector and the downstream layer module is ready to receive the intermediate result corresponding to the part of the vector, the intermediate result is transmitted between the two layer modules in real time.

[0017] Optionally, each layer module includes: multiple computing units for performing calculations at the vector granularity, and completing relevant calculations by obtaining corresponding vectors pre-split from the input, feature map, or weights of the neural network; an arbitration unit for weights to select the weights required for calculation from the initial weights and the cache within the layer; a cache within the layer for storing weights that need to be recycled; an arbitration unit for intermediate results to transfer the intermediate results to be summed from the corresponding computing units to the accumulation unit; an accumulation unit for completing the accumulation or bias addition of the intermediate results; and an activation function unit for processing the result obtained by the accumulation unit according to the activation function required in the multi-layer neural network.

[0018] Optionally, the device is further configured to: set a cache between layers between layer modules for caching data dependent between layers, where when the calculation process of a layer module has the same task as the upstream layer module, directly read the result of the same task from the cache between layers.

[0019] Optionally, when the device is configured as a training device for a multi-layer neural network, simultaneously construct a forward layer module and its corresponding backward layer module, and establish a cache between layers between the two, where when a backward layer module needs to reuse the result calculated by the corresponding forward layer module, directly read the result from the cache between the two layers.

[0020] Optionally, the device is configured to: obtain a parameterized configuration file, and adjust the parallelism of the layer module, the hierarchical calculation process, and the working mode of each computing unit within the layer module according to the configuration file.

[0021] Optionally, the device is further configured to: dynamically adjust the computing resources and architecture of each layer module according to the computing scale and complexity of different layer modules, so that the computing performance of different layer modules matches each other.

[0022] Optionally, the computing functions executed by each layer module include: vector matrix multiplication, element multiplication, bias addition, matrix addition, and activation function calculation, where the required computing functions are selected and enabled in each layer module according to the computing requirements of the multi-layer neural network.

[0023] Optionally, in the layer module of the device, the computing units are connected in a chain, and the input data of the same layer module is sequentially transmitted from the most upstream to the downstream of the chain connection to complete the sequential utilization of vector data.

[0024] Optionally, the multi-layer neural network deployed in the device comes from a deep learning model.

[0025] According to a second aspect of the present invention, there is provided a neural network acceleration method based on stream computing, the method comprising: obtaining a multi-layer neural network to be accelerated, and accelerating the calculation process of the multi-layer neural network by using the device of the first aspect.

[0026] Compared with the prior art, the advantages of the present invention are as follows:

[0027] On the premise of ensuring the calculation accuracy, the device significantly reduces the calculation delay, data transmission delay and resource waste through innovative optimization of the hardware architecture and calculation process, thereby greatly improving the overall calculation performance and resource utilization rate, and finally improving the energy efficiency ratio of the acceleration device. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The following further describes embodiments of the present invention with reference to the accompanying drawings, wherein:

[0029] Figure 1 FIG. is a system schematic diagram of a neural network acceleration device for stream computing according to an embodiment of the present invention;

[0030] Figure 2 FIG. is an internal structure schematic diagram of a layer module according to an embodiment of the present invention;

[0031] Figure 3 FIG. is a working process schematic diagram of the device according to an embodiment of the present invention;

[0032] Figure 4 FIG. is a process schematic diagram of loading initial data in the device according to an embodiment of the present invention;

[0033] Figure 5 FIG. is a calculation process schematic diagram of the device according to an example of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0034] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings through specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0035] As mentioned in the background art section, existing hardware architectures have deficiencies in terms of computing performance and resource utilization. In response to this, the present invention provides a neural network acceleration device based on stream computing, which adopts a brand-new stream computing neural network hardware acceleration architecture and aims to efficiently process the computing tasks of an N-layer neural network (N > 2). The device of the present invention adopts a fine-grained pipeline design, refines the data processing method according to the vector granularity, and constructs an inter-module pipeline at the vector level between layer modules, enabling seamless docking of the calculations between the neural network acceleration module and the front and rear modules. When the upstream layer module completes the relevant calculations of part of the vectors and the downstream layer module is ready to receive the intermediate results corresponding to these part of the vectors, the intermediate results are immediately transmitted between the two layer modules; thus, it allows the data to enter the next layer immediately after completing part of the calculations, avoiding the redundant delay of "waiting for all layers to complete before transmitting" in the traditional method, thereby achieving efficient data circulation and higher throughput. On the premise of ensuring the computing accuracy, through the innovative optimization of the hardware architecture and the computing process, the device significantly reduces the computing delay, data transmission delay, and resource waste, thereby greatly improving the overall computing performance and resource utilization, and ultimately improving the energy efficiency ratio of the acceleration device.

[0036] According to an embodiment of the present invention, a neural network acceleration device based on stream computing is provided. The device is configured to: according to the architecture of a multi-layer neural network, construct layer modules for processing the tasks of each layer of the multi-layer neural network respectively, where each layer module processes data at the vector granularity; between adjacent layer modules, establish an inter-module pipeline at the vector level, and the pipeline is configured to: when the upstream layer module completes the relevant calculations of part of the vectors and the downstream layer module is ready to receive the intermediate results corresponding to these part of the vectors, immediately transmit the intermediate results between the two layer modules. The neural network acceleration device is constructed using a programmable logic unit (FPGA) or an ASIC chip. The device processes data at the vector granularity, such as an input vector data stream of 1×n (n can be 4, 8, 16, etc.). The layer module can support stream computing acceleration with different vector widths by configuring the number of computing units according to the usage requirements. Each layer module performs parallel matrix operations through computing units and realizes data caching and cross-layer data stream transmission in combination with a cache (FIFO). Due to the setting of the inter-module pipeline, by setting a neural network layer module that supports vector matrix multiplication, stream computing acceleration of the neural network can be realized on the hardware with a relatively high resource utilization rate, reducing the waiting time of the initial data, the data injection and evacuation time during the calculation, and realizing processing time hiding from the perspective of the pipeline. In addition, for the matrix blocking problem existing in neural network models based on deep learning (such as DeePMD), the present invention uses vector multiplication instead of traditional matrix multiplication, making each calculation finer-grained and more flexible, thereby further reducing the computing delay and enhancing the resource utilization rate of the computing unit.

[0037] According to an embodiment of the present invention, the device is further configured to: provide an inter-layer cache between layer modules for caching inter-layer dependent data, wherein when there is the same task between the computing process of a layer module and the upstream layer module, directly read the result of the same task from the inter-layer cache. In terms of data caching, the present invention provides an inter-layer cache (FIFO) between layers to cache inter-layer dependent data; at the same time, by means of circularly storing data such as weights in the intra-layer cache (local cache), the data supply of the system can be realized more efficiently, greatly simplifying the system design and meeting the high-throughput computing requirements of the system.

[0038] According to an embodiment of the present invention, when the device is configured as a training device for a multi-layer neural network, a forward layer module and its corresponding backward layer module are simultaneously constructed, and an inter-layer cache is established between the two to cache the result of the forward layer module calculation. When a backward layer module needs to reuse the result of the corresponding forward layer module calculation, directly read the result from the inter-layer cache between the two. Compared with the GPU architecture that needs to recalculate the data passed forward during the backpropagation stage, the device directly stores the result of the forward calculation during the forward calculation and directly reuses these forward calculation results during the backward calculation. Thereby, the amount of calculation is reduced, the calculation performance is significantly improved, and the power consumption is reduced at the same time.

[0039] According to an embodiment of the present invention, the computing functions executed by each layer module include: vector matrix multiplication, element multiplication, bias addition, matrix addition, and activation function calculation (including forward activation function calculation and backward activation function multiplication), wherein according to the computing requirements of the multi-layer neural network, the required computing functions are selected and enabled in each layer module. To adapt to different neural networks, the device of the present invention can, according to the needs of different neural networks, select whether to enable the relevant computing functions in different layer modules through parameter configuration.

[0040] See Figure 1 , which shows a schematic neural network acceleration device for stream computing, or a hardware architecture for neural network acceleration, which is particularly suitable for deep learning tasks in a stream computing environment. The device includes a layer 0 module - layer N module, a backward layer N module - layer 0 module, and an inter-layer cache. In addition, through parametric configuration, users can flexibly adjust the parallelism of the module, the hierarchical computing process, and the working mode of each computing unit according to specific application scenarios and computing requirements, so as to optimize the overall computing efficiency. Taking the acceleration of a three-layer fully connected neural network as an example, the architecture includes the following structures:

[0041] Layer 0 module: Responsible for completing the forward calculation task of the first layer in a three-layer fully connected neural network with data dependencies, mainly including vector matrix multiplication, bias addition, and activation function calculation of FP32 data.

[0042] Layer 1 module: Responsible for completing the forward calculation task of the second layer in a three-layer fully connected neural network with data dependencies, mainly including vector matrix multiplication, bias addition, matrix addition, and activation function calculation of FP32 data.

[0043] Layer 2 module: Responsible for completing the forward calculation task of the third layer in a three-layer fully connected neural network with data dependencies, mainly including vector matrix multiplication, bias addition, matrix addition, and activation function calculation of FP32 data.

[0044] Layer 2g (backward) module: Responsible for completing the backward calculation task of the third layer in a three-layer fully connected neural network with data dependencies, mainly including vector matrix multiplication, element multiplication (multiplication of idt parameter), backward activation function multiplication, bias addition, matrix addition, and activation function calculation of FP32 data.

[0045] Layer 1g (backward) module: Responsible for completing the backward calculation task of the second layer in a three-layer fully connected neural network with data dependencies, mainly including vector matrix multiplication, bias addition, matrix addition, and activation function calculation of FP32 data.

[0046] Layer 0g (backward) module: Responsible for completing the backward calculation task of the first layer in a three-layer fully connected neural network with data dependencies, mainly including vector matrix multiplication, bias addition, matrix addition, and activation function calculation of FP32 data.

[0047] Various caches (FIFO): Used to store initial data (weights, biases, idt parameters, etc.), and complete the transfer of inter-layer dependent data.

[0048] The above six layer modules are responsible for completing the forward and backward calculations of 6 layers in a three-layer fully connected neural network, while the cache is responsible for data storage, input, and transfer. Each layer module adopts a homogeneous design, and each layer module includes a computing unit, an arbitration unit (Mux), an accumulation module, and an activation function unit. The overall structure of the layer module is schematically described below.

[0049] According to an embodiment of the present invention, refer to Figure 2 , each layer module includes:

[0050] Multiple computing units, which are used to perform calculations at the vector granularity and complete relevant calculations by obtaining corresponding vectors pre-split from the input, feature map, or weights of the neural network; the computing units are connected in a chain, and the input data of the same-layer modules is sequentially transmitted from the most upstream to the downstream of the chain connection; the computing units are responsible for completing vector matrix multiplication, transmitting the results to the arbitration unit of the intermediate result, and transmitting data vectors (i.e., input data, which can be the original input or intermediate result) to the computing units downstream of the same layer (downstream refers to Figure 2 the right side in

[0051] The arbitration unit for weights is used to select the weights required for calculation from the initial weights and the cache within the layer.

[0052] The cache within the layer is used to store the weights that need to be recycled.

[0053] The arbitration unit for intermediate results is used to transmit the intermediate results that need to be summed from the corresponding computing units to the accumulation unit; this unit is responsible for selecting the corresponding results to enter the accumulation unit through the handshake signals with different computing units during the vector matrix multiplication process to achieve the accumulation of results.

[0054] The accumulation unit is used to complete the accumulation of intermediate results or bias addition.

[0055] The activation function unit is used to process the results obtained by the accumulation unit according to the activation functions required in the multi-layer neural network.

[0056] Among them, the computing unit, the accumulation unit, and the activation function unit are composed of IP core modules responsible for calculation, which are responsible for completing addition, subtraction, multiplication, accumulation, and multiply-accumulation operations in relevant data formats (such as FP16, FP32, etc.).

[0057] In addition, the arbitration unit for intermediate results can be a multi-layer structure. For example: a 2-layer or 3-layer structure. Taking the arbitration of 240 to 1 and a 3-layer structure as an example: from top to bottom, the first layer is composed of 30 8-to-1 arbitrators, the second layer is composed of 5 6-to-1 arbitrators, and the third layer is composed of 1 5-to-1 arbitrator. According to the different amounts of data selection, the implementer can also use other numbers of layers to construct as needed.

[0058] According to an embodiment of the present invention, valid-ready handshake signals are used to connect the upstream and downstream units directly connected within the layer module, and valid-ready handshake signals are also used to connect the directly connected upstream and downstream layer modules to control the data transfer process of the pipeline through the handshake signals. Only when the valid signal of the output data of the previous module and the ready signal for receiving the input data of the subsequent module are successfully handshaken can the data transfer start. In the entire system, considering the computational requirements of the system-level pipeline, valid-ready handshake signals are used to connect relevant units and between layer modules to ensure the correct connection of handshake signals for all modules in the case of one-to-many or many-to-one module handshakes. At the same time, considering the different computational requirements of different neural networks, the basic computing units in the system can be configured and selected to adapt to the accelerated computing of neural networks with different scales and different parallelism requirements.

[0059] According to an embodiment of the present invention, for the problem that the injection and evacuation time ratio is too high in the traditional systolic array in the calculation within each layer module, the present invention adopts a single-layer pipelined systolic line. By optimizing the organization of the data stream and the weight stream, the computing unit is immediately ready for the next task after completing one calculation, reducing the idle cycle and improving the resource utilization rate. Between layer modules, a single-layer pipeline is used to avoid the injection and evacuation problems of the systolic array: among them, the calculation process in a single layer module includes: the input data gradually completes the calculation of a column of the weight matrix through multiple computing units, and the calculated intermediate result is immediately transferred to the next layer module; cross-layer caching mechanism: cross-layer data is temporarily stored through the cache (FIFO) between layers to achieve parallel computing and data stream synchronization.

[0060] According to an embodiment of the present invention, the device is further configured to: dynamically adjust the computing resources and architectures of each layer module according to the computing scale and complexity of different layer modules, so that the computing performances of different layer modules match each other. For example: for the diversity of different neural network models and computing processes, parametric design and / or configuration are adopted to support multi-way parallel computing and flexible configuration of different computing units and their computing processes. For example: dynamically adjust the number of computing units, the computing process and / or the arbitration logic. The device realizes configurable numbers and types of computing units, can flexibly meet the computing requirements of different neural networks for data of different scales and different formats; can adjust the computing resources of different levels as needed, support the multiplication-accumulation tree structure of different dimensions, and significantly improve the efficiency of large-scale matrix operations; through dynamic adjustment, it can also ensure that the computing performances of each layer match each other and reduce the "blocking" phenomenon of the pipeline caused by uneven performance.

[0061] The following schematically introduces the working process of the device, as Figure 3as shown

[0062] Step 301: Load data, including: starting the data loading process and loading parameters such as weights and biases into the corresponding caches;

[0063] Step 302: Determine whether all data has been loaded. If so, proceed to the next step; otherwise, continue loading data into the corresponding caches;

[0064] Step 303: Start the calculation process, including: inputting data and data numbers into the device. Among them, the data enters the 0th layer module, and the data number enters its corresponding cache;

[0065] Step 304: Obtain the first output data, and successively obtain the output data and data numbers in the reverse 0th layer module;

[0066] Step 305: Obtain all output data, pull down the valid signal of the output data port, and the system waits to load new data or start a new calculation process.

[0067] The following takes accelerating a three-layer fully connected neural network as an example to schematically introduce the process of loading initial data in the device, as Figure 4 as shown

[0068] Step 401: After the device is reset and before the calculation process starts, after the valid-ready handshake is successful, load the weights of the 0th layer (layer refers to the layer module, the same below) through the input port of the device;

[0069] Step 402: After the valid-ready handshake is successful, load the weights of the reverse 0th layer through the input port of the device;

[0070] Step 403: After the valid-ready handshake is successful, load the weights of the 1st layer, 2nd layer, reverse 2nd layer, and reverse 1st layer through the input port of the device;

[0071] Step 404: After the valid-ready handshake is successful, load the bias parameters and idt parameters used by the 0th layer, 1st layer, 2nd layer, and reverse 2nd layer through the input port of the device.

[0072] It should be noted that the above process of loading initial data is only for illustration. Without affecting the working efficiency of the device, the loading order of certain weights, bias parameters, and idt parameters can be adjusted according to the needs of the implementer.

[0073] The following takes accelerating a three-layer fully connected neural network as an example to introduce the calculation process in the device, as Figure 5As shown, it gives the pipeline space-time diagram of parallel computing of a data sequence composed of 6 data in 6 layer modules. Among them, after all six layer modules start computing (the gray part in the figure), the entire device starts to run at full load, and the 6 layer modules perform parallel computing.

[0074] Step 501: The data first enters the 0th layer module, and through the units in the 0th layer, operations such as vector matrix multiplication, accumulation, bias addition, element multiplication (multiplication of idt parameters), and activation function calculation are completed. The ID data enters the cache inside the layer and is cached inside this layer module, waiting to be output from the corresponding cache after the handshake of the output port of the device is successful. At the same time, the output data of the 0th layer module enters the corresponding cache for other modules to reuse the data;

[0075] Step 502: The data calculated by the 0th layer module enters the 1st layer module, and through the units in the 1st layer module, operations such as vector matrix multiplication, accumulation, bias addition, element multiplication (multiplication of idt parameters), and activation function calculation are completed. At the same time, in the 1st layer module, the data after activation function calculation enters the corresponding cache module through the output port of the 1st layer module for other modules to reuse the data;

[0076] Step 503: The data calculated by the 1st layer module enters the 2nd layer module, and through the units in the 2nd layer module, operations such as vector matrix multiplication, accumulation, bias addition, element multiplication (multiplication of idt parameters), and activation function calculation are completed;

[0077] Step 504: The data calculated by the 2nd layer module enters the reverse 2nd layer module, and through the units in the reverse 2nd layer module, operations such as vector matrix multiplication, accumulation, bias addition, element multiplication (multiplication of idt parameters), and activation function calculation are completed in the reverse 2nd layer module; at the same time, in the reverse 2nd layer module, the data after bias addition calculation enters the corresponding cache through the output port of the reverse 2nd layer module for other layer modules to reuse the data;

[0078] Step 505: The data calculated by the reverse 2nd layer enters the reverse 1st layer module, and through the units in the reverse 1st layer, operations such as vector matrix multiplication, accumulation, bias addition, element multiplication (multiplication of idt parameters), and activation function calculation are completed in the reverse 1st layer module.

[0079] Step 506: The data calculated by the reverse 1st layer enters the reverse 0th layer module, and through the units in the 0th layer, operations such as vector matrix multiplication, accumulation, and activation function calculation are completed in the reverse 0th layer module.

[0080] To better verify the effect of the device of the present invention, the inventor also conducted a comparative experiment.

[0081] In the experiment, the device of the present invention takes an accelerated three-layer fully-connected neural network as an example to construct a neural network acceleration device based on stream computing (such as Figure 1 ), and the comparison architecture is a neural network acceleration architecture based on a systolic array, where the dimension of the systolic array is 256*256.

[0082] The experimental results are shown in Table 1 below:

[0083] Table 1

[0084] Implementation architecture Number of DSP slices Number of cycles Systolic array 60000+ 44648 The device of the present invention 6000 29187

[0085] It can be seen from Table 1 that compared with the neural network acceleration architecture based on a systolic array, the stream computing architecture adopted by the device of the present invention has achieved a significant improvement in both computing efficiency and resource utilization. Among them, in the FPGA implementation solution, when the input data width is 8×32 bits, the architecture based on a systolic array requires 44,648 cycles and 65,535 DSP slices to complete the computing task. In the stream computing architecture of the device of the present invention, without pipelining with other modules, only 6,000 DSP slices and 29,187 cycles are required to complete the computing task; in the case of pipelining with other modules, the computing cycle is further shortened to about 2,000 cycles. It can be seen that the stream computing architecture of the device of the present invention not only significantly improves the computing efficiency of the neural network acceleration task, but also effectively optimizes the resource occupancy, showing low resource consumption and high computing throughput capacity. Therefore, compared with the traditional systolic array architecture, the stream computing architecture not only has a significant improvement in computing efficiency, but also has obvious advantages in terms of area and resource usage. It should be noted that in the ASIC chip solution, although DSP resources are not used, based on the same principle, the stream computing architecture is still more efficient, resource-saving and area-saving than the systolic array architecture.

[0086] Generally speaking, the devices of some embodiments of the present invention can at least achieve the following at least one beneficial effect:

[0087] (1) Through fine-grained pipeline design and performance matching between data caching and computing units, the architecture of the present invention can significantly improve the utilization rate of computing resources.

[0088] (2) The inter-layer pipeline design enables the data stream to be efficiently transmitted between each computing layer, avoiding waiting and idle cycles in the traditional computing architecture.

[0089] (3) The reasonable allocation of computing resources and storage resources further reduces power consumption and chip area, while improving the overall computing performance, ensuring that efficient neural network computing tasks can still be completed in a resource-constrained environment.

[0090] It should be noted that although the above steps are described in a specific order, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order, as long as the required functions can be achieved.

[0091] The present invention can be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium having thereon computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0092] The computer-readable storage medium can be a tangible device that retains and stores instructions for use by an instruction execution device. The computer-readable storage medium may include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punched card or raised structures in grooves having instructions stored thereon, and any suitable combination of the foregoing.

[0093] The embodiments of the present invention have been described above. The above description is exemplary and not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art in the technical field without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, the practical application, or the improvement of technology in the market, or to enable other ordinary skill in the art in the technical field to understand the embodiments disclosed herein.

Claims

1. A neural network acceleration device based on stream computing, characterized in that The device is configured to: According to the structure of the multi-layer neural network, layer modules for processing the tasks of each layer of the multi-layer neural network are respectively constructed, wherein each layer module processes data at the vector granularity; Between every two layer modules, an inter-module pipeline at the vector level is established, and the pipeline is configured to: when the upstream layer module completes the relevant calculations of part of the vectors and the downstream layer module is ready to receive the intermediate results corresponding to the part of the vectors, immediately transfer the intermediate results between the two layer modules.

2. The device according to claim 1, characterized in that, Each layer module includes: Multiple computing units for performing calculations at the vector granularity and completing relevant calculations by obtaining the corresponding vectors pre-split from the input, feature map or weights of the neural network; An arbitration unit for weights, which is used to select the weights required for calculation from the initial weights and the cache within the layer; The cache within the layer, which is used to store the weights that need to be recycled; An arbitration unit for intermediate results, which is used to transfer the intermediate results that need to be summed from the corresponding computing units to the accumulation unit; An accumulation unit for completing the accumulation or bias addition of the intermediate results; An activation function unit for processing the results obtained by the accumulation unit according to the activation function required in the multi-layer neural network.

3. The device according to claim 1 or 2, characterized in that, The device is further configured to: set a cache between layers between the layer modules for caching the data dependent between layers, wherein when the calculation process of a layer module has the same task as the upstream layer module, directly read the results of the same task from the cache between layers.

4. The device according to claim 3, characterized in that, When the device is configured as a training device for a multi-layer neural network, the forward layer modules and their corresponding backward layer modules are constructed simultaneously, and a cache between layers is established between them. Among them, when a backward layer module needs to reuse the results calculated by the corresponding forward layer module, directly read the results from the cache between the two layers.

5. The device according to claim 3, characterized in that, The device is configured to: obtain a parameterized configuration file and adjust the parallelism of the layer modules, the hierarchical calculation process, and the working mode of each computing unit within the layer module according to the configuration file.

6. The device according to claim 3, characterized in that, The device is further configured to: Dynamically adjust the computing resources and architectures of each layer module according to the computing scale and computing complexity of different layer modules, so that the computing performances of different layer modules match each other.

7. The device according to claim 3, characterized in that, The computing functions executed by each layer module include: vector matrix multiplication, element multiplication, bias addition, matrix addition, and activation function calculation, wherein, according to the computing requirements of the multi-layer neural network, the required computing functions are selected and enabled in each layer module.

8. The device according to claim 3, characterized in that In the layer modules of the device, the computing units are connected in a chain, and the input data of the same layer module is sequentially transmitted from the most upstream of the chain connection to the downstream to complete the sequential utilization of the vector data.

9. The device according to claim 1 or 2, characterized in that, The multi-layer neural network deployed in the device comes from a deep learning model.

10. A neural network acceleration method based on stream computing, characterized in that, The method includes: Obtain a multi-layer neural network to be accelerated, and use the device according to any one of claims 1-9 to accelerate the calculation process of the multi-layer neural network.

Images