High-parallelism storage architecture for neural network accelerator

Abstract

The invention discloses a high-parallelism storage architecture for a neural network accelerator. The high-parallelism storage architecture is characterized in that a first-layer input module of a convolutional neural network receives original picture data; the input data module is responsible for processing original data and converting the original data into a form suitable for neural network processing; data provided by the input data module is used as input of the weight module, and the weight module extracts picture features useful for tasks. An input data module is divided into a partition group, a unit group and a basic unit, and a hierarchical structure of a weight module is divided into a weight module partition group and a weight module basic unit; and at the same time, splitting the first layer input of the convolutional neural network. According to the method, a single input channel can be split into a plurality of channels, so that the computing parallelism of the first layer is improved, and the waste of computing resources and storage resources is avoided. According to the invention, a multi-unit multi-partition data storage technology is adopted, and input image data of different channels and different sizes can be quickly accessed.

Description

Technical Field

[0001] The present invention belongs to the technical field of neural network and relates to a high parallelism storage architecture for a neural network accelerator. Background Art

[0002] Artificial intelligence has penetrated into every aspect of our daily lives and has become the cornerstone of modern society. It plays a key role in many fields such as speech recognition, autonomous driving, smart home, and medical diagnosis. Among them, the convolutional neural network (CNN) is a deep learning model that has shown excellent performance in image and video recognition, classification, and feature extraction. It is usually composed of layers such as convolutional layers, activation layers, and pooling layers. In convolutional neural networks, convolutional layers occupy a dominant position, and their proportion is usually more than 90%. Through its core component, the convolutional layer, the convolutional neural network (CNN) uses three-dimensional weights as sliding windows, moves on the input data with a certain step size S, and performs three-dimensional convolution operations with the three-dimensional input feature submatrices in the window, respectively, so that a large amount of data on the input data needs to be repeatedly involved in the operation, and the time of participating in the operation is not necessarily continuous. Therefore, the neural network not only has a large amount of calculation, but also a large amount of memory access operations.

[0003] The high data throughput and frequent data access of convolution calculations pose a higher challenge to the storage architecture inside the neural network accelerator. It requires the storage architecture to be able to flexibly access input data of different channels and different sizes while avoiding repeated reading of data. In addition, the first-layer input of a convolutional neural network usually includes a grayscale image of 1 channel or a color image of 3 channels. If the number of channels is not distinguished and they are processed uniformly, it will lead to a huge waste of storage resources. How to design the storage architecture so that the data storage has a higher utilization rate at the first layer of the storage architecture and improve the computing efficiency of the convolutional neural network is an urgent problem to be solved. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that the storage architecture of the neural network in the prior art cannot meet the requirements of neural network calculation and memory access operations, and to provide a high-parallelism storage architecture for a neural network accelerator.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions: A high-parallelism storage architecture for a neural network accelerator, characterized in that it includes: a first-layer input module of a convolutional neural network, a weight module and an input data module, wherein the first-layer input module of the convolutional neural network receives original image data or preprocessed image data; the input data module is responsible for processing the original data and converting it into a form suitable for neural network processing; the data provided by the input data module is used as input to the weight module, and the weight module is used to continuously adjust the parameters of the weight module to extract image features useful for the task.

[0006] Preferably, the number of channels of the weight module matches the number of channels of the input data module.

[0007] Preferably, the weight module extracts image features useful for the task, specifically by using M three-dimensional weights of size K×K×N to extract features from three-dimensional input data of size H×W×N to obtain output data of size R×C×M.

[0008] Preferably, the method uses M three-dimensional weights of size K×K×N to extract features from three-dimensional input data of size H×W×N to obtain output data of size R×C×M, specifically: Each three-dimensional weight slides on the input data with a step size of S, in the order of first left to right and then from top to bottom, and performs convolution operations with the three-dimensional input data sub-vectors constituted by the data in the filter window to obtain an output two-dimensional input data of size R×C, which is spliced ​​into three-dimensional output input data; where M is the output channel, K is the weight size, N is the input channel, H and W are the rows and columns of the input data respectively, and R and C are the rows and columns of the output image respectively.

[0009] Preferably, the input data module includes a partition group, a unit group and a basic unit, the partition group includes Y unit groups; each unit group includes a plurality of basic units, and the horizontal unit group of each basic unit can store data, used to store W-dimensional data.

[0010] Preferably, the unit group comprises data, of which, It means that the data of the first input channel in the first row and first column of the input data is stored. It means that the data of the first input channel in the first row and second column of the input data is stored, and so on. data; , ,..., The data of the dimensions are stored in the corresponding cell groups. , ,..., The data is stored in the corresponding partition group.

[0011] Preferably, the hierarchical structure of the weight module includes a weight module partition group and a weight module basic unit; the weight module partition group includes a plurality of weight module basic units; if the horizontal unit group size of the weight module basic unit can store data, where the cell group contains The data is divided into Q groups according to the different weights K. The weight module can provide Q Parallel processing in P dimensions; where P is the number of weight module partition groups.

[0012] Preferably, the first layer input of the convolutional neural network includes a grayscale image with 1 channel or a color image with 3 channels.

[0013] Preferably, the first layer input of the convolutional neural network is split, and the specific process is as follows: The input data of the first layer is split into L channels or more; the input data module cuts the single-layer output before the first layer convolution into L parts along the H direction, divides the output of the next layer by L in the H direction, obtains L equal-height output results, and then splits the input data of the first layer according to the position of the corresponding input data before convolution; each basic unit of the input data stores the result of the corresponding channel, and the first unit of the partition group stores data according to specific rules to ensure that when the number of rows exceeds X, it is stored in the order of 1st, X+1, 2X+1..., thereby completing the splitting of the first layer of channels.

[0014] Preferably, the input data module is loaded in a streaming loading mode. Assuming that the input data module caches h rows of data and the input data module has H rows of data, the loaded data will be processed with reference to the weight size K, and a threshold threshold is set to determine whether there are K+threshold rows of data in the input data cache. If so, the subsequent calculation module continues to calculate until the data in the input data cache is less than K+threshold rows.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention aims at the storage and computing bottleneck problems in a convolutional neural network accelerator, and proposes a high-parallelism storage architecture for a neural network accelerator. The design significantly improves the performance of the CNN accelerator by optimizing the first-layer channel expansion and multi-dimensional data reuse. The present invention divides the input data module into partition groups, unit groups and basic units, and divides the hierarchical structure of the weight module into weight module partition groups and weight module basic units; at the same time, the first-layer input of the convolutional neural network is split. The present invention can split a single input channel into multiple channels, thereby improving the computing parallelism of the first layer and avoiding the waste of computing resources and storage resources. The present invention adopts a multi-unit multi-partition data storage technology, which can quickly access input image data of different channels and different sizes. This design improves the reuse rate of data, reduces unnecessary data access and calculation, thereby improving computing efficiency, and can support convolution operations of different sizes. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 This is a schematic diagram of the convolution calculation process; Figure 2 It is a schematic diagram of the basic unit corresponding to the input data; Figure 3 Schematic diagram of the basic unit corresponding to the weight; Figure 4 This is a schematic diagram of the storage partitioning of the first-layer data. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0019] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The present invention is further described in detail below in conjunction with the accompanying drawings: The present invention discloses a high-parallelism storage architecture for a neural network accelerator, comprising: a first-layer input module of a convolutional neural network, a weight module and an input data module, wherein the first-layer input module of the convolutional neural network receives original image data or preprocessed image data; the input data module is responsible for processing the original data and converting it into a form suitable for neural network processing; the data provided by the input data module is used as the input of the weight module, and the weight module continuously adjusts the parameters of the weight module to extract image features useful for the task. The invention divides the input data module into partition groups, unit groups and basic units, and the hierarchical structure of the weight module is divided into weight module partition groups and weight module basic units; and the first-layer input of the convolutional neural network is split at the same time. The present invention can split a single input channel into multiple channels, thereby improving the computational parallelism of the first layer and avoiding the waste of computing resources and storage resources.

[0021] See also Figure 1 The weight module extracts image features that are useful for the task. Specifically, it uses M three-dimensional weights of size K×K×N to extract features from three-dimensional input data of size H×W×N to obtain output data of size R×C×M.

[0022] M three-dimensional weights of size K×K×N are used to extract features from three-dimensional input data of size H×W×N to obtain output data of size R×C×M. Specifically, each three-dimensional weight slides on the input data with a step size of S, from left to right and then from top to bottom, and performs convolution operations with the three-dimensional input data sub-vectors composed of the data in the filter window to obtain an output two-dimensional input data of size R×C, which is spliced ​​into three-dimensional output input data; wherein M is the output channel, K is the weight size, N is the input channel, H and W are the rows and columns of the input data, respectively, and R and C are the rows and columns of the output image, respectively.

[0023] The input data module includes a partition group, a cell group and a basic cell, wherein the partition group includes Y cell groups; each cell group includes a plurality of basic cells, and each basic cell transverse cell group can store data, used to store W-dimensional data.

[0024] See also Figure 2 , the unit group contains data, of which, It means that the data of the first input channel in the first row and first column of the input data is stored, and so on. data; , ,..., The data of the dimensions are stored in the corresponding cell groups. , ,..., The data is stored in the corresponding partition group.

[0025] See also Figure 3 The hierarchical structure of the weight module includes a weight module partition group and a weight module basic unit; the weight module partition group includes a plurality of weight module basic units; if the horizontal unit group size of the weight module basic unit can store data, where the cell group contains The data is divided into Q groups according to the different weights K. The weight module can provide Q Parallel processing in P dimensions; where P is the number of weight module partition groups.

[0026] The first layer input of the convolutional neural network includes a grayscale image of 1 channel or a color image of 3 channels. The process of splitting the first layer input of the convolutional neural network is: The input data of the first layer is split into L channels or more; the input data block needs to cut the single-layer output before the first layer convolution into L parts along the H direction. Assuming that the privileged size of the first layer of the network is K×K, the step size is 2, and the original input data has a total of H rows, with pad, and the pad size is recorded as pad_width. Then after adding a circle of padding, it will become H+2pad_width rows, and the right side is the corresponding row number corresponding to L channels; there will be KS repeated parts at each split position; in essence, the output of the next layer is divided by L in the H direction to obtain L equal-height output results, and then the input data of the first layer is split according to the position of the corresponding input data before convolution; each basic unit of input data needs to store the result of one of the channels, and the first unit group of each partition group stores 1-4 rows. For the first unit group of all partition groups, one row of data will be stored respectively, that is, 1-X rows. If there is a situation where the number of rows exceeds X, the first unit group of the partition group will store the 1st, X+1, 2X+1 and subsequent data, thereby completing the splitting of the first layer of channels. The present invention adopts multi-unit multi-partition data storage technology, which can quickly access input image data of different channels and different sizes. This design improves the reuse rate of data, reduces unnecessary data access and calculation, thereby improving calculation efficiency, and can support convolution operations of different sizes.

[0027] When the number of input channels of the first layer is greater than 1, each input channel is processed according to the above block method. For example, when the number of input channels is 3, the input data of each input channel is split into 8 channels, for a total of 24 input channels.

[0028] The input data module is loaded in streaming mode. The neural network accelerator includes a controller, a clock control module, a DDR memory access module, a cache control module, a PE computing array, an input weight cache and other modules. Due to the limited internal storage modules of the accelerator, in order to avoid data rereading caused by input data segmentation, a streaming processing method is adopted; assuming that the input data module caches h rows of data and the input data module has H rows of data, the loaded data will be processed with reference to the weight size K, and a threshold threshold is set to determine whether there are K+threshold rows of data in the input data cache. If so, the subsequent computing module continues to calculate until the data in the input data cache is less than K+threshold rows.

[0029] There are different situations in the weight loading process. When the current layer can cache all the weight data, the next layer of weight data can be pre-loaded. When the current layer intelligently caches part of the weight data, the weight can be updated while calculating.

[0030] See also Figure 4 , taking L=8, the first layer weight is 3×3 and the step size is 1, as an example to show the way of splitting the input data. The gray part is the padding of the first layer, and the white part is the original input data. Assuming that the original input data has 16 rows, it will become 18 rows after adding a circle of padding. The right side is the corresponding row number corresponding to 8 channels. In this way, there will be two repeated rows at each split position. In essence, it is to divide the output H direction of the next layer by 8 to obtain 8 equal-height output results, and then split the input data of the first layer according to the position of the corresponding input data before the convolution.

[0031] Aiming at the problem of a small number of input channels in the first layer of CNN, the present invention proposes a channel expansion technology. This technology can split a single input channel into 8 or more channels, thereby improving the computational parallelism of the first layer and avoiding the waste of computing resources and storage resources.

[0032] 2) The convolution layer generates a large amount of reused data during the operation process. The present invention adopts multi-unit multi-partition data storage technology, which can quickly access input image data of different channels and different sizes. This design improves the reuse rate of data, reduces unnecessary data access and calculation, thereby improving the calculation efficiency, and can support convolution operations of different sizes; The terminal device provided in an embodiment of the present invention. The terminal device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor executes the computer program, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0033] The computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to accomplish the present invention.

[0034] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0035] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0036] The memory may be used to store the computer program and / or module, and the processor implements various functions of the terminal device by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory.

[0037] If the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0038] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0039] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0040] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0041] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A high-parallelism storage architecture for a neural network accelerator, characterized in that: It includes: a first-layer input module of a convolutional neural network, a weight module and an input data module, wherein the first-layer input module of the convolutional neural network receives original image data or preprocessed image data; the input data module is responsible for processing the original data and converting it into a form suitable for neural network processing; the data provided by the input data module is used as the input of the weight module, and the weight module is used to continuously adjust the parameters of the weight module to extract image features useful for the task.

2. The high parallelism storage architecture for a neural network accelerator according to claim 1, characterized in that: The number of channels of the weight module matches the number of channels of the input data module.

3. The high parallelism storage architecture for a neural network accelerator according to claim 1, characterized in that: The weight module extracts image features useful for the task, specifically: using M three-dimensional weights of size K×K×N, extracting features from three-dimensional input data of size H×W×N, and obtaining output data of size R×C×M.

4. The high parallelism storage architecture for a neural network accelerator according to claim 3, characterized in that: The method uses M three-dimensional weights of size K×K×N to extract features from three-dimensional input data of size H×W×N to obtain output data of size R×C×M, specifically: Each three-dimensional weight slides on the input data with a step size of S, in the order of first left to right and then from top to bottom, and performs convolution operations with the three-dimensional input data sub-vectors constituted by the data in the filter window to obtain an output two-dimensional input data of size R×C, which is spliced ​​into three-dimensional output input data; where M is the output channel, K is the weight size, N is the input channel, H and W are the rows and columns of the input data respectively, and R and C are the rows and columns of the output image respectively.

5. The high parallelism storage architecture for a neural network accelerator according to claim 1, characterized in that: The input data module includes a partition group, a cell group and a basic cell. The partition group includes Y cell groups. Each cell group includes a plurality of basic cells. The horizontal cell group of each basic cell can store data, used to store W-dimensional data.

6. The high parallelism storage architecture for a neural network accelerator according to claim 5, characterized in that: The unit group comprises of data, among which, It means that the data of the first input channel in the first row and first column of the input data is stored. It means that the data of the first input channel in the first row and second column of the input data is stored, and so on. data; , ,..., The data of the dimensions are stored in the corresponding cell groups. , ,..., The data is stored in the corresponding partition group.

7. The high parallelism storage architecture for a neural network accelerator according to claim 6, characterized in that: The hierarchical structure of the weight module includes a weight module partition group and a weight module basic unit; one weight module partition group includes a plurality of weight module basic units; If the horizontal unit group size of the basic unit of the weight module can store data, where the cell group contains The data is divided into Q groups according to the different weights K. The weight module can provide Q Parallel processing in P dimensions; where P is the number of weight module partition groups.

8. The high parallelism storage architecture for a neural network accelerator according to claim 7, characterized in that: The first layer input of the convolutional neural network includes a grayscale image with 1 channel or a color image with 3 channels.

9. The high parallelism storage architecture for a neural network accelerator according to claim 8, characterized in that: The first layer input of the convolutional neural network is split. The specific process is: The input data of the first layer is split into L channels or more; the input data module cuts the single-layer output before the first layer convolution into L parts along the H direction, divides the output of the next layer by L in the H direction, obtains L equal-height output results, and then splits the input data of the first layer according to the position of the corresponding input data before convolution; each basic unit of the input data stores the result of the corresponding channel, and the first unit of the partition group stores data according to specific rules to ensure that when the number of rows exceeds X, it is stored in the order of 1st, X+1, 2X+1..., thereby completing the splitting of the first layer of channels.

10. The high parallelism storage architecture for a neural network accelerator according to claim 8, characterized in that: The input data module is loaded in a streaming loading mode. Assuming that the input data module caches h rows of data and has H rows of data, the loaded data will be processed with reference to the weight size K. A threshold threshold is set to determine whether there are K+threshold rows of data in the input data cache. If so, the subsequent calculation module continues to calculate until the data in the input data cache is less than K+threshold rows.

Images