Programmable neural network inference accelerator and method based on memory access

Abstract

The application provides a programmable neural network inference accelerator and method based on storage access. The neural network inference accelerator comprises a data loading module, an instruction distribution module, a data operation module, a data write-back module, and an on-chip storage unit array. The data loading module, the data operation module, and the data write-back module are in one-way or bidirectional data read-write with the on-chip storage unit array. The instruction distribution module distributes the instructions loaded from outside to the remaining modules, and controls the remaining modules to execute or perform corresponding tasks in parallel, so as to complete neural network deployment. The method can decouple the operation module and the data access module, control the module access to the storage by using the generated instructions in a single instruction multi-cycle forced synchronization manner, so as to transfer the data dependency from the modules to the controlled storage addresses, and enhance the programmability and versatility of the accelerator.

Description

Technical Field

[0001] This application relates to the field of neural network algorithms, and in particular to a programmable neural network inference accelerator and method based on memory access. Background Technology

[0002] Artificial Neural Networks (ANNs), also known as Neural Networks (NNs) or Connection Models, are mathematical models that mimic the behavior of animal neural networks to perform distributed parallel information processing. These networks rely on the complexity of the system to adjust the connections between a large number of internal nodes, thereby achieving the purpose of information processing.

[0003] In practical deployment, neural networks require software compilation to break down inference tasks into computational granularities executable by the hardware accelerator. This is done by controlling the accelerator to read, write, and compute data via instructions or configuration information. On the hardware side, this involves designing computational modules for convolution calculations, on-chip storage, data pathways, and corresponding control logic. This typically includes components such as data read / write modules, matrix multiplication modules, vector operation modules, instruction allocation modules, and memory unit control. On the software side, the neural network computation graph needs to be analyzed, and the network needs to be divided into blocks within the constraints of actual hardware resources to meet the computational scale of the accelerator. This segmentation process follows a pre-defined data flow pattern, mapping it to multiple instructions required by the hardware to control the accelerator for neural network inference.

[0004] The design of neural network inference accelerators is crucial for the deployment and application of neural network algorithms on the edge. Neural networks are diverse in type and structure, and current neural network accelerators are typically optimized for a specific network structure, improving hardware computational efficiency through a relatively fixed data flow. In this case, the accelerator operates in a pipelined manner, with tight coupling between modules, making it unfriendly to software programming and difficult to quickly adapt to other types of networks. This rigid hardware-software co-design reduces the flexibility of hardware design, raises the barrier to entry for software developers, and complicates the actual deployment of neural networks. Summary of the Invention

[0005] This application provides a programmable neural network inference accelerator and method based on memory access, in order to solve the problems of high hardware design difficulty, difficult software programming, and narrow network compatibility of current neural network inference accelerators.

[0006] On one hand, this application provides a programmable neural network inference accelerator based on memory access, including a data loading module, an instruction dispatch module, a data processing module, a data write-back module, and an on-chip memory cell array, wherein,

[0007] The data loading module is used to load input data and instruction set, send the instruction set to the instruction distribution module, and write the input data into the on-chip storage unit array. The input data is standard data obtained after segmenting and abstracting the neural network to be deployed. The instruction set includes multiple instructions generated based on the input data. The multiple instructions are arranged in an instruction sequence, and each instruction corresponds to a task.

[0008] The instruction distribution module is used to distribute the instructions in the instruction set to the corresponding data processing module and / or data write-back module, and to control the corresponding module to execute the corresponding task of the instruction according to the instruction sequence;

[0009] The data processing module is configured to receive the instruction, and read the input data from the on-chip storage unit array based on the instruction; perform data processing on the input data, and write the processing result into the on-chip storage unit array;

[0010] The data write-back module is used to receive the instruction, read the calculation result from the on-chip storage unit array based on the instruction, and output the calculation result.

[0011] Optionally, the on-chip storage unit array is used to store the loaded input data and the data generated during data processing. The on-chip storage unit array includes multiple on-chip storage units, including low-width on-chip storage units, high-width on-chip storage units, and output on-chip storage units.

[0012] The low-bit wide-area on-chip storage unit is used to store the input image, network weights, and intermediate feature maps;

[0013] The high-width on-chip storage unit is used to store the bias and a portion thereof;

[0014] The output on-chip storage unit is used to store network operation results and intermediate feature maps.

[0015] Optionally, the on-chip storage unit includes concatenation access logic, split access logic, and multiple static random access memory blocks. The concatenation access logic, the static random access memory blocks, and the split access logic are connected in series. The concatenation access logic is used to provide a write port, and the split access logic is used to provide a read port.

[0016] Optionally, the data processing module includes a computing unit array and a vector computing unit, wherein,

[0017] The computing unit array is used to read the input data from the on-chip storage unit array and perform convolution calculations;

[0018] The vector calculation unit is used to read part of the data from the on-chip storage unit array and perform vector calculations.

[0019] Optionally, the write port of the low-bit wide-chip storage unit is connected to the data loading module and the vector computing unit, and the read port of the low-bit wide-chip storage unit is connected to the computing unit array.

[0020] The write port of the high-bit wide on-chip storage unit is connected to the data loading module, the computing unit array, and the vector computing unit; the read port of the high-bit wide on-chip storage unit is connected to the computing unit array, the vector computing unit, and the data write-back module.

[0021] The write port of the output on-chip storage unit is connected to the data loading module and the vector calculation unit, and the read port of the output on-chip storage unit is connected to the data write-back module.

[0022] Optionally, the data loading module, the instruction dispatch module, the data processing module, and the data write-back module all have an instruction cache unit, wherein,

[0023] The data loading module is used to send the input data to the instruction cache unit corresponding to the instruction distribution module;

[0024] The instruction distribution module is used to distribute the instructions in the instruction set to the instruction cache unit of the corresponding module.

[0025] On the other hand, this application also provides a memory access-based method for accelerating programmable neural network inference, applied to the aforementioned programmable neural network inference accelerator, the method comprising:

[0026] The network layers of the neural network to be deployed are merged to obtain multiple fused layers;

[0027] The fusion layer is divided and abstracted to generate input data, and an instruction set is generated based on the input data. The instruction set includes multiple instructions arranged in an instruction sequence, each instruction corresponding to a task. The input data is the computational granularity that the neural network inference accelerator can execute.

[0028] The data loading module loads the input data and the instruction set, sends the instruction set to the instruction distribution module, and writes the input data into the on-chip storage unit array;

[0029] The instruction distribution module distributes the instructions in the instruction set to the corresponding data processing module and / or data write-back module, and controls the corresponding module to execute the corresponding task according to the instruction sequence;

[0030] The data processing module receives the instruction, reads the input data from the on-chip storage unit array based on the instruction, performs data processing on the input data, and writes the processing result of the data processing into the on-chip storage unit array.

[0031] The data write-back module receives the instruction, reads the calculation result from the on-chip storage unit array based on the instruction, and outputs the calculation result.

[0032] Optionally, the method further includes the step of dividing the fusion layer into blocks:

[0033] Obtain the data scale of the fusion layer;

[0034] Obtain the hardware storage information of the neural network inference accelerator;

[0035] Based on the data scale and the hardware storage information, the fusion layer is divided and abstracted to generate input data.

[0036] Optionally, in the step of generating an instruction set based on the input data, the method further includes:

[0037] Obtain an initial sequence, wherein the initial sequence is the instruction order generated according to the input data;

[0038] Based on the initial sequence, the parallel state of multiple instructions is detected, and the parallel state is used to characterize whether the instructions can be executed in parallel;

[0039] Based on the parallel state, an instruction sequence is determined, and multiple instructions are sorted according to the instruction sequence to obtain the instruction set.

[0040] Optionally, the data processing module includes a computing unit array and a vector computing unit, wherein,

[0041] The computing unit array reads the input data from the on-chip storage unit array and performs convolution calculations;

[0042] The vector computing unit reads a portion of the data from the on-chip storage unit array and performs vector calculations.

[0043] This application provides a programmable neural network inference accelerator and method based on memory access. The neural network inference accelerator includes a data loading module, an instruction distribution module, a data processing module, a data write-back module, and an on-chip memory array. The data loading module, data processing module, and data write-back module all perform unidirectional or bidirectional data read / write operations with the on-chip memory array. The instruction distribution module distributes off-chip loaded instructions to other modules and controls these modules to execute or execute corresponding tasks in parallel to complete the neural network deployment. The method decouples the processing module and the data access module. Through single-instruction multi-cycle forced synchronization, the generated instructions control the module's memory access, thereby transferring data dependencies from between modules to the controlled memory address. This enhances the accelerator's programmability and versatility, ensuring inference efficiency while adapting to various network deployments. Attached Figure Description

[0044] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A schematic diagram of the architecture of a programmable neural network inference accelerator based on memory access provided in an embodiment of this application;

[0046] Figure 2 This is a schematic diagram of the structure of an on-chip storage unit provided in an embodiment of this application;

[0047] Figure 3 This is a schematic diagram of the hardware accelerator data path provided in the embodiments of this application;

[0048] Figure 4 A schematic diagram of the software modeling process provided for embodiments of this application;

[0049] Figure 5 A schematic diagram illustrating the block division process provided in an embodiment of this application;

[0050] Figure 6 This is a flowchart illustrating the generation instructions provided in an embodiment of this application. Detailed Implementation

[0051] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0052] The programmable neural network inference accelerator and method based on memory access provided in this application can be used to quickly deploy a variety of different neural networks. Through a hardware decoupling architecture based on memory access, it achieves hardware-software separation. Furthermore, the modules in the accelerator architecture are completely decoupled and there is no data path between them. When the modules are executed, they are forced to synchronize in multiple cycles with a single instruction. While retaining the simple dependency mechanism of hardware based on instruction control, it also has the programming friendliness of module decoupling, thus solving the problems of high hardware design difficulty, difficult software programming, and narrow network compatibility of neural network inference accelerators.

[0053] See Figure 1 This is a schematic diagram of the architecture of a programmable neural network inference accelerator based on memory access provided in an embodiment of this application. Figure 1 It is understood that the neural network inference accelerator includes a data loading module, an instruction distribution module, a data processing module, a data write-back module, and an on-chip memory array. The data loading module, data processing module, and data write-back module all perform unidirectional or bidirectional data read / write operations with the on-chip memory array.

[0054] The data loading module is used to load input data and instruction set, send the instruction set to the instruction distribution module, and write the input data into the on-chip storage unit array.

[0055] The input data is the standard data obtained after segmenting and abstracting the neural network to be deployed, that is, the computational granularity that the neural network inference accelerator can perform. The data loading module can read input data such as input images, feature maps, weights, biases and partial sums from outside the chip and write them to the on-chip storage unit array to move the input data from outside the chip to on the chip, providing input data for the data operation module.

[0056] The instruction set includes multiple instructions generated based on the input data. These instructions are arranged in an instruction sequence, and each instruction corresponds to a task. The data loading module can load the instruction set from off-chip and write it to the instruction distribution module.

[0057] The instruction distribution module is used to distribute instructions from the instruction set to the corresponding data processing modules and / or data write-back modules, and to control the corresponding modules to execute the tasks corresponding to the instructions according to the instruction sequence. After the neural network inference accelerator is activated and the initialization configuration information is received, the instruction distribution module can control the data loading module to load the instruction set from external sources and write it to the instruction distribution module. After obtaining the instruction set, the instruction distribution module can distribute the instructions in the instruction set to other modules and parse its own instructions. After logical decoding by the instruction distribution module, the instruction sequence is obtained. The instruction sequence includes the execution order of other modules and whether the modules can execute the tasks corresponding to the instructions simultaneously. That is, the instruction corresponding to the instruction distribution module can be used to determine whether other modules start executing instructions simultaneously. After logical decoding by the instruction distribution module, the instruction distribution module can control the remaining modules to execute the tasks corresponding to the instructions according to the instruction sequence in the form of control signals.

[0058] The data processing module is configured to receive the instruction, and read the input data from the on-chip storage unit array based on the instruction; perform data processing on the input data, and write the processing result into the on-chip storage unit array.

[0059] In some embodiments, the data processing module includes a computing unit array and a vector computing unit, wherein the computing unit array is used to perform multiplication and accumulation operations, and the vector computing unit is used to perform various vector operations, such as quantization and activation.

[0060] It is understood that, based on the accelerator framework proposed in this application, the data processing module is not limited to the above-mentioned computing unit array and vector computing unit. Different data processing modules can be replaced to perform dedicated computing for different networks and customized acceleration for different tasks. For example, it can accelerate neural networks including deconvolution, apply sparse optimization to the deployment of edge networks, and quickly generate accelerator prototypes suitable for ASICs or FPGAs.

[0061] The data write-back module is used to receive the instruction, read the calculation result from the on-chip storage unit array based on the instruction, and output the calculation result.

[0062] In some embodiments, the data loading module, computation unit array, vector computation unit, and data write-back unit can be replaced with other modules that have different functions but the same properties. For example, the computation unit array can be replaced with a general systolic array, a matrix multiplication-addition tree, a deconvolution computation array, or a convolution computation array optimized for sparsity. The modules can also be extended, for example, by adding an upsampling module or a pooling module.

[0063] It is understood that the core of this application is that there is no direct data path between modules, but rather an architecture based on storage access. The functions and number of modules can be set according to requirements, and this application does not impose any restrictions on this.

[0064] In some embodiments, such as Figure 1 As shown, the data loading module, instruction distribution module, data processing module, and data write-back module each have a local instruction cache unit (INSN FIFO). The instruction cache unit is used to supply instructions to the corresponding module. Specifically, the data loading module sends the input data to the instruction cache unit corresponding to the instruction distribution module. A control path can exist between the data loading module and the instruction distribution module for writing instructions to the instruction cache unit within the instruction distribution module. The instruction distribution module distributes the instructions in the instruction set to the instruction cache units of the corresponding modules. The instruction cache units are connected to the corresponding modules via control signals and status signals.

[0065] In some embodiments, the instruction dispatch module can perform forced synchronization of single-instruction multi-cycle operations on other modules. When the instruction dispatch module parses its own instructions and obtains one or more other modules whose instructions can be executed, it sends a start signal to the modules that can execute the instructions, thereby controlling the modules to parse the first instruction in their instruction cache units and execute it, and / or to perform parallel operations between modules. When a module completes the task corresponding to the executed instruction, it sends a completion signal back to the instruction dispatch module. After all modules that started working simultaneously have sent completion signals, the instruction dispatch module can determine that the single-instruction multi-cycle synchronization has ended, continue parsing the next instruction, and continue to control the modules that can execute the corresponding instructions to execute the tasks corresponding to the instructions. When all instructions have been executed, one inference task ends. For example, the data loading module can load input data and instruction sets from outside the chip according to the instructions and the control of the instruction dispatch module. The computation unit array can perform multiplication and accumulation operations according to the instructions, the vector computation unit can perform various vector operations according to the instructions, and the data write-back unit can write the results back to outside the chip according to the instructions.

[0066] Understandably, the aforementioned control mechanism can generate instructions through software to control the parallelism between modules without requiring hardware to provide complex and intertwined data and storage dependencies, making software programming more user-friendly and expanding the optimization space for software programming.

[0067] In some embodiments, the on-chip storage unit array is used to store loaded input data and data generated during data processing, and the on-chip storage unit array includes multiple on-chip storage units.

[0068] See Figure 2This is a schematic diagram of the structure of an on-chip storage unit provided in an embodiment of this application. The on-chip storage unit includes CAT SELECT access logic, SPILT SELECT access logic, and a static random access memory (SRAM) block. A single on-chip storage unit internally includes multiple SRAM blocks. The CAT SELECT access logic, the SRAM blocks, and the SPILT SELECT access logic are connected in series. The CAT SELECT access logic provides write ports, and the SPILT SELECT access logic provides read ports. To save bandwidth, the on-chip storage unit can be configured with a limited number of write and read ports through the selection of the SPILT SELECT access logic. The CAT SELECT access logic and the SPILT SELECT access logic can be used to handle non-contiguous addresses and repeated data read / write operations.

[0069] In some embodiments, the on-chip storage unit includes a low-bit-width on-chip storage unit (LOW BITSBUFFER), a high-bit-width on-chip storage unit (HIGH BITSBUFFER), and an output on-chip storage unit (OUTPUTBUFFER). Different on-chip storage units can store different data. Specifically, the low-bit-width on-chip storage unit stores the input image, network weights, and intermediate feature maps; the high-bit-width on-chip storage unit stores the bias and partial sum (psum); and the output on-chip storage unit stores the network computation result and intermediate feature maps.

[0070] See Figure 3 This is a schematic diagram of the hardware accelerator data path provided in this application embodiment. Taking the data processing module as a computing unit array and a vector computing unit as an example, w represents a data read operation and r represents a data write operation. The write port of the low-width on-chip storage unit is connected to the data loading module (LOAD) and the vector computing unit (VCU), and the read port of the low-width on-chip storage unit is connected to the computing unit array (PEA). The write port of the high-width on-chip storage unit is connected to the data loading module (LOAD), the computing unit array (PEA), and the vector computing unit (VCU), and the read port of the high-width on-chip storage unit is connected to the computing unit array (PEA), the vector computing unit (VCU), and the data write-back module (STORE). The write port of the output on-chip storage unit is connected to the data loading module (LOAD) and the vector computing unit (VCU), and the read port of the output on-chip storage unit is connected to the data write-back module (STORE).

[0071] In some embodiments, such as Figure 3As shown, the low-bit wide-chip on-chip storage unit can be configured with two write ports and two read ports. The write ports are configured with an arbiter with two input terminals and two output terminals. The data loading module and the vector calculation unit can be connected to the two input terminals of the arbiter, respectively, and the two output terminals of the arbiter are connected to the low-bit wide-chip on-chip storage unit.

[0072] The high-bit wide-chip on-chip storage unit can be configured with one write port and one read port. The write port is configured with an arbiter with three inputs and one output. The data loading module, the computing unit array, and the vector computing unit can be connected to the three inputs of the arbiter, and one output of the arbiter is connected to the high-bit wide-chip on-chip storage unit. The read port is configured with an arbiter with one input and three outputs. The high-bit wide-chip on-chip storage unit is connected to one input of the arbiter, and the three outputs of the arbiter are connected to the computing unit array, the vector computing unit, and the data write-back module, respectively.

[0073] The output on-chip memory unit can be configured with a write port and a read port. The write port is configured with an arbiter with two inputs and one output. The data loading module and the vector calculation unit are connected to the two inputs of the arbiter, respectively. One output of the arbiter is connected to the output on-chip memory unit.

[0074] like Figure 3 The data path of the hardware accelerator shown below, and the data read / write operations between its modules are illustrated in the table below:

[0075]

[0076]

[0077] The data loading module can read input images, intermediate feature maps, and weights from off-chip storage and write them to low-width on-chip storage units, as well as read biases and partial sums and write them to high-width on-chip storage units. The data loading module is used to move data from off-chip to on-chip, providing input data for the computation unit array and vector computation units. Therefore, the hardware accelerator provides a data path for the input images, weights, biases, and computed intermediate feature maps that will not be altered, as well as for the partial sums that may be modified by the hardware accelerator.

[0078] The computation unit array can read the input image, intermediate feature map, weights, biases, and partial sums from the on-chip storage unit to complete the convolution calculation. The calculation result executed under a single instruction may be the intermediate feature map that has been calculated or only the partial sum of the computation map. It can be seen that the hardware accelerator provides a data path for writing partial sums and writing calculation results.

[0079] The vector computing unit can read parts and data from the on-chip storage unit, perform operations such as quantization and activation, and may obtain an intermediate feature map that completes the calculation, which can be written to the output on-chip storage unit or the low-width on-chip storage unit. It may also recalculate new parts and data, which can be written to the high-width on-chip storage unit.

[0080] The data write-back module can read partial and / or intermediate feature maps from on-chip storage units and write them back off-chip as computation results in order to clear the capacity of on-chip storage units.

[0081] To adapt to the aforementioned programmable neural network inference accelerator, this application also provides a memory access-based programmable neural network inference acceleration method. This method proposes a software programming framework corresponding to the aforementioned hardware framework, which can flexibly extend to various different network types and implement the optimal strategy deployment on the accelerator in the form of instructions. The method includes the following steps:

[0082] The network layers of the neural network to be deployed are merged to obtain multiple fused layers.

[0083] The fusion layer is divided and abstracted into blocks to generate input data, and an instruction set is generated based on the input data.

[0084] The instruction set includes multiple instructions arranged in an instruction sequence, each instruction corresponding to a task, and the input data is the computational granularity that the neural network inference accelerator can execute.

[0085] The data loading module loads the input data and the instruction set, sends the instruction set to the instruction distribution module, and writes the input data into the on-chip storage unit array.

[0086] The instruction distribution module distributes the instructions in the instruction set to the corresponding data processing module and / or data write-back module, and controls the corresponding module to execute the tasks corresponding to the instructions according to the instruction sequence.

[0087] The data processing module receives the instruction, reads the input data from the on-chip storage unit array based on the instruction, performs data processing on the input data, and writes the processing result into the on-chip storage unit array.

[0088] The data write-back module receives the instruction, reads the calculation result from the on-chip storage unit array based on the instruction, and outputs the calculation result.

[0089] The following is a detailed description of the programmable neural network inference acceleration method based on memory access provided in the embodiments of this application.

[0090] The first step is modeling, which mainly includes: inter-layer fusion of neural networks, a block-based strategy based on hardware storage, and abstraction of module black-box behavior. For example... Figure 4 The diagram shown illustrates the modeling process provided in this embodiment of the application. The network layers of the neural network to be deployed are fused to obtain multiple fused layers. Inter-layer fusion of the neural network involves analyzing the structure of the neural network to be deployed, completing processes such as normalization layer fusion, quantization, and pruning. This refines the neural network to contain only hardware-supported computational behaviors, and fuses operations that can be optimized, executed in parallel, and performed adjacently, resulting in a neural network structure based on layers, i.e., fused layers.

[0091] After obtaining the fused network layer, i.e., the fused layer, it is divided into multiple computation blocks. Specifically, based on the fused network layer, the network layer parameters need to be divided into blocks according to the data size of the layer and the limitations of hardware storage to obtain the data size that the hardware accelerator can execute, thus enabling the hardware accelerator to perform the operation. Therefore, as... Figure 5 As shown, when dividing the fusion layer into blocks, the data size of the fusion layer and the hardware storage information of the neural network inference accelerator can be obtained. Based on the data size and the hardware storage information, the fusion layer is divided into blocks.

[0092] The above-described segmentation process can yield different segmentation results depending on the segmentation strategy and hardware resources, and there are also differences in data transfer times, latency, power consumption, etc. Therefore, the segmentation strategy to be selected can be determined according to actual needs.

[0093] After segmentation, the fusion layer can be abstracted based on the computation blocks. That is, based on the hardware-supported computation blocks, the behavior of the hardware modules is abstracted into functions, and the computation block tasks are decomposed and mapped to correspond to the behaviors executed by multiple hardware components to generate input data. The input data is the computational granularity that the neural network inference accelerator can execute after segmentation and abstraction.

[0094] Understandably, software modeling of hardware accelerators is challenging. It's difficult to reasonably abstract the behavior of hardware modules and establish realistic timing-based logic simulations; overly detailed modeling is tantamount to rewriting the underlying hardware logic. In this embodiment, the hardware modules are completely decoupled when executing instructions, and software determines whether parallel execution is possible, making it very user-friendly for software modeling. During software modeling, modules can be treated as black boxes, and their behaviors at the single-instruction level can be abstracted as functions to simulate the hardware computation process in software. This can realistically reflect the state of the hardware during actual execution, and the standard data obtained from software modeling can also serve as a standard answer model for hardware computation, facilitating actual debugging and subsequent development.

[0095] After modeling is completed, the partitioning strategy derived from software modeling and its mapping to the smallest computational unit can be used to generate network deployment instructions. Each module is assigned a specific instruction, which corresponds to a task within that module and can contain one or more specific instructions. The instruction functions corresponding to each module are shown in the table below:

[0096] Module Command Function Instruction distribution module Synchronization instructions Data loading module Load data and / or instructions Computing unit array Multiplication and accumulation Vector Calculation Unit Quantitative activation, pooling, etc. Data write-back module Write back data

[0097] The instruction dispatch module's instruction function is synchronization, controlling how the remaining modules of the hardware accelerator work decoupled and in parallel within a synchronization cycle. The data loading module's instruction function is to move data from off-chip to on-chip; it can move not only data or instructions, but also single instructions. The compute unit array's instruction function is multiplication and accumulation, including reading data from corresponding locations in on-chip memory, performing multiplication and accumulation operations, and writing partial summation / or calculation results back to on-chip memory. The vector computation unit's instruction function is to perform various vector operations, including quantization activation and pooling. The data write-back module's instruction function is to write data from the on-chip cache back to off-chip.

[0098] like Figure 6 The diagram illustrates the instruction generation process provided in this embodiment. An instruction set is generated based on input data. This instruction set includes multiple instructions arranged in an instruction sequence, with each instruction corresponding to a task. Specifically, based on the functions provided by the module, the block-based computation graph is decomposed into multiple instructions for multiple modules, with each instruction representing a task for one module. To ensure the correctness of the computation process, instructions need to be generated sequentially. After sequentially generating the instructions, an initial sequence is obtained, which is the order of instructions generated according to the input data. Based on the initial sequence, the parallel state of multiple instructions is detected. This parallel state characterizes whether the instructions can be executed in parallel. Based on the parallel state, an instruction sequence is determined, and multiple instructions are sorted according to this sequence to obtain the instruction set. That is, it checks whether adjacent instructions sorted according to the initial sequence can be executed in parallel, and rearranges the parallelizable instructions according to their storage operation addresses to generate the instruction set required by the instruction distribution module.

[0099] In some embodiments, when generating instructions sequentially, the order of adjacent instructions can be checked, and a judgment can be made based on port read / write status, storage occupancy, and data validity. For example, before executing a load instruction, it is necessary to determine whether the target write port is idle, the target storage data is invalid, or the storage is empty; before executing a compute instruction, it is necessary to determine whether the target read port is idle and the data in the target storage is valid.

[0100] It should be noted that the instruction-driven method proposed in this paper is based on the following premises: there is no direct data path between modules; instructions are in the smallest unit of static random access memory in on-chip memory cells; during parallel instruction execution, forced synchronization is performed with the completion of all instructions as the end signal; address anti-collision access and port arbitration logic are implemented.

[0101] As can be seen from the above technical solutions, this application provides a programmable neural network inference accelerator and method based on memory access. The neural network inference accelerator includes a data loading module, an instruction distribution module, a data processing module, a data write-back module, and an on-chip memory array. The data loading module, data processing module, and data write-back module can perform unidirectional or bidirectional data read / write operations with the on-chip memory array. The instruction distribution module distributes off-chip loaded instructions to other modules and controls the other modules to execute or execute corresponding tasks in parallel to complete the neural network deployment. The method decouples the processing module and the data access module. Through single-instruction multi-cycle forced synchronization, the generated instructions control the module's access to memory, thereby transferring data dependencies from between modules to the controlled memory address, enhancing the accelerator's programmability and versatility, and adapting to the deployment of various networks while ensuring inference efficiency.

[0102] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.

Claims

1. A storage access based programmable neural network inference accelerator, comprising: The neural network inference accelerator comprises a data loading module, an instruction distribution module, a data operation module, a data write-back module, and an on-chip storage unit array, wherein, The data loading module is configured to load input data and an instruction set, send the instruction set to the instruction distribution module, and write the input data into the on-chip storage unit array, wherein the input data is standard data obtained by splitting and abstracting a neural network to be deployed, and the instruction set comprises a plurality of instructions generated according to the input data, the plurality of instructions are arranged in an instruction sequence, and each instruction corresponds to a task; The instruction distribution module is configured to distribute the instructions in the instruction set to corresponding data operation modules and / or data write-back modules, and control the corresponding modules to perform the corresponding tasks of the instructions according to the instruction sequence; The data operation module is configured to receive the instructions, read the input data from the on-chip storage unit array based on the instructions, perform data operation on the input data, and write the operation result of the data operation into the on-chip storage unit array; The data write-back module is configured to receive the instructions, read the operation result from the on-chip storage unit array based on the instructions, and output the operation result.

2. The programmable neural network inference accelerator of claim 1, wherein, The on-chip storage unit array is configured to store the loaded input data and data generated in the data operation process, and comprises a plurality of on-chip storage units, wherein the on-chip storage units comprise low-bit-width on-chip storage units, high-bit-width on-chip storage units, and output on-chip storage units, wherein The low-bit-width on-chip storage units are configured to store input pictures, network weights, and intermediate feature maps; The high-bit-width on-chip storage units are configured to store biases and partial sums; The output on-chip storage units are configured to store network operation results and intermediate feature maps.

3. The programmable neural network inference accelerator of claim 2, wherein, The on-chip storage units comprise splicing access logic, splitting access logic, and a plurality of static random access memory blocks, the splicing access logic, the static random access memory blocks, and the splitting access logic are connected in series, the splicing access logic is configured to provide a write port, and the splitting access logic is configured to provide a read port.

4. The programmable neural network inference accelerator of claim 1, wherein, The data operation module comprises a calculation unit array and a vector calculation unit, wherein The calculation unit array is configured to read the input data from the on-chip storage unit array and perform convolution calculation; The vector calculation unit is configured to read partial sum data from the on-chip storage unit array and perform vector calculation.

5. The programmable neural network inference accelerator according to claim 3 or 4, wherein The write port of the low-bit-width on-chip storage unit is connected with the data loading module and the vector calculation unit, and the read port of the low-bit-width on-chip storage unit is connected with the calculation unit array; The write port of the high-bit-width on-chip storage unit is connected with the data loading module, the calculation unit array, and the vector calculation unit, and the read port of the high-bit-width on-chip storage unit is connected with the calculation unit array, the vector calculation unit, and the data write-back module; The write port of the output on-chip storage unit is connected with the data loading module and the vector calculation unit, and the read port of the output on-chip storage unit is connected with the data write back module.

6. The programmable neural network inference accelerator of claim 1, wherein, The data loading module, the instruction distribution module, the data operation module and the data write back module all have instruction cache units, wherein, The data loading module is configured to send the input data to the instruction cache unit corresponding to the instruction distribution module. The instruction distribution module is configured to distribute the instructions in the instruction set to the instruction cache units of the corresponding modules.

7. A method for programmable neural network inference acceleration based on memory access, characterized in that, The method applied to the programmable neural network inference accelerator in any one of claims 1 to 6 comprises: fusing network layers of a neural network to be deployed to obtain a plurality of fused layers; performing chunk division and abstraction on the fused layers to generate input data, and generating an instruction set according to the input data, the instruction set comprising a plurality of instructions arranged in an instruction sequence, each instruction corresponding to a task, and the input data being a calculation granularity executable by the neural network inference accelerator; a data loading module loads the input data and the instruction set, and sends the instruction set to an instruction distribution module, and writes the input data into an on-chip storage unit array; the instruction distribution module distributes the instructions in the instruction set to corresponding data operation modules and / or data write back modules, and controls the corresponding modules to execute the corresponding tasks of the instructions according to the instruction sequence; the data operation module receives the instructions, reads the input data from the on-chip storage unit array based on the instructions, performs data operation on the input data, and writes the operation result of the data operation into the on-chip storage unit array; the data write back module receives the instructions, reads the operation result from the on-chip storage unit array based on the instructions, and outputs the operation result.

8. The programmable neural network inference acceleration method of claim 7, wherein, The step of performing chunk division on the fused layers, the method further comprises: obtaining the data scale of the fused layer; obtaining the hardware storage information of the neural network inference accelerator; performing chunk division on the fused layer based on the data scale and the hardware storage information.

9. The programmable neural network inference acceleration method of claim 7, wherein, The step of generating an instruction set according to the input data, the method further comprises: obtaining an initial sequence, the initial sequence being an instruction sequence generated according to the input data; detecting the parallel state of a plurality of instructions according to the initial sequence, the parallel state being used to represent whether the instructions can be executed in parallel; determining an instruction sequence according to the parallel state, and sorting a plurality of instructions according to the instruction sequence to obtain the instruction set.

10. The programmable neural network inference acceleration method of claim 7, wherein, The data operation module comprises a calculation unit array and a vector calculation unit, wherein, the calculation unit array reads the input data from the on-chip storage unit array and performs convolution calculation; the vector calculation unit reads partial sum data from the on-chip storage unit array and performs vector calculation.

Images