RISC-v accelerator system supporting convolutional neural network extension instructions

Abstract

A RISC-V accelerator system supporting convolutional neural network extension instructions, comprising an external memory for saving all instructions and data and an AXI bus for data transmission, and a five-stage pipeline structure of sequentially connected instruction fetching module, decoding module, execution module, memory access module and write back module, the output end of the write back module is connected with a general register stack, the running result of the current instruction is sent into the general register stack for saving and extraction for the decoding module, the application is a general, modular and expandable instruction set capable of processing all convolution layer operations, and the general purpose of the processor is greatly improved based on the RISC-V basic instruction set, the special matrix extension instruction in the application is combined with the RISC-V basic instruction set, the calculation performance is significantly improved and the resource occupation is reduced in the convolutional neural network operation.

Description

Technical Field

[0001] This invention relates to a neural network accelerator. More particularly, it relates to a RISC-V accelerator system that supports extended instructions for convolutional neural networks, implemented using the RISC-V instruction set. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, Convolutional Neural Networks (CNNs), as a commonly used algorithm in deep learning, have seen a rapid increase in both depth and computational complexity. A key characteristic of deep learning technology is its exceptionally large and ever-growing computational load, with convolutional operations accounting for a significant portion. CPUs (Central Processing Units), used for general logic operations, are no longer sufficient to handle such a massive task. Currently, common approaches utilize GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and FPGAs (Field-Programmable Gate Arrays) for specialized neural network computations. However, on embedded mobile platforms with limited computing and storage resources, GPUs and ASICs suffer from high cost, lack of flexibility, and poor scalability. The low-power characteristics of FPGAs and ASICs, on the other hand, extend their application to a wider range of fields, such as power-constrained embedded platforms. Therefore, this paper primarily focuses on the work related to Convolutional Neural Network (CNN) accelerators based on these two types of platforms. However, observations have revealed that in previous work, such accelerators typically only accelerated specific network structures or specific types of layers, with relatively fixed patterns and low flexibility.

[0003] However, current technical solutions in this field, both domestically and internationally, face some challenges. For many convolutional neural networks, existing technologies either improve the computing performance of CPUs and GPUs, which significantly increases power consumption and cost, placing enormous pressure on general-purpose processors that need to support various workloads; or they use separately designed dedicated hardware accelerators, which, while improving efficiency, lack flexibility and are difficult to handle diverse scenarios.

[0004] Combining a versatile, flexible, and low-power processor with a CNN convolutional neural network algorithm to design a RISC-V accelerator that supports custom matrix instructions is a feasible way to solve the problem of flexibility and energy efficiency. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a RISC-V accelerator system that supports extended instructions for convolutional neural networks, which can further save power consumption, reduce costs, and improve flexibility and portability of convolutional neural networks.

[0006] The technical solution adopted in this invention is: a RISC-V accelerator system supporting extended instructions for convolutional neural networks, comprising an external memory for storing all instructions and data and an AXI bus for data transmission, and further comprising a five-stage pipeline structure consisting of an instruction fetch module, a decoding module, an execution module, a memory access module, and a write-back module connected in series. The output of the write-back module is connected to a general-purpose register file, which stores the execution result of the current instruction in the general-purpose register file for retrieval by the decoding module; wherein,

[0007] The instruction fetch module is connected to the external memory via the AXI bus to retrieve instruction information from the external memory and send it to the decoding module.

[0008] The decoding module translates the incoming instruction into instruction type, general-purpose register file address, and memory access related information, retrieves the data required by the instruction from the general-purpose register file, and finally sends the instruction type, memory access related information, and the data required by the instruction to the execution module.

[0009] The execution module performs corresponding calculations based on the type of instruction and the data required by the instruction, and sends the calculation results and the memory access information to the memory access module.

[0010] The memory access module detects whether the current instruction is a memory access instruction. If it is a memory access instruction, it connects to the external memory via the AXI bus, interacts with the external memory based on the memory access information obtained from the execution module, and sends the data obtained from the external memory and the calculation result of the execution module to the write-back module at the same time; otherwise, it sends the calculation of the execution module to the write-back module.

[0011] The write-back module sends the received calculation results into the general-purpose register file;

[0012] The general-purpose register file stores the data required for the instruction. It modifies the data required for the instruction by receiving the calculation results from the write-back module, and retrieves the data required for the instruction from the general-purpose register file through the decoding module.

[0013] The RISC-V accelerator system supporting convolutional neural network extension instructions of this invention is a general-purpose, modular, and scalable instruction set capable of handling all convolutional layer operations, and its versatility is greatly improved by combining the dedicated matrix extension instructions with the basic RISC-V instruction set. The combination of these dedicated matrix extension instructions and the basic RISC-V instruction set significantly improves computational performance and reduces resource consumption in convolutional neural network operations.

[0014] The RISC-V accelerator system of this invention, which supports extended instructions for convolutional neural networks, can perform high-performance computations of convolutional layers in neural networks on embedded platform devices with very limited computing and storage resources, thus enabling powerful neural networks to run smoothly on such devices. Furthermore, this architecture is applicable to almost all neural networks that primarily use convolution, exhibiting great versatility. This invention has been validated on the ZYNQ-7000 series of embedded FPGA heterogeneous platforms, where AlexNet and LeNet-5 convolutional neural networks were tested, achieving high accuracy and prediction performance. Attached Figure Description

[0015] Figure 1 This is a block diagram of the RISC-V accelerator system that supports convolutional neural network extension instructions according to the present invention.

[0016] Figure 2 This is a schematic diagram of the internal structure of the matrix processing unit in this invention;

[0017] Figure 3 yes Figure 2 Internal structure diagram of each processing unit (PE) in the diagram. Detailed Implementation

[0018] The RISC-V accelerator system supporting convolutional neural network extension instructions of the present invention will be described in detail below with reference to embodiments and accompanying drawings. It should be noted that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. To make the objectives, technical solutions, and advantages of the present invention clearer, this embodiment is implemented based on the technical solutions of the present invention, and provides detailed implementation methods and specific operating procedures.

[0019] like Figure 1 As shown, a RISC-V accelerator system supporting extended instructions for convolutional neural networks according to the present invention includes an external memory 1 for storing all instructions and data and an AXI bus 2 for data transmission. It also includes a five-stage pipeline structure consisting of a fetch module 3, a decode module 4, an execution module 5, a memory access module 6, and a write-back module 7, connected in series. The output of the write-back module 7 is connected to a general-purpose register file 8, which stores the execution result of the current instruction for retrieval by the decode module 4. Wherein:

[0020] The external memory 1 mentioned above uses Flash memory as the external memory. Flash memory is a type of memory component and is a non-volatile storage. Due to its large capacity, it is mainly used as an external memory in this invention to store all instructions and data.

[0021] The AXI bus 2 mentioned above: AXI (Advanced eXtensible Interface) is a bus protocol. The AXI-4 bus has five independent transmission channels: read address, read data, write address, write data, and write acknowledge, each with its own handshake protocol. Each channel is independent yet interdependent, making AXI-4 bus data transmission extremely efficient. In this invention, it is mainly used for communication between the processor core and memory.

[0022] The instruction fetch module 3 is connected to the external memory 1 via the AXI bus 2 to retrieve instruction information from the external memory 1 and send it to the decoding module 4.

[0023] The decoding module 4, according to the decoding rules, translates the incoming instruction into instruction type, general-purpose register file address, and memory access information. It then retrieves the required data from the general-purpose register file 8 based on the address and finally sends the instruction type, memory access information, and required data to the execution module 5. The instruction types are divided into scalar instructions and matrix instructions.

[0024] The execution module 5 performs corresponding calculations based on the type of instruction and the data required by the instruction.

[0025] The memory access module 6 detects whether the current instruction is a memory access instruction. If it is a memory access instruction, it connects to the external memory 1 through the AXI bus 2 and interacts with the external memory 1 based on the memory access information obtained from the execution module 5. The data obtained from the external memory 1 and the calculation result of the execution module 5 are sent to the write-back module 7 at the same time. Otherwise, the calculation result of the execution module 5 is sent to the write-back module 7.

[0026] The general-purpose register file 8 stores the data required for the instruction. It modifies the data required for the instruction by receiving the calculation result from the write-back module 7, and retrieves the data required for the instruction from the general-purpose register file 8 through the decoding module 4.

[0027] The execution module 5 includes a scalar processing unit 5.2 for processing all scalar instructions and a matrix processing unit 5.1 for processing matrix instructions. After the decoding module 4 translates the received instructions, it transmits the instructions of type scalar instructions and the data corresponding to the instructions to the scalar processing unit 5.2. For instructions of type matrix instructions, it first checks whether the matrix processing unit 5.1 is ready. If it is ready, it sends the matrix instructions and the data corresponding to the matrix instructions to the matrix processing unit 5.1. If it is not ready, it saves the instruction type and the data corresponding to the instructions, and pauses the pipeline until the next cycle. At the beginning of the next cycle, it repeatedly checks whether the matrix processing unit 5.1 is ready until the matrix processing unit is ready, and then sends the matrix instructions and the data corresponding to the matrix instructions to the matrix processing unit 5.1.

[0028] Since convolutional neural networks often require simultaneous convolution operations across multiple channels, this invention implements three matrix multiplication units to improve processor parallelism. These units can perform matrix operations across three channels simultaneously, significantly enhancing the processor's parallel processing efficiency. Figure 2 As shown, the matrix processing unit 5.1 includes: a matrix multiplication unit controller 5.1.1 for receiving instruction control information output by the decoding module 4; an on-chip cache 5.1.2, a layer controller 5.1.3, and an array unit controller 5.1.4 connected to the matrix multiplication unit controller 5.1.1 and receiving the output information of the matrix multiplication unit controller 5.1.1; the on-chip cache 5.1.2 is connected to the array multiplier 5.1.6 through a multiplexer group 5.1.5; and the outputs of the layer controller 5.1.3 and the array unit controller 5.1.4 are respectively connected to the array multiplier 5.1.6. Wherein:

[0029] The matrix multiplication unit controller 5.1.1 is the control core of the entire matrix processing unit 5.1. It is used to analyze the received instruction control information to obtain data control information and array multiplier control information. The data control information is sent to the cache controller in the on-chip cache 5.1.2. The array multiplier control information includes layer number control information and array unit control information. The layer number control information is sent to the layer number controller 5.1.3, and the array unit control information is sent to the array unit controller 5.1.4.

[0030] The on-chip cache 5.1.2 is the data storage area of ​​the matrix processing unit 5.1, including a cache controller a and two data caches b and c. The cache controller a receives data control information from the matrix multiplication unit controller 5.1.1, and controls the two data caches b and c to interact with the array multiplier 5.1.6 through the multiplexer group 5.1.5 according to the data control information. One data cache is used to store the feature map matrix participating in the operation, and the other data cache is used to store the convolution kernel matrix participating in the operation.

[0031] The layer controller 5.1.3 and array unit controller 5.1.4 control the calculation mode of array multiplier 5.1.6 based on the data control information and array multiplier control information received from matrix multiplication unit controller 5.1.1. Since this invention uses three matrix multiplication units and can support up to 49 processing elements (PEs), the layer controller is used to select the number of channels required for the current instruction, and can select single-channel, dual-channel, and triple-channel modes; the array unit controller is used to select the number of processing units required for the current instruction, and can handle any matrix size up to 7×7.

[0032] The array multiplier 5.1.6 undertakes most of the matrix computation tasks, therefore, choosing an efficient matrix multiplication implementation method is particularly important. The array multiplier 5.1.6 includes three structurally identical channels d, e, and f. Each channel d, e, and f has a feature map matrix input port h and a convolution kernel matrix input port g for receiving the feature map matrix and convolution kernel matrix involved in the computation, 49 processing units PE (7×7) for matrix operations using a systolic array, and a feature map matrix output port m for deriving the computation results. Through hardware multiplexing, it can support matrices with various structures such as 1×1, 3×3, and 5×5. The layer controller 5.1.3 selects 1 to 3 channels for simultaneous computation based on the layer control information of the matrix multiplication unit controller 5.1.1. The array unit controller 5.1.4 selects 1 to 49 processing units PE in each channel for computation based on the array unit control information of the matrix multiplication unit controller 5.1.1, to support smaller-sized matrix multiplication operations. The on-chip cache 5.1.2 inputs a feature map matrix through the feature map matrix input port h to participate in the operation, and the on-chip cache 5.1.2 inputs a convolution kernel matrix through the convolution kernel matrix input port g to participate in the operation; the operation results of each channel d, e, and f are entered into the on-chip cache 5.1.2 through the feature map matrix output port m via the multiplexer group 5.1.5 for the next operation.

[0033] like Figure 3As shown, the 49 processing units (PEs) have the same structure, each including a multiplier C, an accumulator L, a first register J1, a second register J2, and a third register J3. The input of the first register J1 is connected to the first input A, which receives and temporarily stores the feature map matrix sent from the on-chip cache 5.1.2 in row-wise form. The input of the second register J2 is connected to the second input B, which receives and temporarily stores the convolution kernel matrix sent from the on-chip cache 5.1.2 in column-wise form. The output of the first register J1 is connected to the multiplier C and the first output D, respectively, and sends the feature map matrix to the multiplier C and the first output D. The first output is D; the output of the second register J2 is connected to the multiplier C and the second output F respectively, and the convolution kernel matrix is ​​fed into the multiplier C and the second output F respectively; the multiplier C performs a multiplication operation on the received feature map matrix and the convolution kernel matrix, and the output of the multiplier C is connected to one input of the accumulator L, and the operation result is sent to this input of the accumulator L; the other input of the accumulator L is connected to the third register J3, and the accumulator L receives the data from the third register J3. The accumulator L adds the data received from the two inputs and sends the sum to the third register J3; the third register J3 is responsible for storing the sum from the accumulator L.

Claims

1. A RISC-V accelerator system supporting extended instructions for convolutional neural networks, comprising an external memory (1) for storing all instructions and data and an AXI bus (2) for data transmission, characterized in that, It also features a five-stage pipeline structure consisting of a fetch module (3), a decode module (4), an execution module (5), a memory access module (6), and a write-back module (7) connected in series. The output of the write-back module (7) is connected to a general-purpose register file (8), which stores the execution result of the current instruction in the general-purpose register file (8) for retrieval by the decode module (4). The execution module (5) includes a scalar processing unit (5.2) for processing all scalar instructions and a matrix processing unit (5.1) for processing matrix instructions. After the decode module (4) translates the received instruction, it sets the instruction type as a scalar instruction and the instruction type as a matrix instruction. The corresponding data is passed to the scalar processing unit (5.2); for instructions of type matrix instruction, the matrix processing unit (5.1) is first checked for readiness. If it is ready, the matrix instruction and its corresponding data are sent to the matrix processing unit (5.1); if it is not ready, the instruction type and its corresponding data are saved, and the pipeline is paused until the next cycle. At the start of the next cycle, the readiness of the matrix processing unit (5.1) is checked again until it is ready, and the matrix instruction and its corresponding data are sent to the matrix processing unit (5.1); where, The instruction fetch module (3) is connected to the external memory (1) via the AXI bus (2) to obtain instruction information from the external memory (1) and send it to the decoding module (4). The decoding module (4) translates the incoming instruction into instruction type, general register file address, memory access related information, and retrieves the data required by the instruction from the general register file (8). Finally, it sends the instruction type, memory access related information and the data required by the instruction to the execution module (5). The execution module (5) performs corresponding calculations based on the type of instruction and the data required by the instruction, and sends the calculation results and the memory access information to the memory access module. The memory access module (6) detects whether the current instruction is a memory access instruction. If it is a memory access instruction, it connects to the external memory (1) through the AXI bus (2), interacts with the external memory (1) based on the memory access information obtained from the execution module (5), and sends the data obtained from the external memory (1) and the operation result of the execution module (5) to the write-back module (7) at the same time; otherwise, it sends the operation of the execution module (5) to the write-back module (7). The write-back module (7) sends the received calculation results into the general-purpose register file (8); The general-purpose register file (8) stores the data required for the instruction. The data required for the instruction is modified by receiving the operation result from the write-back module (7). The data required for the instruction is retrieved from the general-purpose register file (8) by the decoding module (4).

2. The RISC-V accelerator system supporting convolutional neural network extension instructions according to claim 1, characterized in that, The instruction types are divided into scalar instructions and matrix instructions.

3. The RISC-V accelerator system supporting convolutional neural network extension instructions according to claim 1, characterized in that, The matrix processing unit (5.1) includes: a matrix multiplication unit controller for receiving instruction control information output by the decoding module (4). 5.1.1), respectively with the matrix multiplication unit controller ( 5.1.1) Connected to and receiving matrix multiplication unit controller ( 5.1.1) On-chip buffer for output information (5.1.2), layer controller (5.1.3), and array cell controller ( 5.1.4), the on-chip cache (5.1.2) is connected to the array multiplier (5.1.6) through a multiplexer group (5.1.5), and the outputs of the layer controller (5.1.3) and the array unit controller (5.1.4) are respectively connected to the array multiplier (5.1.6).

4. The RISC-V accelerator system supporting convolutional neural network extension instructions according to claim 3, characterized in that, The matrix multiplication unit controller (5.1.1) is the control core of the entire matrix processing unit (5.1). It is used to analyze the received instruction control information to obtain data control information and array multiplier control information. The data control information is sent to the on-chip cache ( 5.1.2), the array multiplier control information includes layer number control information and array cell control information, and the layer number control information is sent to the layer number controller ( 5.1.3), the array unit control information is sent to the array unit controller ( 5.1.4）。 5. The RISC-V accelerator system supporting convolutional neural network extension instructions according to claim 3, characterized in that, The on-chip cache (5.1.2) mentioned above is a matrix processing unit ( 5.1) The data storage area includes a cache controller (a) and two data caches (b, c), wherein, The cache controller (a) receives data from the matrix multiplication unit controller (a). 5.1.1) data control information, and control the two data buffers (b, c) through the multiplexer group according to the data control information ( 5.1.5) Interacts with the array multiplier (5.1.6) on data; one data buffer is used to store the feature map matrix involved in the operation, and the other data buffer is used to store the convolution kernel matrix involved in the operation.

6. The RISC-V accelerator system supporting convolutional neural network extension instructions according to claim 3, characterized in that, The layer controller (5.1.3) and array unit controller ( 5.1.4) According to the matrix multiplication unit controller ( 5.1.1) Received data control information and array multiplier control information to control the array multiplier ( The calculation mode of 5.1.6).

7. The RISC-V accelerator system supporting convolutional neural network extension instructions according to claim 3, characterized in that, The array multiplier (5.1.6) includes three identical channels (d, e, f), each channel (d, e, f) having a feature map matrix input port (h) and a convolution kernel matrix input port (g) for receiving the feature map matrix and convolution kernel matrix involved in the operation, 49 processing units (PE) in 7×7 arrays for matrix operations using a systolic array, and a feature map matrix output port (m) for deriving the operation result; the layer controller (5.1.3) is based on the matrix multiplication unit controller (… The array unit controller (5.1.1) selects 1 to 3 channel matrices for simultaneous operation based on the layer control information. The array unit controller (5.1.4) then uses the matrix multiplication unit controller (...) The array unit control information in 5.1.1) selects 1 to 49 processing units (PEs) in each channel for operation. The on-chip cache (5.1.2) inputs the feature map matrix through the feature map matrix input port (h) to participate in the operation. The on-chip cache (5.1.2) inputs the convolution kernel matrix through the convolution kernel matrix input port (g) to participate in the operation. The operation result of each channel (d, e, f) is entered into the on-chip cache (5.1.2) through the feature map matrix output port (m) and multiplexer group (5.1.5) for the next operation.

8. The RISC-V accelerator system supporting convolutional neural network extension instructions according to claim 7, characterized in that, The 49 processing units (PEs) have the same structure, each including a multiplier (C), an accumulator (L), a first register (J1), a second register (J2), and a third register (J3). The input of the first register (J1) is connected to the first input (A), which receives and temporarily stores the feature map matrix sent from the on-chip cache (5.1.2) in row form. The input of the second register (J2) is connected to the second input (B), which receives and temporarily stores the convolution kernel matrix sent from the on-chip cache (5.1.2) in column form. The output of the first register (J1) is connected to the multiplier (C) and the first output (D), respectively, and sends the feature map matrix into the multiplier. (C) and the first output (D); the output of the second register (J2) is connected to the multiplier (C) and the second output (F) respectively, and the convolution kernel matrix is ​​sent to the multiplier (C) and the second output (F) respectively; the multiplier (C) performs multiplication operation on the received feature map matrix and the convolution kernel matrix, the output of the multiplier (C) is connected to one input of the accumulator (L), and the operation result is sent to this input of the accumulator (L); the other input of the accumulator (L) is connected to the third register (J3), and the data from the third register (J3) is received. The accumulator (L) adds the data received from the two inputs and sends the sum to the third register (J3); the third register (J3) is responsible for storing the sum from the accumulator (L).

Images