A convolution accelerator based on RISC-V instruction set architecture extension and a method for accelerating convolution operation

Abstract

The application discloses a convolution accelerator based on RISC-V instruction set architecture expansion, which is located in a computer system and is physically connected with a memory through a memory bus, introduces a data buffer, and uses a buffer block strategy to improve convolution operation data multiplexing and reduce the demand of convolution operation on memory access, wherein the modules in the convolution accelerator include a fetch and instruction distribution module, a data carrying module, a feature map buffer, a convolution kernel buffer, a convolution operation unit, a convolution logic operation control register group, an output buffer and an adder; and the convolution accelerator is provided with a self-defined expansion instruction. The application also discloses a method for accelerating convolution operation by using the convolution accelerator.

Description

Technical Field

[0001] This invention belongs to the technical field of convolution operation acceleration, instruction design, and convolution accelerator hardware design, and relates to a convolution operation accelerator based on the RISC-V instruction set architecture extension. Background Technology

[0002] Convolutional Neural Networks (CNNs) have been widely used in artificial intelligence and deep learning. The main operation in CNNs is convolution, which accounts for the majority of their computation time. Therefore, accelerating and optimizing convolution operations is crucial for improving the performance of CNN algorithms. Generally, a convolution operation has two inputs: a feature map and a kernel. The specific steps can be found in [link to documentation]. Figure 1 As shown, the convolution kernel performs dot product operations while sliding across the feature map until its sliding range reaches the upper limit of the feature map boundary, outputting a single-channel convolution result. Finally, the results from multiple channels are summed to obtain the final convolution result. Due to the characteristics of convolution operations, both the input feature map data and the convolution kernel data are repeatedly loaded and used during the operation, resulting in numerous redundant memory accesses. Furthermore, depending on the order in which the feature map data and convolution kernel data are stored in memory, the input data read for convolution operations is often discontinuous in memory, leading to significant memory bandwidth waste due to frequent repeated accesses.

[0003] RISC-V is an open-source instruction set that offers the advantage of being license-free. At the instruction set design level, RISC-V avoids the legacy issues of many mainstream instruction sets such as x86 and Arm, resulting in a streamlined design. RISC-V also provides a design space for a range of custom instructions, allowing researchers and hardware manufacturers to analyze specific high-level application characteristics and design custom extension instructions to achieve better application performance. Building on this foundation, with the increasing application of convolutional neural networks in fields such as autonomous driving, computer vision, and natural language processing, more and more researchers and hardware manufacturers are developing convolutional accelerators based on RISC-V instruction set extensions.

[0004] Most existing convolution accelerators or convolution operation processors based on the RISC-V instruction set architecture suffer from the following problems: 1-3] :

[0005] 1. Most existing convolution accelerators based on the RISC-V instruction set architecture only consider how to load the input data of convolution operations in memory onto the on-chip storage of the accelerator, and only design corresponding instructions for this function. At the same time, they simply partition and store different input data such as feature map data and convolution kernel data, without considering how to efficiently divide the cache partitions on the accelerator. This makes the caching strategy on the accelerator relatively inefficient and the reusability of on-chip data not high enough.

[0006] 2. Existing RISC-V convolution accelerators generally do not consider how to fully utilize memory bandwidth resources during data transfer when designing on-chip caches and RISC-V custom instructions, resulting in a huge waste of memory bandwidth.

[0007] 3. Most existing RISC-V convolution accelerators or dedicated convolution processors do not consider zero-padding when designing instructions. Many zero-padding operations are implemented through special hardware on the accelerator. However, convolution operations with zero-padding are often the most common application scenarios in convolution operations. The lack of zero-padding in instruction design reduces the flexibility of convolution accelerator programming and also places higher demands on the adaptation of upper-level toolchains for convolution accelerators. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention aims to provide a convolution accelerator based on a RISC-V instruction set architecture extension. This convolution accelerator resides in the computer system and is physically connected to memory via a memory bus. It introduces a data buffer on the convolution accelerator and proposes an efficient buffer partitioning strategy, thereby improving the reusability of convolution operation data and significantly reducing the memory access requirements of convolution operations. Furthermore, this invention considers how to fully utilize the computer system's memory bandwidth resources to load convolution input data in the design of the convolution accelerator and its accompanying instructions, thereby improving the efficiency of data transfer between memory and the convolution accelerator and the utilization rate of system memory bandwidth. Simultaneously, the design of the convolution accelerator also considers commonly used zero-padding convolution, and, combined with a custom RISC-V extended instruction set, fully utilizes various hardware resources on the convolution accelerator, effectively improving the operating efficiency of convolution operations.

[0009] This invention will be described in three parts. The first part will describe the overall system architecture in which the convolution accelerator is located. The second part will describe the design of the convolution accelerator. The third part will describe the design of the RISC-V custom extension instructions.

[0010] First, the overall structure of the system containing the convolution accelerator is as follows: Figure 2As shown, the system mainly includes a RISC-V processor, memory, a convolution accelerator, and other I / O devices. It should be noted that this diagram is only a schematic of the system containing the convolution accelerator; the actual system factors, such as the number of processors and the memory coherence architecture, depend on the specific computer system. The various components in this system, such as the convolution accelerator and the RISC-V processor, can be connected via a system bus. To improve overall data transmission efficiency and reduce data transmission pressure on the system bus, the RISC-V processor and memory can be connected via a memory bus. Similarly, to achieve efficient data transfer, the convolution accelerator and memory also need to be connected via a memory bus. It should be noted that the convolution accelerator in the example diagram represents a coprocessor implementation. Specific implementations include, but are not limited to, the coprocessor form described above. It can also be directly embedded into the RISC-V processor design as an extended core; implementation details will be described in the embodiments.

[0011] The design of the convolution accelerator will be explained next. The specific conceptual design of the convolution accelerator is as follows: Figure 3 As shown, the accelerator mainly consists of eight modules: instruction fetch and instruction allocation module, data transport module, feature map buffer, convolution kernel buffer, convolution operation unit, convolution logic operation control register group, FIFO memory, output buffer and adder; the convolution accelerator interacts with memory through the memory bus;

[0012] in,

[0013] (1) Instruction Fetch and Allocation Module: The RISC-V extended instructions used in the accelerator of this invention all adopt custom opcodes reserved in the RISC-V standard manual. This set of opcodes is independent of the opcodes of all instructions in the existing RISC-V standard instruction set manual, thereby avoiding instruction decoding conflicts. After the RISC-V processor parses this series of custom extended instructions during the decoding stage, it will allocate them to the instruction fetch and allocation module on the accelerator. This module will then allocate them to other modules on the accelerator according to the specific instructions, so as to coordinate and control the operation of various hardware modules on the accelerator. It should be noted that in the actual hardware design, the instruction fetch and allocation module needs to be physically connected to the various modules on the convolution accelerator. Here, it should be noted that this module requires the corresponding support of the RISC-V processor, that is, the processor needs to be equipped with the function of decoding and allocating instructions on the accelerator during the design.

[0014] (2) Data transfer module: This module is physically connected to the memory bus, convolution kernel buffer, feature map buffer, and output buffer. It is used to control the data copying function between memory and on-chip buffer of convolution accelerator. The data transfer module can complete the data transfer process between memory and feature map buffer, memory and convolution kernel buffer, and memory and output buffer according to different data transfer instructions. It realizes the mutual transfer of data between memory and data transfer module. Memory transfers data to convolution kernel buffer and / or feature map buffer through data transfer module, and output buffer transfers data to memory through data transfer module.

[0015] (3) Feature Map Buffer: The feature map buffer is physically connected to the data transport module and the convolution operation unit, and is logically controlled by the convolution logic operation control register group. It is used to buffer the feature map input for convolution operations. Logically, it can be divided into blocks with a Tile variable as the size. Here, the Tile variable refers to the size of a single block in the feature map buffer. The length of the Tile is determined by the operation logic control register group on the convolution accelerator. In addition, the feature map buffer adopts a circular logical structure. When the feature map buffer is full, the starting position of the feature map buffer can be reset through relevant instructions, including mvtile, or the blocks on the feature map buffer can be copied, thereby realizing the reuse of the data at the end of the feature map buffer. When the feature map buffer is full, new feature map data usually needs to be replaced as the input for the next round of convolution operations. The data in the blocks at the end of the feature map buffer will still participate in the next round of convolution operations. This part of the data does not actually need to be reloaded from memory. In this invention, by resetting the starting point of the feature map buffer, the convolution operation unit can use the data in the tail section of the feature map buffer as the starting point for reading feature map data in the next round of calculation, thereby reusing this part of the data. This ensures that the convolution accelerator can still correctly output the calculation results without reloading the existing on-chip data. Depending on the specific implementation of the invention, this data in the tail section can also be copied to the head of the feature map buffer to achieve data reuse. Further details regarding this data reuse will be described below along with the workflow of the convolution accelerator.

[0016] (4) Kernel Buffer: The kernel buffer is physically connected to the data transport module and the convolution operation unit, and is subject to the logic control of the convolution logic operation control register group. It is used to buffer the kernel data during the convolution operation. The kernel buffer is divided into blocks according to the amount of data in a single channel of the input kernel. If the size of the kernel buffer cannot be divided by the amount of data in a single channel of the kernel, the total number of blocks in the kernel buffer will be rounded down.

[0017] (5) Convolution Operation Unit: The convolution operation unit is physically connected to the feature map buffer, convolution kernel buffer, adder, and output buffer. It can also be connected to the adder through a FIFO memory and is subject to the logic control of the convolution logic operation control register group. The convolution operation unit can be composed of a multiply-adder or a pulse array. The convolution operation unit reads the feature map and convolution kernel data from the corresponding buffers as inputs for the convolution operation, and sends the result of the convolution operation to the corresponding register in the RISC-V processor or outputs it to the FIFO memory on the accelerator or further outputs it to the output buffer.

[0018] (6) Convolution Logic Operation Control Register Group: This is a set of status control registers used to control convolution logic operations, specifically as follows: Figure 4 As shown, the convolution kernel buffer, feature map buffer, and convolution operation unit implement logical control, which mainly includes three sets of logical control registers: one or more block size registers, one or more convolution kernel height registers, and one or more convolution kernel width registers, which respectively control the logical block size, convolution kernel height, and convolution kernel width in the feature map buffer. The height and width of the convolution kernel determine the logical block size of the convolution kernel buffer, i.e., the size of the single-channel data volume. The logical block size of the feature map buffer determines the jump interval required by the convolution operation unit when reading each line of data the width of the convolution kernel. It should be noted that the number of registers in this set is not limited to three; each set of logical control registers can include multiple registers, that is, it is only necessary to ensure that the registers in the register set can fully specify the values ​​of these three variables.

[0019] (7) FIFO Memory: The convolution accelerator may further include a FIFO memory, which is located between the convolution operation unit and the adder and is physically connected to both. The FIFO memory can be used to improve the throughput of the convolution accelerator. When the convolution operation unit completes the convolution operation and outputs the result, the result of the convolution operation and the output buffer location specified in the conv2buf instruction can be temporarily stored in the FIFO memory. The specific process is as follows: Figure 5As shown, the FIFO memory then transmits a record to the adder. Based on this record, the adder adds the output convolution result to the data in the output buffer specified in the record and writes the accumulated result back to the output buffer. Once the convolution unit writes the result to the FIFO, the next round of convolution can begin. This avoids excessively long operation cycles for a single convolution instruction, which would reduce the throughput of the convolution accelerator. Furthermore, the convolution accelerator using FIFO memory can improve the parallelism of convolution operations. If there are multiple multiply-accumulate units in the convolution unit, the convolution results can be written to the FIFO memory concurrently, so that the parallelism of convolution operations is no longer significantly limited by the number of adders and the computation time, further improving the accelerator's throughput. It should be noted that in this convolution accelerator, the FIFO memory is an optional module; the convolution accelerator proposed in this invention can also directly connect the convolution unit and the adder.

[0020] (8) Output Buffer and Adder: The adder is physically connected to the convolution operation unit, the output buffer and / or the FIFO memory. The output buffer is physically connected to the adder, the convolution operation unit and the data transfer module. The output buffer is used to store the output of the convolution operation result. An adder connected to the output buffer is provided. The adder is used to read the temporary output result of the corresponding channel in the convolution operation in the output buffer, accumulate it with the output result of the current round of convolution operation unit, and write the accumulated result back to the output buffer.

[0021] This invention also includes the design of custom instructions on the accelerator, the encoding design of which is as follows: Figure 6-1 and Figure 6-2 As shown, Figure 6-1 or Figure 6-2 In the text, the numbers above the instructions indicate the start and end positions of each instruction's encoding region. Figure 6-1Taking the `kload` instruction as an example, its `funct3` encoding is located in bits 12 to 14 of the instruction encoding. It should be noted that the instruction design encoding provided in this invention is an instruction format. When applying this invention to a specific production environment, a specific encoding implementation will be performed on this instruction format. That is, in the specific implementation, it is only necessary to ensure that the specific encoding of these instructions does not conflict with the instruction encoding in the RISC-V standard manual. Furthermore, to facilitate the explanation of the instruction design principles, an example implementation will be used to describe these instructions. The specific registers used, such as `rs1` and `rs2`, can have their parameters swapped in order during the specific implementation. In addition, it should be noted that the following specific RISC-V extended instruction is determined by both the opcode bit and the `funct` bit. The `funct` bit includes subsets such as `funct2`, `funct3`, and `funct7`. Any difference in any bit or more of the opcode or `funct` bits in the instruction encoding indicates a different instruction. It should also be noted that the reserve area in the instruction is a reserved area in the instruction design. This area does not have any indicative function in the prototype design of the instruction, but when the present invention is implemented, the reserve area can be expanded into other coding areas to extend the functionality of the instruction.

[0022] This convolution accelerator is primarily equipped with the following extended instructions:

[0023] (1) kload instruction: The kload instruction is responsible for loading data from memory and storing it in the convolution kernel buffer. kload uses an indirect memory access mode. It uses the value stored in the rs1 register as the base address for loading from memory, the value stored in the rs2 register as the number of bytes to be loaded from memory, and the value stored in the rs3 register as the starting position of the loaded data stored in the convolution kernel buffer. reserve is a reserved area for the instruction. When the instruction needs to be extended with some functions, this area can be used for expansion.

[0024] (2) fload instruction: The fload instruction is responsible for loading data from memory and storing it in the feature map buffer. fload uses indirect memory access mode, where rs1 is the base address of the memory to be loaded, rs2 is the number of bytes to be loaded, and rs3 is the starting position of the on-chip storage;

[0025] (3) floadp0fw instruction: The floadp0fw instruction loads data from memory into the feature map buffer and fills the corresponding positions in the feature map buffer with 0s to handle common zero-padding convolution operations. The floadp0fw instruction first fills the feature map buffer with rs4 zeros starting at address rs3, and then loads data from memory address rs1 with a length of rs2 into the Feature Map Buffer at the corresponding positions. In actual hardware design, these two steps can be performed simultaneously; that is, the zero-padding operation in the feature map buffer and the data loading operation from memory can be performed concurrently without any order. Instructions that extend the functionality of the fload instruction can be simply referred to as fload-type instructions. Their usage scenarios and principles can be found in [link to documentation]. Figure 7 As shown, when the feature map side length exceeds the maximum amount of data that can be loaded in a single system bandwidth operation and zero-padding is required, the feature map data needs to be loaded in columns. In this case, data in the first column of the feature map needs to be padded with leading zeros when loaded into the feature map buffer. The `floadp0fw` instruction will be used to load the feature map data in this way. Similarly, when loading feature map data in the later columns into the feature map buffer, the `floadp0bw` instruction will be used, and trailing zeros will be padded when loading the data into the feature map.

[0026] (4) floadp0bw instruction: Similar to the working principle of floadp0fw instruction, floadp0bw adopts the indirect memory access mode. It uses the value stored in rs1 as the starting address for data loading, loads data of length rs2, stores it in the feature map buffer at the address starting from rs3, and adds rs4 zeros after the corresponding data. These two operations can also be performed simultaneously in actual hardware design.

[0027] (5) floadp0ba instruction: Combining the functional characteristics of floadp0fw and floadp0bw instructions, floadp0ba adopts the indirect memory access mode, uses the value stored in the rs1 register as the starting address of the data loading, loads data of length rs2, stores it in the feature map buffer at the address starting from rs3, and adds rs4 zeros to both ends of the corresponding data. The operation of filling zeros on the buffer and the work of loading data from memory can also be performed synchronously.

[0028] (6) ffill0 instruction: Simply using the fload instruction mode with zero padding cannot fully handle zero-padding convolution. For details, see below. Figure 7As shown, when the accelerator processes the zero-padded data in the bottom row of the feature map, it doesn't actually need to load the data from memory. This can be solved by introducing the `ffill0` instruction, which directly fills the feature map buffer with zeros. The `ffill0` instruction can be considered a pseudo-instruction of `floadp0fw` or `floadp0bw`. When `rs2` in `floadp0fw` or `floadp0bw` is specified as zero, the instruction requires the length of the data to be loaded from memory to be zero. Therefore, it will only perform the zero-padding operation in the feature map buffer, i.e., filling the feature map buffer with rs4 zeros starting at position `rs3`. It is required that the length stored in `rs4` be less than the actual length of the feature map buffer; otherwise, a hardware error signal will be returned to the user. Meanwhile, ffill0 is designed as a pseudo-instruction, which requires that the encoding of the ffill0 instruction be consistent with the encoding of the floadp0fw or floadp0bw instruction. That is, the ffill0 instruction is a special floadp0fw or floadp0bw instruction. Apart from the instruction name and specifying that the parameter stored in rs2 is zero, this instruction is not significantly different from the floadp0fw or floadp0bw instruction.

[0029] (7) conv instruction: The conv instruction is responsible for calling the convolution operation unit, sending the data starting at position rs1 in the feature map buffer and the convolution kernel data starting at position rs2 in the convolution kernel buffer into the convolution operation unit, and writing the operation result into the rd register in the RISC-V processor. The specific calculation method can be found in [link to relevant documentation]. Figure 8 .

[0030] (8) ofill0 instruction: This instruction is used to clear the data in the output buffer to ensure the correctness of the output result of the next round of convolution. This instruction will use the value stored in rs1 as the starting address of the output buffer and fill rs2 with zeros. It is required that the length stored in rs2 is less than the actual length of the output buffer, otherwise a hardware error exception signal will be returned to the user.

[0031] (9) sethwtl instruction: This instruction sets the set of operation logic control registers on the accelerator, that is, it can set the height and width of the convolution kernel, as well as the size of the feature map buffer blocks.

[0032] (10) conv2buf instruction: This instruction reads data from the convolution kernel buffer and feature map buffer in the same way as the conv instruction. The difference is that after the convolution operation, it first retrieves the value numbered rs3 from the output buffer, adds it to the output result of the convolution operation, and then writes it back to the position rs3 in the output buffer. If the convolution accelerator is equipped with a FIFO module, the conv2buf instruction will first send the result of the convolution operation and the position in the output buffer to the FIFO queue, and then accumulate the data when the adder obtains the data.

[0033] (11) ostore instruction: This instruction is responsible for writing the data in the output buffer into memory. The specific operation is as follows: the data in the output buffer with the starting address rs2 and the length rs3 is written into the continuous space in memory with the value stored in rs1 as the starting address.

[0034] (12) `mvtile` instruction: This instruction is used to reset the starting position of the feature map buffer or copy blocks on the feature map buffer. It can be implemented in two ways. First, it can directly set `rs1` as the starting position of the circular buffer, thus avoiding on-chip data copying during the next round of feature map data reuse. Second, it can also be implemented as an on-chip data copy, using the data stored in `rs1` as the starting position, the data stored in `rs2` as the length of the copy, and the data stored in `rs3` as the destination address. Depending on the specific implementation of this invention, `mvtile` can be implemented by resetting the starting point of the feature map buffer for data reuse, or by copying on-chip data to achieve data reuse.

[0035] The following is combined with Figure 8 This paper explains the specific workflow and principles of using a convolution accelerator with custom extension instructions. The specific steps are as follows:

[0036] Step 1: When the convolution accelerator starts running, it first configures the convolution logic operation control register group in the convolution accelerator to determine the width and height of the convolution kernel and the block size of the feature map buffer. Based on the width and height of the convolution kernel, the logical block size of the convolution kernel buffer can be determined.

[0037] Step Two: To avoid loading convolution kernel data from memory multiple times, the accelerator prioritizes loading the convolution kernel data, fixing it within the convolution kernel buffer. At this point, the accelerator uses the `kload` instruction and utilizes all memory bandwidth to load the convolution kernel data. Specifically, the `kload` instruction sets the value in the `rs2` register to the size of the memory bandwidth to load the convolution kernel data in batches. This fully utilizes system bandwidth resources and improves data transfer efficiency between memory and the accelerator. The accelerator repeats this step until the convolution kernel data is fully loaded or the convolution kernel buffer is full. Furthermore, based on the convolution kernel height and width specified in the convolution operation logic control register, the convolution kernel buffer will be logically divided into blocks of single-channel data size.

[0038] Step 3: Using the maximum number of convolutional kernel channels that the current convolutional buffer can hold as an upper bound, the feature map data is loaded in batches. If the amount of convolutional kernel data is greater than the size of the convolutional kernel buffer, the convolutional kernel data will be loaded in batches. After one batch of convolutional kernel data is loaded, the next batch of feature map data will be loaded sequentially according to the channel numbers in the convolutional kernel buffer. When feature map data is loaded into the buffer on the convolutional accelerator, it will be loaded in a top-to-bottom order, followed by a left-to-right order. During feature map data loading, the memory bandwidth will be fully utilized. That is, when using fload-type instructions to load feature map data, the value stored in the rs2 register will be set to the size of the memory bandwidth to improve data loading efficiency. If the amount of data in a single row of the currently processed feature map is less than the memory bandwidth, the floadp0ba instruction will be used, and the value stored in its rs2 register will be set to the size of the row of data in the feature map for loading. The convolutional accelerator will use fload-type instructions to fill the feature map buffer as much as possible. At this point, the tile size of the feature map buffer is determined by the logic operation control register. The above-mentioned "load as much as possible" means that when the tile is the tile size, this step will load the feature map data with the maximum number of tile tiles that can be filled in the feature map buffer. If the feature map data has been loaded in this case, this step can be ended and the process can proceed to step four.

[0039] Step 4: Using the `conv` or `conv2buf` instruction, the convolution operation unit reads data from the feature map buffer and the convolution kernel data buffer, performs convolution operations, and sends the result to the RISC-V processor's register or the accelerator's output buffer. If the `conv2buf` instruction is used, the result is sent to the output buffer. If the convolution accelerator is equipped with a FIFO memory module, the `conv2buf` instruction first sends the convolution operation and the location of the output buffer to the FIFO memory. Then, the FIFO memory sequentially sends the convolution operation results in the FIFO queue to the adder. The adder retrieves the result and the data at the output buffer location specified in the `conv2buf` instruction, adds them together, and then writes the sum back to the output buffer. If the accelerator is not equipped with a FIFO memory module, the step of writing the result to the FIFO memory is skipped. The data at the corresponding location in the output buffer is directly retrieved and added to the result output by the convolution operation unit, and the accumulated result is written back to the output buffer.

[0040] Step 5: When the data in the feature map buffer is processed in this round of convolution, the feature map data for the next round is loaded. If the data in this tile of the feature map has not yet been processed, and the convolution stride is less than the kernel width, data reuse will occur. The advantage of using a circular buffer is evident here; the starting point and reserved length of the circular buffer can be reset using the `mvtile` instruction, thus avoiding the additional overhead of copying on-chip data. The specific principles of circular buffers and the `mvtile` instruction are as follows... Figure 9 As shown, assuming the convolution kernel width and height are 3, the maximum number of blocks that the feature map buffer can hold is 4, and the stride of the convolution operation is 1, then when loading feature map data in the next batch, the last two blocks in the feature map buffer can be reused. At this time, the starting point of the circular buffer can be reset by the mvtile instruction, so that the convolution operation unit starts reading feature map data from this point in the next round of operation. In this way, the reuse of the tail data of the feature map buffer can be completed without data copying. At the same time, when new feature map data is introduced, the data only needs to be loaded after the two reserved blocks.

[0041] Step Six: Repeat steps Three, Four, and Five in a loop until all convolution operations in this batch are completed, or until the output buffer is full. At this point, the ostore instruction is used to write the result data from the output buffer back to memory. If the size of the output buffer is insufficient to store all the output results, the output buffer will be processed in batches again. After writing the results of the previous batch into memory, the ofill0 instruction is used to clear the cached value on the output buffer, thereby ensuring that the next round of convolution operations can obtain the correct output results.

[0042] It should be noted that the order of batch processing of the output buffer in step six can be interchanged with the order of batch processing of the convolution kernel data in step two. That is, the output buffer can be batch processed first, and then the convolution kernel buffer can be batch processed. The order of batch processing of the buffer can be reversed.

[0043] Based on the above principles and steps, this invention also implements an example convolution operation library. This library provides users with an interface to call the accelerator. At the same time, the library embeds the extended instructions provided by this invention to fully utilize the various modules on the convolution accelerator.

[0044] The convolution accelerator proposed in this invention can be embedded into a RISC-V processor design as an extended core, or it can accelerate convolution operations as a coprocessor. Users can invoke the accelerator by embedding custom accelerator instructions within their programs, or by using a compiler to generate the corresponding instructions. The convolution accelerator design allows for multi-core expansion to handle different convolution tasks simultaneously. In multi-core accelerators, the buffer organization has two forms: first, each processing core has its own dedicated buffer, i.e., each accelerator processing core has its own feature map buffer and convolution kernel buffer; second, multiple accelerator processing cores share some or all of the buffers. The convolution accelerator proposed in this invention is also compatible with convolution operation libraries of currently popular deep learning frameworks, which can invoke the accelerator by embedding custom instructions.

[0045] The beneficial effects of this invention include:

[0046] (1) This invention proposes a strategy for buffer partitioning and buffer block division of convolution accelerators. Taking into account the memory bandwidth of the computing system, it is introduced into the design of convolution accelerators to make full use of the memory bandwidth resources of the computing system, greatly reduce the amount of data loaded into external storage for convolution operations, improve the reusability of data in the buffer, reduce data copying between buffers on the convolution accelerator, and thus improve the execution efficiency of convolution operations. According to experimental statistics, for commonly used convolutional neural networks, such as ResNet-34, when using this convolution accelerator to complete the same scale of computation as the convolutional neural network, the total number of operation instructions in different convolutional layers can be reduced to 1 / 5 to 1 / 10 times that of the unused accelerator, and the memory access requirement can be reduced to 1 / 2 to 1 / 10 times that of the unused accelerator.

[0047] (2) Based on (1), this invention also provides a set of instruction set design based on RISC-V extension, which can be used in conjunction with various modules of the accelerator. From the perspective of instruction design, it takes into account the zero-padding convolution commonly used in actual production environments, simplifies the difficulty for users to write upper-level algorithms, and further improves the flexibility of programming convolution accelerator.

[0048] (3) The present invention also provides an implementation of a convolution operator operation library, which is based on the above-mentioned accelerator principle and simplifies the steps for users to use the accelerator. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of a commonly used two-dimensional convolution operation.

[0050] Figure 2 This is a schematic diagram of the overall system architecture of the convolution accelerator.

[0051] Figure 3 This is a design diagram of the convolution accelerator of the present invention.

[0052] Figure 4 This is a schematic diagram of the structure of the convolution logic operation control register group in this invention.

[0053] Figure 5 This is a schematic diagram of the FIFO storage structure and process in this invention.

[0054] Figure 6-1 , Figure 6-2 This is the instruction design diagram of the convolution accelerator of the present invention.

[0055] Figure 7 This is a flowchart of the convolution accelerator of the present invention.

[0056] Figure 8 This is a schematic diagram illustrating the working principle of the fload and ffill0 instructions of this invention.

[0057] Figure 9 This is a schematic diagram of the design principle of the circular feature map buffer and the mvtile instruction.

[0058] Figure 10 This is a schematic diagram of Example 1.

[0059] Figure 11 This is a schematic diagram of Example 2.

[0060] Figure 12 This is a schematic diagram of Example 3.

[0061] Figure 13 This is a schematic diagram of Example 4.

[0062] Figure 14 This is a schematic diagram of Example 5.

[0063] Figure 15 This is a schematic diagram of Example 6. Detailed Implementation

[0064] The present invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the present invention are all common knowledge and general knowledge in the art, and the present invention does not have any particular limitations.

[0065] This invention provides a convolution accelerator based on the RISC-V instruction set architecture extension. The convolution accelerator is located in a computer system and physically connected to memory via a memory bus. It introduces a data buffer and uses a buffer partitioning strategy to improve data reuse for convolution operations and reduce memory access requirements. The modules in the convolution accelerator include: an instruction fetch and allocation module, a data transport module, a feature map buffer, a convolution kernel buffer, a convolution operation unit, a convolution logic operation control register group, an output buffer, and an adder. The convolution accelerator is equipped with custom extended instructions.

[0066] The convolution accelerator proposed in this invention can be embedded into a RISC-V processor design as an extended core, or it can accelerate convolution operations as a coprocessor. Users can invoke the accelerator by embedding custom accelerator instructions within their programs, or by using a compiler to generate the corresponding instructions. The convolution accelerator design allows for multi-core expansion to handle different convolution tasks simultaneously. In multi-core accelerators, the buffer organization has two forms: first, each processing core has its own dedicated buffer, i.e., each accelerator processing core has its own feature map buffer and convolution kernel buffer; second, multiple accelerator processing cores share some or all of the buffers. The convolution accelerator proposed in this invention is also compatible with convolution operation libraries of currently popular deep learning frameworks, which can invoke the accelerator by embedding custom instructions.

[0067] Example 1: Hardware Design in the Form of Processor Extended Cores

[0068] like Figure 10 As shown, the accelerator can be embedded into the RISC-V processor design as an extended core. If an extended core hardware design is used, the accelerator is embedded as a processing core within the RISC-V processor design. When an instruction applied to the accelerator is successfully decoded by a processing core in the RISC-V processor, the instruction allocation unit on that processing core allocates the instruction to the convolution accelerator. The instruction allocation module on the convolution accelerator then allocates the instruction to various modules on the accelerator according to its function. After the accelerator completes the convolution workload, it inputs the convolution result into the registers or memory of a specific processing core, depending on the instruction, such as the `conv` instruction or the `conv2buf` instruction.

[0069] Example 2: Accelerating Convolution Operations Using Coprocessors

[0070] Unlike the processor design in Embodiment 1, where the accelerator is used as a processor extension core, this accelerator can also accelerate convolution operations as a coprocessor, such as... Figure 11 As shown, when the main RISC-V processor decodes a custom instruction on the accelerator, the instruction can be directly dispatched to the convolution accelerator, which acts as a coprocessor. After the coprocessor completes the convolution operation, it returns the result to the RISC-V processor or writes it directly into memory. Compared to Example 1, this approach is beneficial for improving the area utilization of the RISC-V processor design.

[0071] In this embodiment, the convolution accelerator can also, as described in Embodiment 1, first use the conv instruction to write the result of the convolution operation into the RISC-V processor, and after the processor completes the corresponding calculation work, write the result of the convolution operation into memory; the accelerator can also choose to use the conv2buf instruction to directly write the accelerator's calculation result into memory.

[0072] Example 3: The user embeds a custom accelerator command within the program to invoke the accelerator.

[0073] Users can invoke the convolution accelerator in their own computational programs by embedding accelerator assembly instructions, specifically as follows: Figure 12 As shown. After the user finishes writing the source program file containing the accelerator instructions, they can use a compiler and assembler with accelerator instruction extension support to generate an executable file containing the accelerator instructions. The user can then run the executable file to access the hardware module resources on the accelerator.

[0074] Example 4: Using a compiler to generate the corresponding instructions for the accelerator and call the accelerator.

[0075] Unlike the manual inline assembly method in Example 3, users can also extend the instructions on the accelerator by calling the compiler's extension interfaces for accelerator instructions, such as builtin and intrinsic functions. In addition, the compiler can be modified to add a corresponding pass processing mechanism, allowing the compiler to automatically generate extended instructions for the accelerator by recognizing specific instruction templates and replacing ordinary convolution operation instruction templates. Here, the compiler can also generate additional instructions for the RISC-V host processor to cooperate with the accelerator's operation. Figure 13 Explain the specific ways in which these two things are done.

[0076] Example 5: Accelerator design can be scaled from single-core to multi-core.

[0077] This invention currently only provides a design diagram of a single-core accelerator, but it can be extended to a multi-core design based on this single-core structure, forming a multi-core convolution operation accelerator, as detailed below. Figure 14 As shown. This accelerator can be extended from a single-core design to a multi-core design, and the multi-core form of the convolution accelerator can handle different convolution tasks simultaneously.

[0078] There are two types of buffer organization on accelerators. The first type is where each processing core in a multi-core accelerator has its own dedicated buffer, that is, each accelerator processing core has its own feature map buffer and convolution kernel buffer. The second type is where multiple accelerator processing cores share some or all of the buffers. For example, multiple accelerator processing cores only share the convolution kernel buffer, but the feature map buffer and output buffer are dedicated to each processing core.

[0079] Example 6: Providing a convolution operator library for integration with deep learning frameworks

[0080] Taking any popular deep learning framework as an example, these frameworks encapsulate convolution operators to varying degrees, providing users with a simple and easy-to-use interface, meaning users are unaware of the underlying implementation of the convolution operators. This invention provides a corresponding convolution operation library adapted to this accelerator, which calls the accelerator through embedded custom instructions. For example... Figure 15 As shown, users can rewrite the implementation of convolution operations in deep learning frameworks to integrate them with the example convolution operator library provided by this accelerator or their own self-written convolution operator library to achieve an overall acceleration effect.

[0081] References

[0082] [1]Z.Li, W.Hu and S.Chen, "Design and Implementation of CNN CustomProcessor Based on RISC-V Architecture," 2019IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference onData Science and Systems(HPCC / SmartCity / DSS), 2019, pp.1945-1950, doi:10.1109 / HPCC / SmartCity / DSS.2019.00268.

[0083] [2] Fu Siyang, Chen Hua, Yu Faxin. Design and implementation of convolutional neural network processor based on RISC-V [J]. Microelectronics & Computer, 2020, 37(4):49-54.

[0084] [3] Lou, W., Wang, C., Gong, L., & Zhou, X. (2019, August). RV-CNN: flexible and efficient instruction set for CNNs based on RISC-V processors. In International Symposium on Advanced Parallel Processing Technologies (pp. 3-14). Springer, Cham. The scope of protection of this invention is not limited to the above embodiments. Variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of this invention are included in this invention and are protected by the appended claims.

Claims

1. A convolutional accelerator based on an extension of the RISC-V instruction set architecture, characterized in that, The convolution accelerator is located in the computer system and is physically connected to memory via a memory bus. It introduces a data buffer and uses a buffer block strategy to improve the reusability of data for convolution operations and reduce the memory access requirements of convolution operations. The modules in the convolution accelerator include: an instruction fetch and instruction allocation module, a data transport module, a feature map buffer, a convolution kernel buffer, a convolution operation unit, a convolution logic operation control register group, an output buffer, and an adder; the convolution accelerator is equipped with custom extended instructions. The data transfer module is physically connected to the memory bus, convolution kernel buffer, feature map buffer, and output buffer. It is used to control the data copying function between the memory and the on-chip buffer of the convolution accelerator. It can complete the data transfer between the memory and the feature map buffer and / or the convolution kernel buffer and / or the output buffer according to different data transfer instructions. The feature map buffer is physically connected to the data transport module and the convolution operation unit, and is subject to the logical control of the convolution logic operation control register group. It is used to cache the feature map input for convolution operations. The feature map buffer adopts a ring logic structure and is divided into blocks with a Tile variable as the size. When the feature map buffer is full, the starting position of the feature map buffer is reset through relevant instructions, including mvtile, or the blocks on the feature map buffer are copied, thereby realizing the reuse of the data at the end of the feature map buffer. The convolution kernel buffer is physically connected to the data transport module and the convolution operation unit, and is subject to the logic control of the convolution logic operation control register group. It is used to cache the convolution kernel data during the convolution operation. The convolution kernel buffer is divided into blocks according to the amount of data in a single channel of the input convolution kernel. If the size of the convolution kernel buffer cannot be divided by the amount of data in a single channel of the convolution kernel, the total number of blocks on the convolution kernel buffer will be rounded down. The convolution accelerator also includes a FIFO memory, which is located between the convolution operation unit and the adder and is physically connected to both. The FIFO memory temporarily stores the result of the convolution operation performed by the convolution operation unit and the output buffer position specified in the instruction, and transmits the data to the adder.

2. The convolution accelerator as described in claim 1, characterized in that, The instruction fetching and instruction allocation module is physically connected to each module on the convolution accelerator. It allocates the custom extended instructions parsed by the processor during the decoding stage to other modules on the convolution accelerator and coordinates the operation of each module.

3. The convolution accelerator as described in claim 1, characterized in that, The convolution operation unit is physically connected to the feature map buffer, the convolution kernel buffer, the adder, and the output buffer, and is subject to the logical control of the convolution logic operation control register group. The convolution operation unit is composed of a multiply-adder or a pulse array. It reads feature map and / or convolution kernel data from the feature map buffer and / or the convolution kernel buffer as input for the convolution operation, and inputs the result of the convolution operation into the corresponding register in the RISC-V processor or outputs it to the subsequent output buffer.

4. The convolution accelerator as described in claim 1, characterized in that, The convolution logic operation control register group implements logical control over the convolution kernel buffer, feature map buffer, and convolution operation unit; the convolution logic operation control register group includes three sets of logic control registers: one or more block size registers, one or more convolution kernel height registers, and one or more convolution kernel width registers, which respectively control the logical block size, convolution kernel height, and convolution kernel width in the feature map buffer.

5. The convolution accelerator as described in claim 1, characterized in that, The adder is physically connected to the convolution operation unit and the output buffer. The output buffer is physically connected to the adder, the convolution operation unit and the data transport module. The output buffer is used to store the output of the convolution operation result. The adder is used to read the temporary output result of the corresponding channel in the convolution operation in the output buffer, accumulate it with the output result of the current round of convolution operation unit, and write the accumulated result back to the output buffer.

6. A method for accelerating convolution operations using the convolution accelerator according to any one of claims 1-5, characterized in that, The method includes the following steps: Step 1: Configure the convolution logic operation control register group in the convolution accelerator to determine the block size of the feature map buffer and the convolution kernel buffer; Step 2: Load and fix the convolution kernel data in batches, and logically divide the convolution kernel buffer into blocks; Step 3: Using the maximum number of convolutional kernel channels that the current convolutional buffer can hold as the upper bound, load the feature map data in batches and fill it into the feature map buffer; Step 4: The convolution operation unit reads data from the feature map buffer and the convolution kernel data buffer respectively, performs convolution operation, and sends the result of the convolution operation into the processor's register or the output buffer of the convolution accelerator according to different convolution instructions. Step 5: Load new feature map data. When data reuse occurs, reset the starting point and reserved length of the feature map buffer, perform convolution operations, and repeat the operation until all convolution operations are completed. Then, write the result data back to memory.

Images