RISC-V-based convolutional neural network acceleration processing system and coprocessor thereof

Abstract

The invention discloses a convolutional neural network acceleration processing system based on RISC-V. The convolutional neural network acceleration processing system comprises a main processor and a coprocessor, the main processor is used for sending a self-defined input data instruction to the coprocessor, and the coprocessor is used for sending a storage request signal to the main processor based on the input data instruction; the main processor is used for loading input data stored in the data tight coupling memory of the main processor into the coprocessor based on the storage request signal, and the coprocessor is used for caching the input data; after the loading of the input data is completed, the main controller is used for sending a convolution operation instruction to the coprocessor; the coprocessor is used for performing convolution calculation on input data based on the convolution operation instruction to obtain and cache output data; the main processor is also used for sending an output data instruction to the coprocessor; the co-processor is further to write back the output data to the main processor based on the output data instruction. According to the invention, the problems of low CPU computing parallelism and over-high GPU power consumption can be solved.

Description

Technical Field

[0001] The present invention relates to the field of semiconductor integrated circuit technology, and in particular to a RISC-V-based convolutional neural network acceleration processing system and a coprocessor thereof. Background Art

[0002] As people's demands for convolutional neural networks become increasingly diversified and the requirements for computational accuracy continue to increase, the number of layers of convolutional neural networks has gradually increased, and the scale and number of parameters have also increased significantly. If the algorithm is deployed on embedded systems such as video surveillance and mobile terminals, the insufficient computing performance of embedded processors makes it difficult to meet the computing power requirements. Therefore, hardware acceleration technology for convolutional neural networks has become one of the current research hotspots in academia and industry.

[0003] Domestic and foreign scholars have accelerated convolutional neural network operations in different ways. Xu Genglin et al. proposed a neural network coprocessor controlled by 32-bit dedicated instructions and with a switchable convolution calculation structure to realize the accelerated calculation of complex neural networks on the FPGA platform. The MAC structure or multiplication-addition tree structure can be controlled by instructions. The coprocessor has a good acceleration effect. However, in order to pursue flexibility, FPGA requires each of the most basic CLBs to be configurable, resulting in a large amount of redundancy, which is inefficient. The limited bandwidth and on-chip storage also limit the application of FPGA in neural network accelerators. Liu Fang et al. proposed to customize a dedicated acceleration processing system RCP (RISC-V CNNProcessor) for convolutional neural networks based on the RISC-V architecture, and accelerate convolution operations from a hardware perspective through customized processor technology. However, the acceleration processing system does not design output data and output data buffer modules. Multiple convolution operations require frequent reading of data from memory, wasting unnecessary time. Miao Ruixia et al. used the 8-bit data operation instructions unique to the P extension instruction set of RISC-V to accelerate the neural network convolution algorithm on the embedded platform ARM CMSIS-NN, and optimized the low memory usage efficiency caused by the lack of DSP instruction 8-bit data operation instructions in the ARM CMSIS-NN library. This method also did not design a data cache module, and the acceleration effect was not ideal. Fu Siyang et al. designed and implemented a low-power embedded convolutional neural network acceleration processing system, and optimized the convolution operation by performing convolution operations on only one row of data in a single cycle, followed by an accumulator to accumulate the row convolution results to obtain the operation results. This design has low versatility and only supports convolution calculations with a convolution kernel size of 5*5. It does not support convolution operations with a convolution kernel size greater than 5*5, and when the convolution size is less than 5*5, it will cause idle resources. Yang Weike et al. proposed a convolutional neural network acceleration module based on the Eyeriss structure based on the Rocket-Chip open source processor. This accelerator structure greatly reduces the time consumption of the convolution layer by reusing data in the horizontal direction (convolution kernel weights), vertical direction (output calculation results) and diagonal direction (input image). However, the Eyeriss structure in this design easily causes waste of PE computing unit resources. The above work has problems such as large resource consumption, unsatisfactory acceleration effect and low versatility. Therefore, designing a high-performance density coprocessor that can accelerate convolution operations, has low resource consumption and is applicable to different convolution sizes is an effective way to solve the above problems. Summary of the invention

[0004] In order to solve the technical problems existing in the background technology, the present invention proposes a RISC-V-based convolutional neural network acceleration processing system and its coprocessor.

[0005] In a first aspect, the present invention proposes a RISC-V-based convolutional neural network acceleration processing system, comprising: a main processor and a coprocessor, wherein the main processor and the coprocessor are connected via an extended acceleration interface;

[0006] The main processor is used to send a custom input data instruction to the coprocessor through the extended acceleration interface: wherein the input data instruction includes a load input feature map instruction and a load weight instruction;

[0007] The coprocessor is used to send a storage request signal to the main processor through the extended acceleration interface after receiving the input data instruction sent by the main processor;

[0008] The main processor is used to load the input data stored in the data tightly coupled memory of the main processor into the coprocessor through the extended acceleration interface after receiving the storage request signal, and the coprocessor is used to cache the input data; wherein the input data includes an input feature map and weight parameters;

[0009] When the input data is loaded, the main controller is used to send a convolution operation instruction to the coprocessor through the extended acceleration interface;

[0010] The coprocessor is used to perform convolution calculation on the input data after receiving the convolution operation instruction, obtain output data, and cache the output data; wherein the output data includes an output feature map;

[0011] The main processor is also used to send an output data instruction to the coprocessor through the extended acceleration interface; wherein the output data instruction includes an instruction to derive an output feature map;

[0012] The coprocessor is further used to write the output data back to the data tightly coupled memory of the main processor through the extended acceleration interface after receiving the output data instruction.

[0013] In this embodiment, the coprocessor includes a top-level control module, an input data cache module, a convolution operation module, and an output data cache module;

[0014] The top-level control module is connected to the main processor through the extended acceleration interface. The top-level control module is responsible for communicating with the main processor to receive input data instructions, input data, convolution operation instructions and output data instructions, and convert the input data instructions, convolution operation instructions and output data instructions into corresponding enable signals respectively, and then transmit the enable signals to the corresponding modules;

[0015] The input buffer module is used to buffer the input data instructions received by the top-level control module from the main processor, and transmit the corresponding parameters and data in the input data instructions to the convolution operation module;

[0016] The convolution module is used to perform convolution operation according to the enable signal sent by the top-level control module and the corresponding parameters and data transmitted by the input buffer module, store the output data obtained by the convolution operation into the output buffer module, and return the convolution completion signal to the top-level control module;

[0017] The output cache module is used to cache the output feature map data obtained by the convolution operation, and is also used to transmit the output data to the top-level control module according to the enable signal sent by the top-level control module. The top-level control module is also used to write the output data back to the data tightly coupled memory of the main processor through the extended acceleration interface.

[0018] In this embodiment, the convolution operation module includes a reconfigurable processing unit and a reconfiguration control unit;

[0019] Wherein, the reconstruction control unit is used to generate a reconstruction instruction according to the size of the input convolution kernel in the input data instruction;

[0020] The reconfigurable processing unit includes a plurality of PE processing units, and the plurality of PE processing units are used to form a PE array; wherein the reconfigurable processing unit is used to reconstruct the PE array according to a reconstruction instruction, and perform convolution calculation on input data after reconstruction to obtain output data.

[0021] In this embodiment, the reconfigurable processing unit includes 50 PE processing units, and the reconfigurable processing unit can be reconfigured to form 5 3*3, 2 5*5 or 1 7*7 PE arrays.

[0022] In this embodiment, the coprocessor uses 1111011 as the opcode, and the opcode includes four instructions: Custom3_load_imp, Custom3_store_omp, Custom3_load_weigh, and Custom3_conv;

[0023] Among them, the Custom3_load_imp instruction is the instruction for loading the input feature map;

[0024] Custom3_store_omp is an instruction to export the output feature map, which is used to output the convolved output feature map to the DTCM of the main processor;

[0025] Custom3_load_weigh is a load input weight instruction, which is used to transfer the weight parameters stored in the main processor DTCM to the coprocessor;

[0026] Custom3_conv is a convolution operation instruction, which is used to start the convolution operation.

[0027] In a second aspect, the present invention also proposes a coprocessor in a RISC-V-based convolutional neural network acceleration processing system, comprising: a top-level control module, an input data cache module, a convolution operation module, and an output data cache module;

[0028] The top-level control module is used to connect to the main processor through the extended acceleration interface. The top-level control module is responsible for communicating with the main processor to receive input data instructions, input data, convolution operation instructions and output data instructions, and convert the input data instructions, convolution operation instructions and output data instructions into corresponding enable signals respectively, and then transmit the enable signals to the corresponding modules;

[0029] The input buffer module is used to buffer the input data instructions received by the top-level control module from the main processor, and transmit the corresponding parameters and data in the input data instructions to the convolution operation module;

[0030] The convolution module is used to perform convolution operation according to the enable signal sent by the top-level control module and the corresponding parameters and data transmitted by the input buffer module, store the output data obtained by the convolution operation into the output buffer module, and return the convolution completion signal to the top-level control module;

[0031] The output cache module is used to cache the output feature map data obtained by the convolution operation, and is also used to transmit the output data to the top-level control module according to the enable signal sent by the top-level control module. The top-level control module is also used to write the output data back to the data tightly coupled memory of the main processor through the extended acceleration interface;

[0032] Among them, the input data instruction includes an instruction to load an input feature map and an instruction to load an input weight, the input data includes an input feature map and weight parameters, the output data includes an output feature map, and the output data instruction includes an instruction to export an output feature map.

[0033] In this embodiment, the convolution operation module includes a reconfigurable processing unit and a reconfiguration control unit;

[0034] Wherein, the reconstruction control unit is used to generate a reconstruction instruction according to the size of the input convolution kernel in the input data instruction;

[0035] The reconfigurable processing unit includes a plurality of PE processing units, and the plurality of PE processing units are used to form a PE array; wherein the reconfigurable processing unit is used to reconstruct the PE array according to a reconstruction instruction, and perform convolution calculation on input data after reconstruction to obtain output data.

[0036] In this embodiment, the reconfigurable processing unit includes 50 PE processing units, and the reconfigurable processing unit can be reconfigured to form 5 3*3, 2 5*5 or 1 7*7 PE arrays.

[0037] In this embodiment, the coprocessor uses 1111011 as opcode, and the opcode includes four instructions: Custom3_load_imp, Custom3_store_omp, Custom3_load_weigh, and Custom3_conv;

[0038] Among them, the Custom3_load_imp instruction is the instruction for loading the input feature map;

[0039] Custom3_store_omp is an instruction to export the output feature map, which is used to output the convolved output feature map to the DTCM of the main processor;

[0040] Custom3_load_weigh is a load input weight instruction, which is used to transfer the weight parameters stored in the main processor DTCM to the coprocessor;

[0041] Custom3_conv is a convolution operation instruction, which is used to start the convolution operation.

[0042] In the present invention, the proposed RISC-V-based convolutional neural network acceleration processing system and its coprocessor adopt heterogeneous cooperation between the main processor and the coprocessor, use the main processor to process the completed tasks, and use the coprocessor to process convolutional neural network related operations. The coprocessor is connected to the main processor by extending the acceleration interface, which can solve the problems of low CPU calculation parallelism and excessive GPU power consumption in traditional convolutional neural network acceleration methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A block diagram of a RISC-V-based convolutional neural network acceleration processing system in one embodiment of the present invention.

[0044] Figure 2 This is the instruction encoding format of the coprocessor in one embodiment of the present invention.

[0045] Figure 3 FIG. 4 is a structure of a PE processing unit in an embodiment of the present invention.

[0046] Figure 4 This is a schematic structural diagram of a PE array in an embodiment of the present invention.

[0047] Figure 5 It is a schematic diagram of five 3*3 PE arrays in one embodiment of the present invention.

[0048] Figure 6 It is a schematic diagram of two 5*5 PE arrays in one embodiment of the present invention.

[0049] Figure 7 It is a schematic diagram of a 7*7 PE array in one embodiment of the present invention.

[0050] Figure 8 A block diagram of a coprocessor in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0052] Reference Figure 1 , a RISC-V-based convolutional neural network acceleration processing system proposed in the present invention includes: a main processor and a coprocessor, the main processor and the coprocessor are connected via an extension accelerator interface (EAI);

[0053] The main processor is used to send a custom input data instruction to the coprocessor through the extended acceleration interface: wherein the input data instruction includes a load input feature map instruction and a load weight instruction;

[0054] The coprocessor is used to send a storage request signal to the main processor through the extended acceleration interface after receiving the input data instruction sent by the main processor;

[0055] The main processor is used to load the input data stored in the data tightly coupled memory of the main processor into the coprocessor through the extended acceleration interface after receiving the storage request signal, and the coprocessor is used to cache the input data; wherein the input data includes an input feature map and weight parameters;

[0056] When the input data is loaded, the main controller is used to send a convolution operation instruction to the coprocessor through the extended acceleration interface;

[0057] The coprocessor is used to perform convolution calculation on the input data after receiving the convolution operation instruction, obtain output data, and cache the output data; wherein the output data includes an output feature map;

[0058] The main processor is also used to send an output data instruction to the coprocessor through the extended acceleration interface; wherein the output data instruction includes an instruction to derive an output feature map;

[0059] The coprocessor is further used to write the output data back to the data tightly coupled memory of the main processor through the extended acceleration interface after receiving the output data instruction.

[0060] The present invention adopts the heterogeneous cooperation of the main processor and the coprocessor, uses the main processor to process the completed tasks, and uses the coprocessor to process the convolutional neural network related operations. The coprocessor is connected to the main processor by extending the acceleration interface, which can solve the problems of low CPU calculation parallelism and high GPU power consumption in traditional convolutional neural network acceleration methods.

[0061] like Figure 8 As shown, in this embodiment, the coprocessor includes: a top-level control module, an input data cache module, a convolution operation module and an output data cache module;

[0062] The top-level control module is used to connect to the main processor through the extended acceleration interface. The top-level control module is responsible for communicating with the main processor to receive input data instructions, input data, convolution operation instructions and output data instructions, and convert the input data instructions, convolution operation instructions and output data instructions into corresponding enable signals respectively, and then transmit the enable signals to the corresponding modules;

[0063] The input buffer module is used to buffer the input data instructions received by the top-level control module from the main processor, and transmit the corresponding parameters and data in the input data instructions to the convolution operation module;

[0064] The convolution module is used to perform convolution operation according to the enable signal sent by the top-level control module and the corresponding parameters and data transmitted by the input buffer module, store the output data obtained by the convolution operation into the output buffer module, and return the convolution completion signal to the top-level control module;

[0065] The output cache module is used to cache the output feature map data obtained by the convolution operation, and is also used to transmit the output data to the top-level control module according to the enable signal sent by the top-level control module. The top-level control module is also used to write the output data back to the data tightly coupled memory of the main processor through the extended acceleration interface;

[0066] Among them, the input data instruction includes an instruction to load an input feature map and an instruction to load an input weight, the input data includes an input feature map and weight parameters, the output data includes an output feature map, and the output data instruction includes an instruction to export an output feature map.

[0067] During the convolution operation, the PE array has the characteristics of regularity, scalability, and reconfigurability, which can satisfy the operation of a large amount of different data at the same time. However, the conventional PE array can only calculate the convolution window of fixed size, such as the commonly used 3*3, 5*5, 7*7, etc., but this single-size convolution window will cause a lot of PE resource waste when the convolution kernel size changes.

[0068] Taking the design of a 5*5 convolution window as an example, at the beginning of the convolution operation, when the input feature map size is large, the 5*5 PE array can meet the current size operation, but as the convolution proceeds, the input feature map size continues to decrease, and the convolution kernel size will also decrease. Using a 3*3 convolution kernel, it is impossible to perform convolution operations on the 5*5 PE array. If PE arrays of multiple sizes are designed in the accelerator to solve this problem, it is bound to cause idle waste of redundant PE processing units. For example, when calculating the convolution operation with a convolution window size of 3*3, the PE arrays of sizes 5*5 and 7*7 will be idle.

[0069] Therefore, in order to satisfy the convolution operation of convolution windows of different sizes, greatly improve the versatility of the acceleration processing system, and avoid the waste of PE resources when the convolution kernel size is changed, in this embodiment, the convolution operation module includes a reconfigurable processing unit and a reconstruction control unit; wherein the reconfigurable processing unit includes a plurality of PE processing units, and the plurality of PE processing units form a PE array; the reconstruction control unit is used to reconstruct the PE array in the reconfigurable processing unit according to the size of the input convolution kernel in the input data instruction, so as to form a plurality of PE arrays of different sizes;

[0070] The reconstructed reconfigurable processing unit is used to perform convolution calculation on the input data to obtain output data.

[0071] Specifically, the input convolution kernel size can be configured in the custom load input weights instruction.

[0072] in, Figure 3 The hardware structure diagram of the PE processing unit required for constructing a PE array with fixed weight parameters in this embodiment. The PE processing unit has two input signals and two output signals, wherein the weight parameter is fixed after entering the PE processing unit as one input signal filter, and the other input signal is used to import the input feature map data left_ifmap, the input feature map data is multiplied by the weight parameter once, and then directly assigned to the output signal output right_ifmap, and the output signal is used as the input feature map data of the right PE processing unit to realize the right shift of the input feature map data; in addition, the product obtained by multiplying the input feature map data and the weight parameter in the current PE processing unit is assigned to another output signal sum_down, which is output and added to the sum_down calculated by other PE processing units in the PE array to obtain the final output feature map data.

[0073] like Figure 4As shown in the figure, when the PE array is a 3*3 PE array, the convolution process of the reconfigurable processing unit includes: after the coprocessor receives the convolution operation instruction, the weight parameter input PE array is arranged and remains unchanged, and the first three rows of the input feature map flow into the PE array in sequence. After three clock cycles, the input feature map data fills the PE array; at this time, the PE processing unit starts to perform the product operation and outputs the product to the adder. The nine product values ​​are added at the same time, and the sum obtained is the convolution value of the output feature map in the current window; in addition, the input feature map data after the product operation starts to shift right until the convolution operation of the current row is completed and jumps to the next row. Among them, the multiplication operation in the PE processing unit and the addition operation after its output product can be completed within one clock cycle.

[0074] In one specific embodiment, the reconfigurable processing unit in this embodiment includes 50 PE processing units. Through the reconstruction control unit, these 50 PE processing units can reconstruct the PE array according to the size of the input convolution kernel, which can meet the convolution operations of 5 convolution windows of different sizes such as 3*3, 2 5*5, and 1 7*7, greatly improving the versatility of the coprocessor and avoiding the waste of PE resources when the convolution kernel size is changed.

[0075] For example, when the convolution kernel size is 3*3, up to 5 convolution kernels can be convolved at the same time. Figure 5 As shown in the figure, the weight parameters will be written into the corresponding PE array in sequence according to the number of convolution kernels configured in the input weight instruction and remain fixed. The input feature maps will flow into the PE array in sequence to perform convolution operations with the weight parameters. At this time, only 5 PE processing units are idle.

[0076] For example, when the convolution kernel size is 7*7, the PE processing unit will be reconstructed, and 49 PE processing units will form a PE array, which can perform convolution operations on at most one feature map, such as Figure 6 As shown, only one PE processing unit is idle at this time.

[0077] For example, when the convolution kernel size is 7*7, the PE processing unit will be reconstructed, and 49 PE processing units will form a PE array, which can perform convolution operations on at most one feature map, such as Figure 7 As shown, only one PE processing unit is idle at this time.

[0078] It should be noted that the RISC-V architecture supports instructions of various lengths, such as 16-bit and 32-bit. Among them, the four groups of Custom type instructions in the 32-bit instructions support users to customize instructions according to their needs. The 0th to 6th bits of the Custom instruction are the opcode encoding segments used to distinguish different instructions, as shown in Table 1.

[0079] Table 1 Predefined instruction types

[0080]

[0081] Therefore, in order to accelerate, this embodiment designs the instruction encoding format of the convolutional neural network acceleration processing system based on custom instructions, such as Figure 2 As shown. Among them, the opcode segment is 1111011, and its corresponding predefined instruction type is Custom_3; Xd, xs1, and xs2 are used to control whether to write to the target register rd and whether to read the source registers rs1 and rs2. If the value of xd is 1, it means that the data needs to be written back to the target register rd; if the value of xd is 0, it means that the data does not need to be written back to the target register rd. If the value of xs1 is 1, it means that the data in the source register rs1 needs to be read; if the value of xs1 is 0, it means that the data in the source register rs1 does not need to be read. If the value of xs2 is 1, it means that the data in the source register rs2 needs to be read; if the value of xs2 is 0, it means that the data in the source register rs2 does not need to be read. Funct7 is an additional encoding area for instructions, which is mainly used to distinguish Custom_3 instructions with different encodings.

[0082] This embodiment designs four instructions as shown in Table 2 to accelerate the convolution operation that accounts for more than 90% of the CNN operation. Among them, funct3 is the collection of xs1, xs2, and xd, and is located at the 12th to 14th bits of the instruction.

[0083] Table 2 Custom instruction codes and functions

[0084]

[0085] Among them, Custom3_load_imp is the instruction for loading the input feature map. The instruction funct7 is 0000001, which is used to distinguish it from other instructions. Funt3 is 011, which means that the source registers rs1 and rs2 are used instead of the target register rd, because the instruction only needs to read the data from the source register without writing it back. In addition, rs1 is used to save the address of the input feature map stored in the DTCM of the main processor, and rs2 is used to save the size of the input feature map. The size of the input feature map can be controlled by configuring the source register rs2.

[0086] Among them, Custom3_store_omp is the instruction for exporting the output feature map, which is used to output the convolved output feature map to the DTCM of the main processor. Its funct7 is 0000010, and funct3 is 011. This instruction also does not need to read the values ​​of the source registers rs1 and rs2, and does not need to write back to the destination register rd, where rs1 stores the address of the output feature map that needs to be exported to the main processor DTCM, and rs2 stores the size of the output feature map.

[0087] Among them, Custom3_load_weigh is a load input weight instruction, which is used to transfer the weight parameters stored in the main processor DTCM to the coprocessor, its funct7 is 0000011, funct3 is 011, where rs1 saves the address information of the weight parameters, the lower three bits of rs2 are used to save the size of the convolution kernel, and the upper three bits are used to save the number of convolution kernels. By configuring the source register rs2 of the instruction, the number and size of the convolution kernels of the convolution operation can be controlled. Specifically, the coprocessor designed in this embodiment supports up to 5 3*3 convolution kernels to perform convolution operations simultaneously.

[0088] Among them, Custom3_conv is a convolution operation instruction, which is used to start the convolution operation, so there is no need to configure the source register and the target register.

[0089] In one specific embodiment, if only one convolution operation is required, the main processor only needs to call the Custom3_load_imp instruction once to load the input feature map, and then call Custom3_load_weigh to load the weight parameters, and then call the Custom3_conv instruction once to start the convolution operation. After the coprocessor convolution operation is completed and the data is temporarily stored in the output data cache module, the Custom3_store_omp instruction is called to export the output feature map. Of course, if the number of convolution kernels required is more than the preset maximum number of convolution kernels, the above operations only need to be repeated multiple times.

[0090] In a second aspect, the present invention further proposes a coprocessor in a RISC-V-based convolutional neural network acceleration processing system, wherein the coprocessor is used to connect to a main processor through an extended acceleration interface;

[0091] The coprocessor includes: a top-level control module, an input data cache module, a convolution operation module and an output data cache module;

[0092] The top-level control module is used to connect to the main processor through the extended acceleration interface. The top-level control module is responsible for communicating with the main processor to receive input data instructions, input data, convolution operation instructions and output data instructions, and convert the input data instructions, convolution operation instructions and output data instructions into corresponding enable signals respectively, and then transmit the enable signals to the corresponding modules;

[0093] The input buffer module is used to buffer the input data instructions received by the top-level control module from the main processor, and transmit the corresponding parameters and data in the input data instructions to the convolution operation module;

[0094] The convolution module is used to perform convolution operation according to the enable signal sent by the top-level control module and the corresponding parameters and data transmitted by the input buffer module, store the output data obtained by the convolution operation into the output buffer module, and return the convolution completion signal to the top-level control module;

[0095] The output cache module is used to cache the output feature map data obtained by the convolution operation, and is also used to transmit the output data to the top-level control module according to the enable signal sent by the top-level control module. The top-level control module is also used to write the output data back to the data tightly coupled memory of the main processor through the extended acceleration interface;

[0096] Among them, the input data instruction includes an instruction to load an input feature map and an instruction to load an input weight, the input data includes an input feature map and weight parameters, the output data includes an output feature map, and the output data instruction includes an instruction to export an output feature map.

[0097] The coprocessor in this embodiment can cooperate with the main processor, use the main processor to process the completed tasks, and use the coprocessor to process convolutional neural network related operations. By connecting the coprocessor to the main processor through an extended acceleration interface, problems in traditional convolutional neural network acceleration methods can be solved, such as low CPU calculation parallelism and excessive GPU power consumption.

[0098] In this embodiment, the convolution operation module includes a reconfigurable processing unit and a reconfiguration control unit;

[0099] Wherein, the reconstruction control unit is used to generate a reconstruction instruction according to the size of the input convolution kernel in the input data instruction;

[0100] The reconfigurable processing unit includes a plurality of PE processing units, and the plurality of PE processing units are used to form a PE array; wherein the reconfigurable processing unit is used to reconstruct the PE array according to a reconstruction instruction, and perform convolution calculation on input data after reconstruction to obtain output data.

[0101] In this embodiment, the reconfigurable processing unit includes 50 PE processing units, and the reconfigurable processing unit can be reconfigured to form 5 3*3, 2 5*5 or 1 7*7 PE arrays.

[0102] In this embodiment, the coprocessor uses 1111011 as the opcode, and the opcode includes four instructions: Custom3_load_imp, Custom3_store_omp, Custom3_load_weigh, and Custom3_conv;

[0103] Among them, the Custom3_load_imp instruction is the instruction for loading the input feature map;

[0104] Custom3_store_omp is an instruction to export the output feature map, which is used to output the convolved output feature map to the DTCM of the main processor;

[0105] Custom3_load_weigh is a load input weight instruction, which is used to transfer the weight parameters stored in the main processor DTCM to the coprocessor;

[0106] Custom3_conv is a convolution operation instruction, which is used to start the convolution operation.

[0107] Specifically, the convolution operation implemented by the ordinary processor and the convolution operation using the coprocessor were tested respectively by using the same set of input feature maps and weight parameters, and the results shown in Table 3 were obtained.

[0108] As can be seen from Table 3, compared with ordinary processors, the acceleration effect of the coprocessor can be up to 129 times.

[0109] Table 3

[0110]

[0111] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A RISC-V-based convolutional neural network acceleration processing system, characterized in that: include: A main processor and a coprocessor, the main processor and the coprocessor are connected via an extended acceleration interface; The main processor is used to send a custom input data instruction to the coprocessor through the extended acceleration interface: wherein the input data instruction includes a load input feature map instruction and a load weight instruction; The coprocessor is used to send a storage request signal to the main processor through the extended acceleration interface after receiving the input data instruction sent by the main processor; The main processor is used to load the input data stored in the data tightly coupled memory of the main processor into the coprocessor through the extended acceleration interface after receiving the storage request signal, and the coprocessor is used to cache the input data; wherein the input data includes an input feature map and weight parameters; When the input data is loaded, the main controller is used to send a convolution operation instruction to the coprocessor through the extended acceleration interface; The coprocessor is used to perform convolution calculation on the input data after receiving the convolution operation instruction, obtain output data, and cache the output data; wherein the output data includes an output feature map; The main processor is also used to send an output data instruction to the coprocessor through the extended acceleration interface; wherein the output data instruction includes an instruction to derive an output feature map; The coprocessor is further used to write the output data back to the data tightly coupled memory of the main processor through the extended acceleration interface after receiving the output data instruction.

2. The RISC-V-based convolutional neural network acceleration processing system according to claim 1, characterized in that: The coprocessor includes a top-level control module, an input data cache module, a convolution operation module and an output data cache module; The top-level control module is connected to the main processor through the extended acceleration interface. The top-level control module is responsible for communicating with the main processor to receive input data instructions, input data, convolution operation instructions and output data instructions, and convert the input data instructions, convolution operation instructions and output data instructions into corresponding enable signals respectively, and then transmit the enable signals to the corresponding modules; The input buffer module is used to buffer the input data instructions received by the top-level control module from the main processor, and transmit the corresponding parameters and data in the input data instructions to the convolution operation module; The convolution module is used to perform convolution operation according to the enable signal sent by the top-level control module and the corresponding parameters and data transmitted by the input buffer module, store the output data obtained by the convolution operation into the output buffer module, and return the convolution completion signal to the top-level control module; The output cache module is used to cache the output feature map data obtained by the convolution operation, and is also used to transmit the output data to the top-level control module according to the enable signal sent by the top-level control module. The top-level control module is also used to write the output data back to the data tightly coupled memory of the main processor through the extended acceleration interface.

3. The RISC-V-based convolutional neural network acceleration processing system according to claim 2, characterized in that: The convolution operation module includes a reconfigurable processing unit and a reconstruction control unit; Wherein, the reconstruction control unit is used to generate a reconstruction instruction according to the size of the input convolution kernel in the input data instruction; The reconfigurable processing unit includes a plurality of PE processing units, and the plurality of PE processing units are used to form a PE array; wherein the reconfigurable processing unit is used to reconstruct the PE array according to a reconstruction instruction, and perform convolution calculation on input data after reconstruction to obtain output data.

4. The RISC-V-based convolutional neural network acceleration processing system according to claim 3, characterized in that: The reconfigurable processing unit includes 50 PE processing units, and the reconfigurable processing unit can be reconfigured to form 5 3*3, 2 5*5 or 1 7*7 PE arrays.

5. The RISC-V-based convolutional neural network acceleration processing system according to claim 1, characterized in that: The coprocessor uses 1111011 as the opcode, and the opcode includes four instructions: Custom3_load_imp, Custom3_store_omp, Custom3_load_weigh, and Custom3_conv; Among them, the Custom3_load_imp instruction is the instruction for loading the input feature map; Custom3_store_omp is an instruction to export the output feature map, which is used to output the convolved output feature map to the DTCM of the main processor; Custom3_load_weigh is a load input weight instruction, which is used to transfer the weight parameters stored in the main processor DTCM to the coprocessor; Custom3_conv is a convolution operation instruction, which is used to start the convolution operation.

6. A coprocessor in a RISC-V-based convolutional neural network acceleration processing system, characterized in that: include: Top-level control module, input data cache module, convolution operation module and output data cache module; The top-level control module is used to connect to the main processor through the extended acceleration interface. The top-level control module is responsible for communicating with the main processor to receive input data instructions, input data, convolution operation instructions and output data instructions, and convert the input data instructions, convolution operation instructions and output data instructions into corresponding enable signals respectively, and then transmit the enable signals to the corresponding modules; The input buffer module is used to buffer the input data instructions received by the top-level control module from the main processor, and transmit the corresponding parameters and data in the input data instructions to the convolution operation module; The convolution module is used to perform convolution operation according to the enable signal sent by the top-level control module and the corresponding parameters and data transmitted by the input buffer module, store the output data obtained by the convolution operation into the output buffer module, and return the convolution completion signal to the top-level control module; The output cache module is used to cache the output feature map data obtained by the convolution operation, and is also used to transmit the output data to the top-level control module according to the enable signal sent by the top-level control module. The top-level control module is also used to write the output data back to the data tightly coupled memory of the main processor through the extended acceleration interface; Among them, the input data instruction includes an instruction to load an input feature map and an instruction to load an input weight, the input data includes an input feature map and weight parameters, the output data includes an output feature map, and the output data instruction includes an instruction to export an output feature map.

7. The coprocessor of the RISC-V-based convolutional neural network acceleration processing system according to claim 6, characterized in that: The convolution operation module includes a reconfigurable processing unit and a reconstruction control unit; Wherein, the reconstruction control unit is used to generate a reconstruction instruction according to the size of the input convolution kernel in the input data instruction; The reconfigurable processing unit includes a plurality of PE processing units, and the plurality of PE processing units are used to form a PE array; wherein the reconfigurable processing unit is used to reconstruct the PE array according to a reconstruction instruction, and perform convolution calculation on input data after reconstruction to obtain output data.

8. The coprocessor of the RISC-V-based convolutional neural network acceleration processing system according to claim 7, characterized in that: The reconfigurable processing unit includes 50 PE processing units, and the reconfigurable processing unit can be reconfigured to form 5 3*3, 2 5*5 or 1 7*7 PE arrays.

9. The coprocessor of the RISC-V-based convolutional neural network acceleration processing system according to claim 1, characterized in that: The coprocessor uses 1111011 as opcode, and opcode includes four instructions: Custom3_load_imp, Custom3_store_omp, Custom3_load_weigh, and Custom3_conv; Among them, the Custom3_load_imp instruction is the instruction for loading the input feature map; Custom3_store_omp is an instruction to export the output feature map, which is used to output the convolved output feature map to the DTCM of the main processor; Custom3_load_weigh is a load input weight instruction, which is used to transfer the weight parameters stored in the main processor DTCM to the coprocessor; Custom3_conv is a convolution operation instruction, which is used to start the convolution operation.

Images