Coprocessor based on RISC-V architecture and processing system thereof

Abstract

The invention relates to a coprocessor based on an RISC-V architecture and a processing system thereof. The coprocessor comprises a decoding unit, a control unit, an arithmetic unit, a cross-clock transmission unit and a cache unit. The decoding unit receives the value of the source register sent by the main processor and decodes the value to output an operation parameter, the control unit controls the working process of the operation unit based on the operation parameter, and the cross-clock transmission unit communicates with the main processor and comprises transmission of configuration parameters and feedback signals. Therefore, the main coprocessor can run in different clock domains; the cache unit is used for caching part of weight data and all bias data required in single operation, can be used for realizing operation of an expansion instruction, does not occupy the storage space of the main processor, and greatly improves the operation efficiency of the neural network.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a coprocessor based on RISC-V architecture and a processing system thereof. Background Art

[0002] Edge computing is a distributed computing paradigm that places computing, storage, and network services close to the data source or user's location to reduce latency, improve response speed, and enhance data security. This computing method is particularly important for application scenarios that require real-time processing of large amounts of data, such as the Internet of Things, smart manufacturing, smart cities, and self-driving cars. In the edge computing model, data does not have to be transmitted to a remote data center for processing, but is processed instantly at or near the location where the data is generated, which places dual requirements on the processor to have both efficient computing capabilities and low power consumption and low cost.

[0003] The drawback of the prior art is that in traditional processor systems, neural network calculations often face the problems of low efficiency and large storage space usage. Summary of the invention

[0004] The purpose of this application is to overcome the shortcomings of the prior art and provide a coprocessor based on the RISC-V architecture, which can be used to implement the operation of extended instructions without occupying the storage space of the main processor, thereby greatly improving the efficiency of neural network operations.

[0005] The purpose of this application is achieved through the following technical solutions:

[0006] The coprocessor includes:

[0007] A decoding unit receives the value of the source register sent by the main processor and decodes it to output an operation parameter;

[0008] A control unit and a computing unit, wherein the control unit controls the working process of the computing unit based on the operating parameters;

[0009] A cross-clock transmission unit communicates with the main processor, including the transmission of configuration parameters and feedback signals, thereby supporting the main coprocessor to run in different clock domains;

[0010] The cache unit is used to cache part of the weight data and all the bias data required in a single operation.

[0011] Different from the traditional operation mode, this application is based on the principle of distributed computing and places the neural network operations with high computing load and large memory usage on the branch processor. That is, a coprocessor is constructed to assist the main processor in calculating the extended instructions, thereby releasing the computing power and storage space of the main processor and greatly improving the computing efficiency.

[0012] Exemplarily, the decoding unit decodes the value of the source register sent by the main processor, reads the operand of the source register and splits it, determines the specific operation referred to by the instruction, obtains the operation parameters required for the operation and sends them to the control unit.

[0013] Exemplarily, the control unit includes a memory management module and an operation management module;

[0014] in,

[0015] The operation management module dispatches the computing tasks to the operation unit according to the operation parameters given by the decoding unit, and controls the working process of the operation unit by receiving the feedback signal from the operation unit;

[0016] The memory management module is responsible for the management of data transmission between the coprocessor and the external storage unit, including the calculation and allocation of memory addresses and the jump of memory read and write status.

[0017] Exemplarily, the operation unit adopts a pipeline architecture consisting of a booth coded multiplier and a Wallace number structure adder, and the calculation is performed in a non-blocking manner.

[0018] Exemplarily, the operation unit includes a row cache module, a convolution operation module, a pooling module, an activation function calculation module, a quantization module and a data storage module. The modules are completely decoupled and have exclusive enable signal ports and parameter input ports.

[0019] Exemplarily, the row buffer module is composed of two FIFOs and outputs a 3×3 sliding window in parallel, for converting serial data input into parallel multi-source data output.

[0020] Exemplarily, the convolution operation module includes two parts: a 1×1 convolution module and a 3×3 convolution module; in a single operation process, each module can simultaneously operate on four output channels, and in the output channel operation process, the four input channels are also calculated simultaneously.

[0021] Exemplarily, the data temporary storage module is used to cache the accumulated results of all channels that have not been completed, and superimpose the existing results with the cached data and then store them. When all channels have completed the operation, the memory data is read out and added with the bias data.

[0022] Exemplarily, the quantization module is used to perform quantization calculation on the convolution result with a width of more than eight bits and the relevant input parameters, and convert the output result back into an integer with a width of eight bits.

[0023] Exemplarily, the activation function calculation module supports Relu function operation and LeakyRelu function operation, wherein the selection of the activation function can be achieved by configuring parameters.

[0024] Exemplarily, the pooling module has a built-in row cache unit suitable for pooling calculations and related control logic, which converts the input serial data into two rows of parallel data. The control logic is responsible for sampling and outputting the input data according to specific parameter configurations.

[0025] Exemplarily, the cache unit is composed of a weight data cache module and a bias data cache module, which are used to cache weight data and bias data respectively;

[0026] When performing convolution operations and quantization operations, the corresponding weight data and bias data are effectively output according to the preset reading order.

[0027] The second aspect of the present application further provides a processing system of a coprocessor based on a RISC-V architecture, the system comprising:

[0028] Main processor;

[0029] and at least one external memory;

[0030] and coprocessors;

[0031] The coprocessor responds to the extended instructions of the main processor and completes the computing tasks based on the extended instructions. The external memory serves as the memory medium of the coprocessor to access the processing and data to be processed.

[0032] Furthermore, the main processor includes a nice module for processing extended instructions, and the coprocessor is connected to the nice module to complete communication.

[0033] The beneficial effects of this application are:

[0034] 1) Taking advantage of the scalable nature of instructions in the RISC-V instruction set architecture, the coprocessor is directly deployed and controlled by a special extended instruction set, and the model can be built at the instruction level, which can reduce the number of required instructions and avoid the problems of redundant instructions and large clock cycle consumption that occur in traditional processors when processing neural networks.

[0035] 2) Integrate a variety of neural network operators. Users can configure the corresponding internal parameters of the source register according to their own needs to enable and configure the corresponding operators.

[0036] 3) The operation units are all composed of high-performance multipliers or high-performance adders designed by themselves, and the dependence on DSP is relatively low, so the design can be adapted to a variety of FPGA platforms.

[0037] 4) In edge computing scenarios, due to performance and resource limitations, most of the deployed models are lightweight models, and their model parameters are quantized, converting floating-point type parameters into integer parameters, thereby avoiding problems such as high resource consumption and high computational complexity caused by floating-point operations.

[0038] 5) Support integer operations, which means that before building the model, the model needs to be integer quantized, usually using the INT8 quantization type.

[0039] 6) Supports cross-clock domain transmission and is compatible with RISC-V processors with various upper clock frequency limits.

[0040] 7) The technical solution of the present invention has high portability and can greatly improve the computing efficiency of the processor when facing neural network tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a schematic diagram of the system architecture of an embodiment of the present application;

[0042] Figure 2 This is a schematic diagram of the extended instruction signal flow of an embodiment of the present application;

[0043] Figure 3 This is the operation flow of the embodiment of this application;

[0044] Figure 4 This is a schematic diagram of the architecture of the cache module according to the embodiment of the present application;

[0045] Figure 5 This is a schematic diagram of the output of the row cache module in the embodiment of the present application;

[0046] Figure 6 This is a schematic diagram of the convolution module architecture of an embodiment of the present application;

[0047] Figure 7 This is a schematic diagram of the computing unit architecture of an embodiment of the present application;

[0048] Figure 8 This is a schematic diagram of the data temporary storage module architecture of an embodiment of the present application;

[0049] Fig. 9 This is a schematic diagram of the quantization module architecture of an embodiment of the present application;

[0050] Fig.10 This is a schematic diagram of the activation function calculation module architecture of an embodiment of the present application;

[0051] Fig.11 This is a schematic diagram of the pooling module architecture of an embodiment of the present application;

[0052] Fig.12A schematic diagram of a branch calculation of the yolov3-tiny model network in the embodiment of the present application;

[0053] Fig.13 In the embodiment of this application Fig.12 Schematic diagram of branch calculation operation time distribution shown. DETAILED DESCRIPTION

[0054] The technical solution of the present application is further described in detail below in conjunction with specific embodiments, but the protection scope of the present application is not limited to the following.

[0055] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0056] In the description of the present application, it should be understood that if the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. appear, the orientation or position relationship indicated by these terms is based on the orientation or position relationship shown in the accompanying drawings, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.

[0057] In addition, if the terms "first" or "second" appear, these terms are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, if the term "plurality" appears, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0058] In this application, unless otherwise clearly specified and limited, if the terms "installed", "connected", "connected", "fixed" and the like appear, these terms should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integrated connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to the specific circumstances.

[0059] In the present application, unless otherwise clearly specified and limited, if there is a description that a first feature is "above" or "below" a second feature, etc., or similar descriptions appear, it may mean that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediate medium. Moreover, the first feature being "above", "above" and "above" the second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. The first feature being "below", "below" and "below" the second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0060] It should be noted that if an element is referred to as being "fixed to" or "disposed on" another element, it may be directly on the other element or there may be a central element. If an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. If any, the terms "vertical", "horizontal", "upper", "lower", "left", "right" and similar expressions used in this application are for illustrative purposes only and do not represent the only implementation method.

[0061] like Figure 1 As shown, a coprocessor based on RISC-V architecture includes a decoding unit, a control unit, an operation unit, a cross-clock transmission unit, and a cache unit.

[0062] The decoding unit receives the value of the source register sent by the main processor and decodes it to output the operation parameters. The control unit controls the working process of the operation unit based on the operation parameters. The cross-clock transmission unit communicates with the main processor, including the transmission of configuration parameters and feedback signals, thereby supporting the main coprocessor to run in different clock domains; the cache unit is used to cache part of the weight data and all bias data required in a single operation.

[0063] Among them, the decoding unit decodes the value of the source register sent by the main processor, reads the operand of the source register and splits it, determines the specific operation referred to by the current instruction, obtains the parameters required for the operation and sends them to the subsequent control unit.

[0064] Among them, the control unit is composed of a memory management module and an operation management module. The operation management module dispatches computing tasks to the operation unit according to the parameters given by the decoding module and controls the working process of the operation unit by receiving feedback signals from the operation unit. The memory management module is responsible for the management of data transmission between the coprocessor and the external storage unit, including the calculation and allocation of memory addresses and the jump of memory read and write status.

[0065] Among them, the cross-clock domain transmission unit is responsible for the communication between the main processor and the coprocessor, including the transmission of configuration parameters and feedback signals, thereby supporting the main coprocessor to run in different clock domains, so that when the clock frequency of the main processor is not high, the coprocessor can still work at a relatively high clock frequency, thereby ensuring its high performance characteristics.

[0066] Among them, the cache unit is responsible for caching part of the weight data and all the bias data required in a single operation. The unit is composed of a weight data cache module and a bias data cache module. When performing convolution operations and quantization operations, the weight bias data cache unit effectively outputs the corresponding weight data and bias data in a certain reading order. The function of this unit is to reduce the frequency of access to the external storage unit and reduce the impact of memory access on the overall computing efficiency.

[0067] As a specific embodiment, this example mounts the coprocessor on the hummingbird e203 main processor to form a processing system. The hummingbird e203 main processor includes an instruction fetch module, a decoding module, an execution module, and a nice module. The coprocessor part interface is connected to the extended instruction processing module in the hummingbird e203 main processor, that is, Figure 1 The nice modules above are connected to realize the communication between the main and coprocessors, so that the computing tasks indicated by the extended instructions can be executed. In this example, the coprocessor uses the external memory onboard DDR as the memory medium of the coprocessor to access the processing and waiting data.

[0068] At the beginning, the instruction fetch module of the Hummingbird e203 main processor reads the instruction to be processed and sends it to the decoding module. After receiving the instruction, the decoding module determines the type of the current instruction and the specific operation. After completing the decoding operation, the execution module accesses memory and operates according to the instruction type and the required operands. If the current instruction is not an extended instruction, the execution module will perform corresponding operations based on the given information and data.

[0069] like Figure 2As shown, if it is an extended instruction, the nice module will be enabled for processing, and the data of the register required to be read by the current instruction will be transferred to the nice module, waiting for the coprocessor to complete the operation indicated by the current extended instruction. Here, the extended instruction execution process is a blocking process, rather than a non-blocking process like a pipeline architecture. Only after the current instruction is executed can the nice module be released to receive the next extended instruction.

[0070] In order to improve transmission stability and meet timing requirements, the nice module will encode the operations referred to by all extended instructions. Each operation will have a special ID identifier. When receiving an instruction, the nice module will generate a 32-bit ID identification data according to the operation referred to by the instruction. The ID identification data will be transmitted to the coprocessor together with the data of the given register. The instruction operations adapted to the coprocessor are divided into three types: configuration, reset, and startup. The order of use is generally reset first, then configuration, and finally startup.

[0071] The data transmitted to the coprocessor is first transferred across the clock domain by the cross-clock transmission unit, and then input into the decoding unit of the coprocessor after the processing is completed. The decoding unit decodes and determines the current operation type according to the given ID. If it is a configuration type, the operands transmitted together are split and partially calculated, and then transmitted to the control unit and the calculation unit, such as the starting address of the read and write data, the selection of convolution and pooling types, etc. If it is the other two types, there is no need to configure the operands and it is executed directly.

[0072] The control unit performs static parameter configuration after receiving data from the decoding unit. The enabling of the operator and the form of operation are determined by the parameter configuration.

[0073] like Figure 3 The figure shows the general process of the operation. When the operation is started, the control unit allocates and manages the operation unit and reads and writes the memory in real time according to the state of the operation. Among them, the operation unit and the cache unit are responsible for task operation and weight bias parameter cache respectively, and both are managed and monitored by the control unit.

[0074] The control unit first determines whether the activation function used for this execution is the LeakyRelu function according to the static parameter configuration. If the activation function is LeakyRelu, the quantized static LeakyRelu lookup table data is first loaded into the LeakyRelu cache module in the activation function calculation module in the operation unit, and then all the bias data and part of the weight data required for the current operation are placed in the bias cache module and weight cache module in the cache unit. After the above behavior is completed, the control unit reads the weight data from the cache unit and transmits it to the operation unit. The convolution module in the operation unit will output a loading completion flag signal after the weight loading is completed. After receiving the signal, the control unit reads the feature input graph data in the memory to the operation unit. If the current configuration convolution type is 3×3 convolution, the input data enters the row cache module, and the serial data is processed by the row cache module to output parallel sliding window data. The row cache module supports padding and step selection functions, and the selection of all its functions can be determined by the parameter configuration. The convolution module can operate on 4 input channels of 4 output channels at a time. If the number of input channels is greater than 4, the control unit will enable the data temporary storage module to temporarily store part of the convolution operation results. When all input channels have completed the operation, the control unit reads the data in the bias cache module in the cache unit, adds it to all the data in the temporary storage module, and then outputs it to the quantization module. The quantization module performs integer quantization processing on the data after the convolution operation according to the quantization parameters given by the configuration, and outputs it to the activation function calculation module after processing. If the configuration is selected as the LeakyRelu function calculation, the input quantized data will look up the table value of the LeakyRelu cache module in the cache unit according to its own value, that is, use its own value as the cache address for addressing, and take out the value stored in the corresponding address. If it is a Relu function calculation, the quantized data will be compared with the zero-point data. If it is less than or equal to the zero-point value, the zero-point value will be output, and if it is greater than the zero-point value, the original data will be output. After being processed by the activation function calculation module, if the current calculation requires pooling operation, the data after activation function processing will enter the pooling module. The specific operation of pooling is also determined by the parameter configuration. The pooling operation of the present invention only supports 2×2 maximum pooling, supports adjustable step size and zeropad function. After the data is input, the row cache will also be processed, and then a parallel type 2×2 sliding window will be output. The sliding window data is compared and then the pooled data is output.

[0075] When all operations are completed, the control unit outputs an end mark signal, which is processed by the cross-clock transmission unit and sent to the main processor, releasing the nice module to process and execute the next extended instruction.

[0076] The following is an explanation of the working principle and design architecture of the internal modules of the computing unit.

[0077] The computing unit includes a row cache module, a convolution operation module, a pooling module, an activation function calculation module, a quantization module, and a data storage module. Each module is completely decoupled and has a dedicated enable signal port and parameter input port. The corresponding instructions can be configured as needed. The entire computing unit is built on the basis of a high-performance multiplier-accumulator consisting of a booth-coded multiplier and a Wallace number-structured adder. The unit adopts a pipeline architecture, and the calculation is performed in a non-blocking manner, which greatly improves the overall data throughput.

[0078] like Figure 4 As shown in the figure, the row cache module is mainly composed of two FIFOs and a sliding window output control module. The row cache module converts serial data input into parallel multi-row data output, and outputs a 3×3 sliding window in parallel. At the same time, the module supports the filling function, which supports the filling of up to two rows and two columns. The filling value can be configured by parameters. The two FIFOs are cache sequence 1 and cache sequence 2 shown in the figure. In the running state, the input data will be input into sequence 2 as sequence 2 input data, and the output of sequence 2 will also be input into sequence 1 as sequence 1 input data. When the third row is input, sequence 1, sequence 2 and input data form parallel output. Generally, when the filling function is not used, the initial formation and output of the 3×3 sliding window requires 3 clock cycles. Because it is a pipeline design, subsequent sliding windows will be generated at the rising edge of each clock cycle. If the filling function is enabled in the configuration, take zero as the filling value for example. The input feature map size is nxn, such as Figure 5 As shown in the figure, when the second row is input, sequence 2 starts to read out data and input data to form two rows of parallel output. The top row and the left column of the sliding window will be filled. When the input reaches the end of the feature map, the right column of the sliding window will be filled. Similarly, the bottom output is similar to the top output, and the final output is also as follows. Figure 5 as shown in .

[0079] like Figure 6In the convolution module, the convolution module is mainly composed of 4 output channel operation units, which are specifically divided into 1×1 convolution module and 3×3 convolution module. In a single operation process, each module can operate on four output channels at the same time. In the output channel operation process, the four input channels are also calculated at the same time. In the 3×3 convolution module, the row cache is used for parallel calculation of multiplication and accumulation, and the number of parallel executions of multiplication is 144. Similarly, in the 1×1 convolution module, the number of parallel executions of multiplication is 16. Each output channel operation unit is composed of 4 sliding window calculation units, an adder and a saturation truncation. A single output channel operation unit can perform convolution operations on the feature maps of 4 input channels at the same time. The entire convolution module is also designed based on the pipeline structure. The sliding window calculation unit is also composed of 1×1 sliding window calculation unit and 3×3 sliding window calculation unit. Among them, the 1×1 sliding window calculation unit has a relatively simple structure and is composed of only one booth code multiplier, while the architecture of the 3×3 sliding window calculation unit is relatively complex. Its structure is as follows Figure 7 As shown, the sliding window calculation unit is mainly composed of a multiplier array and a multi-layer adder. The multiplier array is composed of 9 booth coded multipliers, which perform parallel multiplication operations of all data in the sliding window and their respective weight data. The multi-layer adder is composed of two 4-2Wallace tree adders, a 5-2Wallace tree adder and an ordinary adder. For the 9 products obtained from the multiplier array, the multi-layer adder selects a 4-2Wallace tree adder and a 5-2Wallace tree adder as the first layer. The second layer uses a 4-2Wallace tree adder to add two partial sum data and two carry data. The third layer uses an ordinary adder to perform addition calculations on the partial sum and carry data generated by the second layer, and outputs the multiplication and accumulation results of the sliding window calculation unit. After the calculation of the four input channels is completed, the four multiplication and accumulation results will also enter a 4-2Wallace tree adder for addition, channel superposition and then output. After a series of multiplication and accumulation calculations, the bit width of the output result of each channel will exceed 8 bits, and theoretically should be converted to 32 bits. However, 32-bit wide data will greatly increase the consumption of hardware resources in subsequent processing. A large number of experimental observations have found that the quantized model is often much smaller than the maximum value that INT32 can represent after INT8 multiplication and accumulation. In fact, the multiplication of INT8 and INT8 is accumulated to INT16, and only INT18 is needed to ensure that most of the multiplication and accumulation values ​​do not overflow. Therefore, in order to save resources, the output result will be processed by the saturation truncation and converted to INT18 type data. At the same time, four 18-bit wide data will be converted into a 72-bit wide data output to the next processing module.

[0080] like Figure 8It is a data temporary storage module. Its main function is to cache the accumulated results of all channels that have not been completed, and to superimpose the existing results with the input data and then store them. When all channels have completed the operation, the memory data is read out and added with the bias data. The data temporary storage module is mainly responsible for caching and processing the data after the convolution operation. When the number of input channels is greater than 4, the coprocessor cannot complete the calculation of all input channels at one time, so the module is required to cache the currently completed input channel calculation results. When the subsequent input channel calculation results are input to the cache module, the module reads out the cached input channel calculation results and superimposes the subsequent input channel calculation results and then caches them. When all input channels have completed the operation, the memory data will be read out and added with the bias data.

[0081] The data temporary storage module consists of a FIFO and an adder, which is responsible for caching some of the completed calculated results in the output channel. A single operation can only complete the calculation task of the feature map of 4 input channels. When the input channel is greater than 4, a single operation cannot complete all the calculation tasks of an output channel for a single output channel. The function of this module is to temporarily store the completed convolution calculation results, and read the data in the FIFO and add it to the input data when there is data input. If the calculation of a single output channel is completed, the data is sent to the bias addition unit for addition with the bias data, and then output to the next operation unit. If all the calculation tasks of a single output channel are not completed, that is, there are input channels that have not been calculated, at this time, the data will re-enter the FIFO and wait for the next reading.

[0082] like Fig. 9As shown, similar to the convolution module, the quantization module consists of 4 quantization operation units, which perform quantization calculations on the super eight-bit width result after convolution and the relevant parameters of the input, and convert the output result back to an eight-bit integer. The quantization operation unit has two saturation truncators, a quantization multiplier and a shifter. The quantization operation unit is a hardware implementation of Pytorch's quantization reasoning, and its quantized results are basically guaranteed to be consistent with the results after Pytorch quantization processing. The quantization factor used in the hardware quantization process is consistent with the model when the software is generated, but in the software, the quantization factor is a floating point number and cannot be calculated on the coprocessor. For this reason, the floating-point type quantization factor is split into two coefficients, namely, an integer M coefficient and an integer S coefficient. The above operation needs to be performed in the software-generated model parameters. The hardware quantization process converts the floating-point multiplication operation of the quantization factor into an integer multiplication with the M coefficient and then shifts according to the S coefficient, and performs substitution in this way. After completing the convolution operation and the addition operation of the bias data, the bit width of the input data is 72 bits, where, starting from the low bit upwards, every 18 bits of data represents the operation result of an output channel, and one input data contains the operation results of 4 output channels. The 4 quantization operation units are responsible for the operation results of the 4 output channels respectively. The operation process can be summarized as the INT18 type data is transformed back to the INT8 type data after scaling transformation. The specific operation process is as follows: First, the 18-bit unquantized data is saturated and truncated, from INT18 to INT16, and then multiplied with the M coefficient of the UINT16 type in the quantization factor to obtain an INT32 type of data. Then, the INT32 data is shifted according to the shift factor in the quantization factor to obtain an INT16 type of data. Finally, the INT16 data is saturated and truncated from INT16 to INT8 to obtain an INT8 type of output result.

[0083] like Fig.10 The figure shows the activation function calculation module, which supports Relu function calculation and LeakyRelu function calculation. The LeakyRelu function is implemented in the form of a lookup table. The data in the lookup table is the data after all possible data that may appear during the model training process is quantized. Because of quantization, the Relu function calculation also requires zero-point data. The input data is compared with the zero-point data and then the corresponding result is output. The selection of the activation function can be achieved by configuring parameters.

[0084] Exemplarily, the activation function calculation module is composed of 4 Relu function calculation units and 4 LeakyRelu function calculation units and a function selector, wherein the function selector is controlled by a selection signal and is responsible for selecting the activation function type, and the selection signal comes from the decoding unit of the coprocessor. The Relu function calculation unit is essentially a comparator. When the value of the input data is less than the output zero point, the value of the output data is the output zero point value. If it is greater than the output zero point, the value of the output data is equal to the value of the input data. The LeakyRelu function calculation process is a table lookup process, which uses the value of the input data as the address, takes out the data of the memory unit corresponding to the address, and then outputs it as output data. The architectures of the two are relatively simple.

[0085] like Fig.11 The figure shows a pooling module, which consists of 4 pooling calculation units with a size distribution of 2×2. The type is the maximum pooling type and supports configurable step size and feature map edge filling functions. There are row cache units and related control units specially suitable for pooling calculations inside the module, which convert the input serial data into two rows of parallel data. The control unit is responsible for sampling the input data according to the specific parameter configuration. The sampled data will be sent to the sliding window processing module to output 2×2 sliding window data.

[0086] Among them, the pooling calculation unit is composed of a pooling row cache module, a pooling sliding window module and a pooling comparator. The pooling row cache unit is composed of FIFO, which is similar to the row cache module in the convolution. However, because the pooling size only supports 2×2, only one FIFO is used in this unit. The specific execution process is: the input data of the first row enters the FIFO first. When the second row of data is input, the FIFO will output the input data of the first row, and form a parallel output with the second row of input data at this time. At the same time, the input data of the second row will also be input into the FIFO. The pooling sliding window unit is composed of a controller and a 2×2 output window. The controller controls the data to enter the 2×2 output window according to the step size and padding related parameter configuration. For example, when the step size is set to 2, the controller will determine whether the parallel input data meets the step size requirement based on the row number counter to control the input enable. If it is an odd number, the input of data is allowed. If it is an even number, the input enable is turned off. At the same time, the input data will be sampled every other cycle and sent to the output window. When the step size is 1, the input is always enabled and the data is sampled and sent to the output window every cycle. This module supports padding, but it is limited to the step size of 1 and right bottom padding. Its implementation process is similar to the padding of convolution. When the last column of data is at the leftmost of the output window, the right side of the sliding window will be supplemented with zero data to achieve the effect of right edge padding. When the last row of data completely enters the FIFO, the pooling sliding window unit will issue a read enable, so that the FIFO outputs the last row of data. The data will appear at the top of the sliding window, and the bottom of the window will be filled with zero data. At this point, the 2×2 output window completes the right bottom padding. The comparison unit consists of 4 comparison channels, which are also designed for pipelines. One comparison channel is responsible for comparing one 2×2 window data. The specific implementation process is as follows: First, the 32-bit wide sliding window data is split into 4 channel input data, and the input data bit width of each channel is 8 bits. Then, in the first clock cycle, the comparison channel completes the pairwise comparison to obtain two comparison numbers. In the second clock cycle, the two comparison numbers are compared to obtain the comparison result of the 2×2 window data.

[0087] The present invention will be installed on the main processor core of the Hummingbird e203 and deployed on the Gaoyun FPGA platform. The FPGA chip model used is GW5AST-LV138FPG676A. In order to test the computing power of the present invention, the main processor of the Hummingbird e203 equipped with the present invention will be tested for computing speed in a specific convolutional neural network model calculation scenario. The computing speed is expressed by the program running time. In this test, the clock operating frequency of the e203 is 10MHz, and the clock operating frequency of the coprocessor is 100MHz. The following table is a comparison of resource usage before and after the present invention is installed.

[0088]

[0089]

[0090] like Fig.12 As shown in the figure, this scenario is a branch calculation of the yolov3-tiny model network, which is divided into 10 layers. The specific parameter type and output feature map size of each layer can be obtained in the figure. The specific running time is as follows Fig.13 As shown, the operation time of the entire branch is 7.347ms.

[0091] The above is only a preferred embodiment of the present application. It should be understood that the present application is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge in the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present application, and shall be within the scope of protection of the claims attached to the present application.

Claims

1. A coprocessor based on RISC-V architecture, characterized in that: The coprocessor includes: A decoding unit receives the value of the source register sent by the main processor and decodes it to output an operation parameter; A control unit and a computing unit, wherein the control unit controls the working process of the computing unit based on the operating parameters; A cross-clock transmission unit communicates with the main processor, including the transmission of configuration parameters and feedback signals, thereby supporting the main coprocessor to run in different clock domains; The cache unit is used to cache part of the weight data and all the bias data required in a single operation.

2. The RISC-V architecture-based coprocessor according to claim 1, characterized in that: The decoding unit decodes the value of the source register sent by the main processor, reads the operand of the source register and splits it, determines the specific operation referred to by the instruction, obtains the operation parameters required for the operation and sends them to the control unit.

3. The RISC-V architecture-based coprocessor according to claim 1, characterized in that: The control unit includes a memory management module and a calculation management module; in, The operation management module dispatches the computing tasks to the operation unit according to the operation parameters given by the decoding unit, and controls the working process of the operation unit by receiving the feedback signal from the operation unit; The memory management module is responsible for the management of data transmission between the coprocessor and the external storage unit, including the calculation and allocation of memory addresses and the jump of memory read and write states.

4. The RISC-V architecture-based coprocessor according to claim 1, characterized in that: The operation unit adopts a pipeline architecture to form a booth coded multiplier and a Wallace number structure adder, and the calculation is performed in a non-blocking form.

5. The RISC-V architecture-based coprocessor according to claim 4, characterized in that: The operation unit includes a row cache module, a convolution operation module, a pooling module, an activation function calculation module, a quantization module and a data temporary storage module. The modules are completely decoupled and have exclusive enable signal ports and parameter input ports.

6. The RISC-V architecture-based coprocessor according to claim 5, characterized in that: The row buffer module is composed of two FIFOs and outputs a 3×3 sliding window in parallel, which is used to convert serial data input into parallel multi-row data output.

7. The RISC-V architecture-based coprocessor according to claim 5, characterized in that: The convolution operation module includes two parts: a 1×1 convolution module and a 3×3 convolution module; in a single operation process, each module can simultaneously operate on four output channels, and in the output channel operation process, the four input channels are also calculated simultaneously.

8. The RISC-V architecture-based coprocessor according to claim 5, characterized in that: The data temporary storage module is used to cache the accumulated results of all channels that have not been completed, and to superimpose the existing results with the cached data and then store them. When all channels have completed the operation, the memory data is read out and added with the bias data.

9. The RISC-V architecture-based coprocessor according to claim 5, characterized in that: The quantization module is used to perform quantization calculation on the convolution result with a width of more than eight bits and the relevant input parameters, and convert the output result back into an integer with a width of eight bits.

10. The RISC-V architecture-based coprocessor according to claim 5, characterized in that: The activation function calculation module supports Relu function operation and LeakyRelu function operation, wherein the selection of the activation function can be achieved by configuring parameters.

11. The RISC-V architecture-based coprocessor according to claim 5, characterized in that: The pooling module has a built-in row cache unit suitable for pooling calculation and related control logic, which converts the input serial data into two rows of parallel data. The control logic is responsible for sampling and outputting the input data according to specific parameter configuration.

12. The RISC-V architecture-based coprocessor according to claim 1, characterized in that: The cache unit is composed of a weight data cache module and a bias data cache module, which are used to cache weight data and bias data respectively; When performing convolution operations and quantization operations, the corresponding weight data and bias data are effectively output according to the preset reading order.

13. A processing system for a coprocessor based on a RISC-V architecture, characterized in that: The system includes: Main processor; at least one external memory; And a coprocessor as claimed in any one of claims 1 to 12; The coprocessor responds to the extended instruction of the main processor and completes the computing task based on the extended instruction. The external memory serves as the memory medium of the coprocessor to access the processing and to-be-processed data.

14. The processing system of the coprocessor based on the RISC-V architecture according to claim 13, characterized in that: The main processor includes a nice module for processing extended instructions, and the coprocessor is connected to the nice module to complete communication.

Images