A flexible convolutional network accelerator based on a RISC-V processor

Abstract

The application provides a flexible convolution network accelerator based on a RISC-V processor, the accelerator comprising a TileLink bus module configured to obtain to-be-convoluted data corresponding to an external storage device; a plurality of convolution processing cores configured to perform convolution calculation on the to-be-convoluted data, the convolution processing cores being constructed based on a deep separable convolution calculation hardware architecture, and the convolution processing cores comprising a reconfigurable processing unit and a MAT unit; and a RISC-V processor configured to control the TileLink bus module to obtain the to-be-convoluted data corresponding to the device and control the convolution processing cores to perform corresponding convolution calculation on the to-be-convoluted data. The application solves the problems of low data reuse rate, memory bandwidth bottleneck and low resource utilization rate existing in the existing convolution hardware technology through the above accelerator.

Description

Technical Field

[0001] This invention relates to the field of neural network technology, and in particular to a flexible convolutional network accelerator based on a RISC-V processor. Background Technology

[0002] Lightweight convolutional neural networks (CNNs) are a class of CNNs specifically designed to reduce computational complexity and memory footprint. These networks significantly reduce computational resource requirements while maintaining high accuracy. For example, MobileNet uses depthwise separable convolution, decomposing standard convolution into depthwise convolution and pointwise convolution, greatly reducing the number of parameters and computational cost.

[0003] However, traditional hardware architectures suffer from low data reuse rates, memory bandwidth bottlenecks, and low processing unit utilization when performing such operations, resulting in overall performance not being fully realized. Specific problems are as follows: Low data reuse rate: Depthwise separable convolution decomposes standard convolution into depthwise convolution and pointwise convolution. Although theoretically the amount of computation is reduced, in actual hardware implementation, due to the low data reuse rate, frequent data loading and storage operations result in a lot of time being wasted on memory access.

[0004] Memory bandwidth bottleneck: Depthwise separable convolutions involve a large amount of intermediate data, requiring frequent memory access. Existing hardware architectures cannot effectively manage these data streams, making memory bandwidth a bottleneck and limiting overall computational performance.

[0005] Low resource utilization: Due to poor data flow management and unreasonable allocation of hardware resources, the utilization rate of processing units is low, and many computing resources are idle, failing to fully utilize hardware performance. Summary of the Invention

[0006] This application provides a flexible convolutional network accelerator based on a RISC-V processor to address the problems of low data reuse rate, memory bandwidth bottleneck, and low resource utilization in existing convolutional hardware technologies.

[0007] The accelerator includes: The TileLink bus module is configured to acquire the convolutional data from the corresponding external storage device. Several convolutional processing kernels are provided, all of which are connected to the TileLink bus module. The convolutional processing kernels are configured to perform convolution calculations on the data to be convolved. The convolution calculations include ordinary convolution calculations, pointwise convolution calculations, and depthwise convolution calculations. The convolutional processing kernels are built based on a depthwise separable convolutional calculation hardware architecture and include reconfigurable processing units and MAT units. A RISC-V processor is configured to control the TileLink bus module to acquire the convolutional data to be convolved from the corresponding device and to control the convolutional processing core to perform corresponding convolutional calculations on the convolutional data to be convolved.

[0008] Preferably, the reconfigurable processing unit includes a plurality of PE columns, wherein the PE columns include a plurality of parallel multiply-accumulate units and a plurality of multiplexers; The multiply-accumulate unit includes a multiplier, an adder, and a data register connected in sequence. The data input of the multiplexer is connected to the adder in the previous row of the multiply-accumulate unit and the data register in the next row of the multiply-accumulate unit, respectively. The data output of the multiplexer is connected to the adder in the next row of the multiply-accumulate unit. The multiplier is configured to perform cumulative multiplication on the data to be convolved, the adder is configured to perform cumulative addition on the data to be convolved, and the data register is configured to store and output the calculation result; The bottom row of the PE column's multiply-accumulate unit includes the multiplier, the adder, the data register, and the bottom row multiplexer. The bottom row multiplexer is located between the multiplier and the adder. The data input of the bottom row multiplexer is connected to the adder in the previous row's multiply-accumulate unit and the multiplier in the current row's multiply-accumulate unit, respectively. The data output of the bottom row multiplexer is connected to the adder in the current row's multiply-accumulate unit. After receiving the data transmitted by the adder in the current row, the data register in the bottom row multiply-accumulate unit sends the data back to the adder in the current row. After completing processing, the multiplier in the bottom row multiply-accumulate unit sends the processed data to the bottom row multiplexer and the multiplexer in the top row multiply-accumulate unit, respectively.

[0009] Preferably, the reconfigurable processing unit is configured to perform the ordinary convolution calculation, the pointwise convolution calculation, and / or the depthwise convolution calculation on the data to be convolved.

[0010] Preferably, the reconfigurable processing unit is further configured to: Perform convolution calculations in DW mode or PW mode, wherein DW mode supports depthwise convolution calculations and PW mode supports both normal convolution calculations and pointwise convolution calculations. When performing convolution calculations in DW mode, the data port in the multiplexer used to transmit data between the upper and lower rows is closed; When performing convolution calculations in PW mode, the data port in the path selector used to transfer data between the adder and the data register in the same row is closed.

[0011] Preferably, the MAT unit is configured to perform ordinary convolution calculations and / or pointwise convolution calculations on the data to be convolved, and the MAT unit and the reconfigurable processing unit perform convolution calculations simultaneously or separately.

[0012] Preferably, the convolutional processing kernel further includes a data cache unit A, a data cache unit B, a data cache unit C, and an auxiliary cache unit. All the convolutional processing kernels share the data cache unit A and the data cache unit C. Each convolutional processing kernel includes a corresponding data cache unit B and the auxiliary cache unit. The auxiliary cache unit is configured to cache the calculation results of the reconfigurable processing unit and provide input data for the MAT unit and the reconfigurable processing unit; When performing the depthwise convolution calculation, the data cache unit A is configured to store the input feature map data of the data to be convolved, and the data cache unit B is configured to store the convolution kernel weight data of the data to be convolved. During the pointwise convolution calculation, the data cache unit C is configured to store the convolution kernel weight data of the data to be convolved, and the data cache unit B is configured to store the input feature map data of the data to be convolved. The convolution kernel is also configured to perform convolution calculations on the data to be convolved using a computation data stream, the computation data stream including time-division multiplexing computation and pipelined computation.

[0013] Preferably, the time-division multiplexing calculation includes: When performing the pointwise convolution calculation or the ordinary convolution calculation, the convolution calculation is performed jointly by the reconfigurable processing unit and the MAT unit. When performing the depthwise convolution calculation, the convolution calculation is performed by the reconfigurable processing unit; When performing the depthwise convolution calculation, the pointwise convolution calculation, and the ordinary convolution calculation, the MAT unit performs them first, and after the reconfigurable processing unit completes the depthwise convolution calculation, the MAT unit and the reconfigurable processing unit jointly perform the pointwise convolution calculation and the ordinary convolution calculation.

[0014] Preferably, the pipelined computation includes, when the depthwise convolution computation is performed on the reconfigurable processing unit, caching the computation result of the reconfigurable processing unit as input data for the pointwise convolution computation in the auxiliary cache unit, and transmitting the input data cached in the auxiliary cache unit to the MAT unit.

[0015] Preferably, the convolution kernel further includes: An intermediate processing unit is disposed between the reconfigurable processing unit and the auxiliary cache unit, and the intermediate processing unit is configured to shift, truncate and rearrange the calculation results of the reconfigurable processing unit. A post-processing unit is disposed after the MAT unit, and the post-processing unit is configured to perform shifting, truncation and activation processing on the calculation results of the MAT unit. An output cache unit is configured to cache the final calculation result of the convolution calculation on the data to be convolved.

[0016] Preferably, the accelerator further includes: The BootRom module is configured to store a firmware bootloader that powers on the convolutional processing kernel; An interrupt controller is configured to handle interrupt requests from external devices or software programs to the convolutional processing kernel. A DDR controller, configured as an interface for connecting to the memory of an external storage device; PCIe controller, the PCIe controller being configured as a redundant interface for connecting external device memory; The AXI module is configured to arbitrate and route the data to be convolved and the final calculation result, respectively.

[0017] As described above, this application provides a flexible convolutional network accelerator based on a RISC-V processor. The accelerator includes a TileLink bus module configured to acquire convolutional data from a corresponding external storage device; several convolutional processing cores, all connected to the TileLink bus module, configured to perform convolution calculations on the convolutional data. These convolution calculations include ordinary convolution, pointwise convolution, and depthwise convolution. The convolutional processing cores are built on a depthwise separable convolutional computing hardware architecture and include reconfigurable processing units; and a RISC-V processor configured to control the TileLink bus module to acquire the convolutional data from the corresponding device and to control the convolutional processing cores to perform corresponding convolution calculations on the convolutional data. This application solves the problems of low data reuse rate, memory bandwidth bottleneck, and low resource utilization in existing convolutional hardware technologies through this accelerator. Attached Figure Description

[0018] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 The flowchart shows the logic of applying convolution operations individually to each input channel and pointwise convolution for depthwise convolution.

[0020] Figure 2 This is a schematic diagram of the basic structure of MAT; Figure 3 This is a schematic diagram of a flexible convolutional network accelerator based on a RISC-V processor according to this application; Figure 4 This is a schematic diagram of the reconfigurable processing unit of this application; Figure 5 This is a schematic diagram of the reconfigurable processing unit of this application executing DW mode; Figure 6 A schematic diagram illustrating the loading rules for input and weight data in DW mode; Figure 7 This is a schematic diagram of the reconfigurable processing unit of this application executing PW mode; Figure 8 A schematic diagram illustrating the loading rules for input and weight data in PW mode; Figure 9 This is a schematic diagram of the convolution processing kernel in this application; Figure 10 This is a schematic diagram illustrating the computational data flow of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] In the field of modern deep learning, Convolutional Neural Networks (CNNs) are a core technology widely used in tasks such as image recognition and object detection. While standard convolution operations are highly efficient in feature extraction, they are computationally intensive and have a large number of parameters, especially when processing high-resolution images and deep networks, demanding significant computational and storage resources. To improve computational efficiency and reduce model size, the MobileNet model introduced the innovative technique of Depthwise Separable Convolution. Depthwise Separable Convolution is an efficient convolution operation that breaks down standard convolution into two smaller operations: Depthwise Convolution and Pointwise Convolution.

[0023] 1. Depthwise Convolution (DWConv) Depthwise convolution is a convolution operation applied individually to each input channel. For example... Figure 1 As shown on the left, at the start of the computation, it is assumed that the input feature map has The convolution kernel has K×K channels and a depthwise convolution kernel (with K×K channels). It performs convolution operations independently on each channel, thus producing The intermediate feature map of each channel. Compared with standard convolution, depthwise convolution significantly reduces computation because it does not perform convolution across channels.

[0024] 2. Pointwise Convolution (PWConv) Convolution one point by point The convolution operation is used to recombine the feature maps generated by depthwise convolution. For example... Figure 1 As shown on the right, it passes through Convolutional kernels linearly combine all channels, thereby changing the number of channels in the output feature map. Pointwise convolution has relatively low computational cost, but it can flexibly adjust the number of output channels, thus increasing the expressive power of the model.

[0025] By decomposing standard convolution into depthwise convolution and pointwise convolution, depthwise separable convolution can significantly reduce computational cost and parameter count. Specifically, if the computational cost of a standard convolution operation is... ,in and These are the height and width of the input feature map, respectively. and It is the number of input and output channels. If that's the kernel size, then the computational cost of depthwise separable convolution is... + This significantly reduces computational complexity. Neural networks using depthwise separable convolutional layers, such as the MobileNet model, can fully leverage the advantages of depthwise separable convolutions, significantly reducing computational complexity and model size while maintaining high accuracy. This makes them ideal for applications in environments with limited computing resources, such as mobile devices.

[0026] Convolution computations in modern neural networks can fundamentally be decomposed into a series of multiplication and addition operations. Therefore, parallel hardware architectures based on multiplier and adder-trees (MATs) provide a direct and efficient solution for both ordinary convolution (Conv) and pointwise convolution (essentially ordinary convolution with a 1×1 kernel). The basic structure of MAT is as follows: Figure 2 As shown, data from the input buffer and weight buffer can be broadcast to all processing elements (PEs) of the MAT via the data loader to leverage the data reusability of convolution computation. Within each MAT, multiple multipliers are responsible for parallel computation along the input channel dimension to generate partial sums, which are then added together via an addition tree module, and finally accumulated by an accumulator to obtain the final output. A FIFO (First Input First Output) partial sum queue is used to buffer intermediate data during the computation process.

[0027] However, the MAT-based hardware architecture has a fixed structure and data flow, which cannot effectively support depthwise convolution computation in depthwise separable convolution. Because depthwise convolution computation calculates data within each channel separately, it can only sum and accumulate the product of the input feature map and the convolution kernel within the same convolution window of the same channel. If the MAT structure is used for computation, on the one hand, considering the relationship between the computational parallelism of the PE columns within MAT and the convolution kernel size of the depthwise convolution, the parallelism of the PE columns is set to a fixed convolution kernel size (e.g., ...). This will limit MAT's flexibility in supporting various kernel sizes and its large-scale scalability. On the other hand, depthwise convolutions with different kernel sizes and strides will produce different data overlap patterns between adjacent sliding convolution windows during computation, resulting in additional linear buffer overhead or complex memory management requirements.

[0028] Based on the above problems, this application provides the following implementation method.

[0029] Figure 3 This is a schematic diagram of a flexible convolutional network accelerator based on a RISC-V processor according to this application.

[0030] Figure 9 This is a schematic diagram of the convolution processing kernel in this application.

[0031] See Figure 3 and Figure 9 As can be seen, this embodiment provides a flexible convolutional network accelerator based on a RISC-V processor, the accelerator comprising: TileLink bus module 100 is configured to acquire convolutional data from a corresponding external storage device. Specifically, in this embodiment, TileLink bus module 100 is used to establish a communication connection with the external storage device and acquire the convolutional data from the external storage device.

[0032] The accelerator also includes: Several convolutional processing cores 200 are provided, all of which are connected to the TileLink bus module 100. Each convolutional processing core 200 is configured to perform convolution calculations on the data to be convolved. These convolution calculations include ordinary convolution, pointwise convolution, and depthwise convolution. The convolutional processing cores 200 are built on a depthwise separable convolutional computing hardware architecture. Each convolutional processing core 200 includes a reconfigurable processing unit 210 and a MAT unit 220. Specifically, in this embodiment, the convolutional processing core 200 is the core component for convolution in the accelerator. Through the convolutional processing core 200, ordinary convolution, pointwise convolution, and depthwise convolution calculations on the data to be convolved can be completed. These calculations are jointly performed by the reconfigurable processing unit 210 and the MAT unit 220, thereby improving convolution efficiency.

[0033] Figure 4 This is a schematic diagram of the reconfigurable processing unit of this application.

[0034] See Figure 4Furthermore, in some embodiments, the reconfigurable processing unit 210 includes several PE columns, each PE column including several parallel multiply-accumulate units and several multiplexers; each multiply-accumulate unit includes a multiplier, an adder, and a data register connected in sequence; the data input of the multiplexer is connected to the adder in the previous row of the multiply-accumulate unit and the data register in the next row of the multiply-accumulate unit, respectively; the data output of the multiplexer is connected to the adder in the next row of the multiply-accumulate unit; the multiplier is configured to perform cumulative multiplication on the data to be convolved, the adder is configured to perform cumulative addition on the data to be convolved, and the data register is configured to store and output the computation. As a result, the bottom row of the PE column multiply-accumulate unit includes the multiplier, the adder, the data register, and the bottom row multiplexer. The bottom row multiplexer is located between the multiplier and the adder. The data input of the bottom row multiplexer is connected to the adder in the previous row multiply-accumulate unit and the adder in the current row multiply-accumulate unit of the multiplexer. After receiving the data transmitted by the adder in the current row, the data register in the bottom row multiply-accumulate unit sends the data back to the adder in the current row. After completing the processing, the multiplier in the bottom row multiply-accumulate unit sends the processed data to the bottom row multiplexer and the multiplexer in the top row multiply-accumulate unit, respectively.

[0035] Based on the above embodiments, it can be seen that the reconfigurable processing unit 210 is configured to perform the ordinary convolution calculation, the pointwise convolution calculation, and / or the depthwise convolution calculation on the data to be convolved.

[0036] The hardware architecture of the reconfigurable processing element (RPE) 210 provided in this embodiment can effectively support various types of convolutional operations in lightweight networks such as MobileNet. Figure 4 As shown, the P×Q configured RPE contains P PE columns, and each PE column has Q parallel multiplication-accumulation (MAC) units. Each MAC unit consists of a multiplier, an adder, and a data register. The RPE can be configured to run in DW mode and PW mode to support depthwise convolution and pointwise convolution / normal convolution computations, respectively.

[0037] It should be noted that the data port of the multiplexer can be turned on or off depending on different needs.

[0038] Figure 5This is a schematic diagram of the reconfigurable processing unit of this application executing DW mode.

[0039] Figure 7 This is a schematic diagram of the reconfigurable processing unit of this application executing PW mode.

[0040] See Figure 5 and Figure 7 It is understood that, furthermore, in some embodiments, the reconfigurable processing unit 210 is also configured to: Perform convolution calculations in DW mode or PW mode, wherein DW mode supports depthwise convolution calculations and PW mode supports both normal convolution calculations and pointwise convolution calculations. When performing convolution calculations in DW mode, the data port in the multiplexer used to transmit data between the upper and lower rows is closed; When performing convolution calculations in PW mode, the data port in the path selector used to transfer data between the adder and the data register in the same row is closed.

[0041] Specifically, in this embodiment, the DW mode and PW mode are implemented by adjusting the multiplexer.

[0042] When the reconfigurable processing unit 210 (RPE) operates in DW mode, its internal connection method is as follows: Figure 5 As shown, the adder and its adjacent register to the right form a multiply-accumulate unit. The calculation result of the multiplier is input into the accumulator and accumulated as needed. During the calculation process, the product of the input feature value and the weight within the sliding convolution window in each channel is calculated by the multiplier in each row and accumulated within each multiply-accumulate unit, forming a self-accumulating data stream. The figure illustrates how the reconfigurable processing unit, defined as [specific unit name], executes a 3×3 depthwise convolution (DWConv) with a stride of 1. For details, see [reference needed]. Figure 6 .

[0043] The loading rules for input and weight data are as follows: Figure 6 As shown on the left.

[0044] For example, in the first computation cycle, the input data 1 to 4 in the first row of the first channel of the input feature map are loaded and transmitted along the shift register queue to the upper input of the multiplier in the top row of the four PE columns of the RPE, and the weight data 1 is loaded and broadcast to the left input of the multiplier in the four PE columns. In the second calculation cycle, the input data is shifted to the right along the shift register. Input data 1 is removed from the queue, and input data 5 is moved into the queue. At this time, input data 2 to 5 are loaded onto the upper input of the multiplier in the first row of the top four PE columns of RPE. At the same time, weight data 2 is loaded and broadcast to the left input of the multiplier in the four PE columns.

[0045] In the third calculation cycle, following the same rules, the input data is updated from 3 to 6, and the weight data is updated to 3.

[0046] Starting from the 4th cycle, the computation will shift to the next row of data from the input feature map (10 to 15) and the convolution kernel (4 to 6).

[0047] After 3×3=9 calculation cycles, once 3 rows of data have been processed, the first row of the 4 PE columns can be multiplied and accumulated to obtain the final output data 1 to 4 (the first 4 data in the first row of the first channel of the output feature map) through self-accumulation.

[0048] The four rows of multipliers and multiply-accumulate units in each PE column can perform calculations for the four output channels in parallel.

[0049] When calculating a depthwise convolution with a kernel size of k×k and a stride of 1, following the same rules as the 3×3 convolution, in the first calculation cycle, the input data 1 to 4 in the first row of the first channel of the input feature map are loaded. Then, the input data is shifted in the first row for k-1 cycles. After k cycles, the calculation of the first row is completed. After a total of k×k cycles, the k rows of data are processed, and the final output data 1 to 4 can be obtained.

[0050] When computing depthwise convolutions with a stride of 2, the data loading method for RPE needs to be adjusted accordingly. Taking a depthwise convolution with a kernel size of 3×3 and a stride of 2 as an example, the data loading rules are as follows: Figure 6 As shown on the right, it has the following characteristics.

[0051] In the first computation cycle, the input data 1, 3, 5, 7 in the first row of the first channel of the input feature map are loaded and transmitted along the shift register queue to the upper input of the multiplier in the top row of the four PE columns of the RPE. The weight data 1 is loaded and broadcast to the left input of the multiplier in the four PE columns. In the second calculation cycle, the input data is shifted to the right along the shift register. Input data 1 is removed from the queue, and input data 9 is moved into the queue. At this time, input data 3, 5, 7, and 9 are loaded onto the upper input of the multiplier in the first row of the top four PE columns of RPE. At the same time, weight data 3 is loaded and broadcast to the left input of the multiplier in the four PE columns.

[0052] In the third computation cycle, the input data 2, 4, 6, 8 in the first row of the first channel of the input feature map are loaded and transmitted along the shift register queue to the upper input of the multiplier in the top row of the four PE columns of the RPE. At the same time, the weight data 2 is loaded and broadcast to the left input of the multiplier in the four PE columns.

[0053] Starting from the 4th cycle, the computation will move to the next row of the input feature maps (10 to 18) and convolution kernels (4 to 6), loading data in the same order as the previous 3 cycles.

[0054] After 3×3=9 calculation cycles, once 3 rows of data have been processed, the first row of the 4 PE columns can be multiplied and accumulated to obtain the final output data 1 to 4 (the first 4 data in the first row of the first channel of the output feature map) through self-accumulation.

[0055] The four rows of multipliers and multiply-accumulate units in each PE column can perform calculations for the four output channels in parallel.

[0056] When the reconfigurable processing unit 210 (RPE) operates in PW mode, its internal connection is as follows: Figure 7 As shown, it supports pointwise convolution (PWConv) and regular convolution, working in a manner similar to the MAT architecture. Specifically, as... Figure 7 As shown, each multiplier in each PE column calculates the product of the input data and weights along the input channel, and then accumulates the product through the adders in the PE column, forming a downward-accumulating data stream. The accumulated result is further accumulated in the multiply-accumulate unit at the bottom of the PE column until the final result is obtained. Taking a pointwise convolution with a kernel size of 1×1 and a stride of 1 as an example (a pointwise convolution with a stride of 2 is usually not used) on a 4×4 reconfigurable processing unit, the data loading rules are as follows: Figure 8 As shown.

[0057] For example, four parallel 1×1 size convolutional kernels have their weight data grouped into groups of Q=4 along the channel direction. Figure 8 Groups 1 and 2 each contain Q=4 weight data points and belong to the first convolutional kernel. Similarly, groups 3 and 4 belong to the second convolutional kernel, groups 5 and 6 belong to the third convolutional kernel, and groups 7 and 8 belong to the fourth convolutional kernel.

[0058] like Figure 8 As shown, the input feature map data is also grouped along the channel direction in groups of Q=4. In the first computation cycle, the four weights of the first group of the first convolution kernel are loaded onto the upper input of the four multipliers in the first PE column; the four weights of the third group of the second convolution kernel are loaded onto the upper input of the four multipliers in the second PE column, and so on.

[0059] In the first calculation cycle, the data of the first channel of the first group of the input feature map is loaded and broadcast to the left input terminal of the multiplier in the first row of the four PE columns, the data of the second channel is loaded and broadcast to the left input terminal of the multiplier in the second row of the four PE columns, and so on.

[0060] In the second computation cycle, the four weights of the second group of the first convolution kernel are loaded onto the upper input of the four multipliers in the first PE column; the four weights of the fourth group of the second convolution kernel are loaded onto the upper input of the four multipliers in the second PE column, and so on.

[0061] In the second calculation cycle, the data of the first channel of the second group of the input feature map is loaded and broadcast to the left input terminal of the multiplier in the first row of the four PE columns, and the data of the second channel is loaded and broadcast to the left input terminal of the multiplier in the second row of the four PE columns.

[0062] In this way, the computation traverses the input channel dimensions of the feature map and the convolutional kernels until the final computation result is obtained, and then the computation continues along the remaining positions of the input feature map and the dimensions of the other convolutional kernels (output channels).

[0063] Based on the above example, it can be seen that the MAT unit 220 is configured to perform ordinary convolution calculation and / or pointwise convolution calculation on the data to be convolved, and the MAT unit 220 and the reconfigurable processing unit 210 perform convolution calculation simultaneously or separately.

[0064] Figure 10 This is a schematic diagram illustrating the computational data flow of this application.

[0065] See Figure 10 It is understood that, further, in some embodiments, the convolutional processing kernel 200 also includes a data cache unit A230, a data cache unit B240, a data cache unit C250, and an auxiliary cache unit 260. All the convolutional processing kernels 200 share the data cache unit A230 and the data cache unit C250, and each convolutional processing kernel 200 includes the corresponding data cache unit B240 and the auxiliary cache unit 260. The auxiliary cache unit 260 is configured to cache the calculation results of the reconfigurable processing unit 210 and provide input data for the MAT unit 220 and the reconfigurable processing unit 210; During the depthwise convolution calculation, data cache unit A230 is configured to store the input feature map data of the data to be convolved, and data cache unit B240 is configured to store the convolution kernel weight data of the data to be convolved; during the pointwise convolution calculation, data cache unit A230 and data cache unit C250 are configured to store the convolution kernel weight data of the data to be convolved, and data cache unit B240 is configured to store the input feature map data of the data to be convolved. The convolution processing kernel 200 is further configured to perform convolution calculations on the data to be convolved using a computation data stream, the computation data stream including time-division multiplexing computation and pipelined computation.

[0066] Specifically, in this embodiment, considering the efficiency of the MAT-based architecture in performing pointwise convolution (PWConv) and ordinary convolution, and the flexibility of the RPE hardware structure in supporting various convolution operation types, the present invention designs... Figure 9 The architecture combines the advantages of these two hardware architectures. For example... Figure 9 As shown, the hardware architecture mainly includes multiple on-chip data cache units and L parallel convolutional processing cores 200. Each processing core contains a P×Q configured RPE module and an S×T configured MAT module. Specifically, the hardware architecture design proposed in this invention has the following characteristics.

[0067] Data caching units A, B, and C are used to cache various types of data: During the depthwise convolution (DWConv) computation, data caching unit A230 is used to cache the input feature map data, and data caching unit B240 is used to cache the convolution kernel weight data; during the pointwise convolution (PWConv) computation, data caching units A230 and C250 are used to cache the convolution kernel weight data, and data caching unit B240 is used to cache the input feature map data.

[0068] The auxiliary cache unit 260 is used to cache the calculation results of the RPE module and provide input data for the MAT unit 220. It can directly interact with the off-chip storage and transfer data through the TileLink bus module 100.

[0069] The time-division multiplexing calculation includes: When performing the pointwise convolution calculation or the ordinary convolution calculation, the reconfigurable processing unit 210 and the MAT unit 220 jointly perform the convolution calculation. When performing the depthwise convolution calculation, the convolution calculation is performed by the reconfigurable processing unit 210; When performing the depthwise convolution calculation, the pointwise convolution calculation, and the ordinary convolution calculation, the MAT unit 220 performs them first, and after the reconfigurable processing unit 210 completes the depthwise convolution calculation, the MAT unit 220 and the reconfigurable processing unit 210 jointly perform the pointwise convolution calculation and the ordinary convolution calculation.

[0070] The pipelined computation includes, when the depthwise convolution computation is performed on the reconfigurable processing unit 210, caching the computation result of the reconfigurable processing unit 210 as input data for the pointwise convolution computation in the auxiliary cache unit 260, and transmitting the input data cached in the auxiliary cache unit 260 to the MAT unit 220.

[0071] Specifically, in this embodiment, based on a hardware architecture designed with RPE and MAT structures, to further improve hardware utilization and thus enhance the computational efficiency of depthwise separable convolutional networks, this invention proposes a time-division multiplexing and pipelined data flow design. Taking the MBConv block in the classic lightweight network MobileNet as an example, it contains two pointwise convolutional layers and a middle depthwise convolutional layer (pointwise convolution ① + depthwise convolution + pointwise convolution ②). This MBConv block executes according to the time-division multiplexing and pipelined data flow design on the proposed hardware architecture, the principle of which is as follows: Figure 10 As shown.

[0072] Time-sharing reuse planning: The MAT-based architecture is highly efficient in performing pointwise convolutions, and the RPE structure can flexibly support various convolution operations. For Figure 10 Pointwise convolution ① is executed in parallel by the MAT and RPE units; depthwise convolution is executed separately by the RPE module; for... Figure 10 In the pointwise convolution ②, it is first executed by MAT, and after RPE completes the depthwise convolution, it is executed in parallel by MAT and RPE units.

[0073] Pipeline Design: When depthwise convolution is executed on the RPE module, its computation result, i.e., the input to pointwise convolution ②, can be cached in an auxiliary cache unit and then immediately transferred to an idle MAT module as the input for subsequent pointwise convolution ②, so as to fuse depthwise convolution and subsequent pointwise convolution computations on-chip. Since depthwise convolution involves less computation than pointwise convolution, after the RPE module finishes processing depthwise convolution, it can add pointwise convolution ② computation to the MAT module in parallel to improve hardware utilization.

[0074] Furthermore, in some embodiments, the convolutional processing kernel 200 further includes: An intermediate processing unit 270 is disposed between the reconfigurable processing unit 210 and the auxiliary cache unit 260. The intermediate processing unit 270 is configured to shift, truncate and rearrange the calculation results of the reconfigurable processing unit 210. A post-processing unit 280 is disposed after the MAT unit 220, and the post-processing unit 280 is configured to perform shifting, truncation and activation processing on the calculation results of the MAT unit 220. Output cache unit 290 is configured to cache the final calculation result of convolution calculation on the data to be convolved.

[0075] Specifically, in this embodiment, the output cache unit 290 is used to cache the final calculation result of the processing core and transmit the result to off-chip storage through the TileLink bus module 100.

[0076] Data cache unit A230, data cache unit C250 and output cache unit 290 are shared by L convolution processing cores 200, and each processing core has its own data cache unit B240 and auxiliary cache unit 260.

[0077] The intermediate processing unit 270 is used to perform operations such as shifting, truncating and rearranging the calculation results of the reconfigurable processing unit 210, and save the results to the auxiliary cache unit 260; the post-processing module is used to perform operations such as shifting, truncating and activating the calculation results of the MAT unit 220, and save the calculation results to the auxiliary cache unit 260.

[0078] The accelerator also includes: The RISC-V processor 300 is configured to control the TileLink bus module 100 to acquire the convolution data to be convolved from the corresponding device and to control the convolution processing core 200 to perform corresponding convolution calculations on the convolution data. Specifically, in this embodiment, the accelerator provided in this application is built based on the RISC-V processor 300, which plays a role in coordinating and managing other modules in the accelerator.

[0079] Furthermore, in some embodiments, the accelerator further includes: BootRom module 400, which is configured to store a firmware boot program that enables the convolutional processing core 200 to start upon power-up; An interrupt controller 500 is configured to handle interrupt requests from external devices or software programs to the convolution processing core 200. DDR controller 600, the DDR controller 600 being configured as an interface for connecting to the memory of an external storage device; PCIe controller 700, the PCIe controller 700 being configured as a redundant interface for connecting external device memory; AXI module 800 is configured to arbitrate and route the data to be convolved and the final calculation result, respectively.

[0080] Specifically, in this embodiment, the RISC-V instruction set architecture has advantages such as low power consumption, low cost, open source, modularity, simplicity, small size, and high speed. Based on the RISC-V processor architecture, this embodiment designs a RISC-V processor architecture that can be used for flexible convolution calculations. Figure 3 The specific architecture of a flexible convolutional network accelerator for RISC-V processors is demonstrated.

[0081] The accelerator includes a debugging module, an interrupt controller 500, a BootRom module 400, a RISC-V processor 300, a TileLink bus module 200, and L convolution processing cores 200 as proposed in this embodiment.

[0082] The interrupt controller 500 is used to handle interrupt requests from external storage devices or software programs to the CPU.

[0083] The BootRom module 400 is used to store the firmware boot program that is generated when the CPU powers on.

[0084] The accelerator primarily interacts with data via the TileLink bus module 200. TileLink is a chip-level bus interconnect standard that allows multiple masters to access memory and other slaves in a consistent memory-mapped manner. It provides a high-speed, scalable on-chip interconnect with low latency and high throughput to connect general-purpose multiprocessors, coprocessors, accelerators, and various simple or complex devices.

[0085] The DDR controller 600 and PCIe controller 700 are interconnected with the processor's internal bus via the TileLink bus module 200 to the AXI module 800, and their data is arbitrated and routed through the AXI module 800.

[0086] This embodiment has the following advantages: A reconfigurable convolution processing unit and data loading method are designed to flexibly and efficiently support ordinary convolution, depthwise convolution and pointwise convolution computations without the need for expensive additional on-chip cache resources.

[0087] Based on the reconfigurable processing unit structure and the traditional multiplier-adder tree structure, a parallel overall hardware architecture was designed to take full advantage of the advantages of both structures.

[0088] A time-division multiplexing and pipelined data flow is designed to fuse the computation of depthwise convolution and pointwise convolution on-chip, thereby further improving hardware utilization and enhancing the computational efficiency of depthwise separable convolutional networks.

[0089] A flexible convolutional network accelerator architecture based on the RISC-V processor architecture is designed, which has the advantages of low power consumption, low cost, open source, modularity, simplicity, small size and high speed.

Claims

1. A flexible convolutional network accelerator based on a RISC-V processor, characterized in that, The accelerator includes: TileLink bus module (100), the TileLink bus module (100) is configured to acquire the convolutional data to be processed from the corresponding external storage device; A plurality of convolutional processing cores (200) are provided, all of which are connected to the TileLink bus module (100). The convolutional processing cores (200) are configured to perform convolutional calculations on the data to be convolved. The convolutional calculations include ordinary convolutional calculations, pointwise convolutional calculations, and depthwise convolutional calculations. The convolutional processing cores (200) are built on a depthwise separable convolutional calculation hardware architecture. The convolutional processing cores (200) include reconfigurable processing units (210) and MAT units (220). A RISC-V processor (300) is configured to control the TileLink bus module (100) to acquire the convolutional data of the corresponding device and to control the convolutional processing core (200) to perform corresponding convolutional calculations on the convolutional data.

2. The flexible convolutional network accelerator based on a RISC-V processor according to claim 1, characterized in that, The reconfigurable processing unit (210) includes several PE columns, each PE column including several parallel multiply-accumulate units and several multiplexers; The multiply-accumulate unit includes a multiplier, an adder, and a data register connected in sequence. The data input of the multiplexer is connected to the adder in the previous row of the multiply-accumulate unit and the data register in the next row of the multiply-accumulate unit, respectively. The data output of the multiplexer is connected to the adder in the next row of the multiply-accumulate unit. The multiplier is configured to perform cumulative multiplication on the data to be convolved, the adder is configured to perform cumulative addition on the data to be convolved, and the data register is configured to store and output the calculation result; The bottom row of the PE column's multiply-accumulate unit includes the multiplier, the adder, the data register, and the bottom row multiplexer. The bottom row multiplexer is located between the multiplier and the adder. The data input of the bottom row multiplexer is connected to the adder in the previous row's multiply-accumulate unit and the multiplier in the current row's multiply-accumulate unit, respectively. The data output of the bottom row multiplexer is connected to the adder in the current row's multiply-accumulate unit. After receiving the data transmitted by the adder in the current row, the data register in the bottom row multiply-accumulate unit sends the data back to the adder in the current row. After completing processing, the multiplier in the bottom row multiply-accumulate unit sends the processed data to the bottom row multiplexer and the multiplexer in the top row multiply-accumulate unit, respectively.

3. The flexible convolutional network accelerator based on a RISC-V processor according to claim 2, characterized in that, The reconfigurable processing unit (210) is configured to perform the ordinary convolution calculation, the pointwise convolution calculation, and / or the depthwise convolution calculation on the data to be convolved.

4. The flexible convolutional network accelerator based on a RISC-V processor according to claim 3, characterized in that, The reconfigurable processing unit (210) is further configured to: Perform convolution calculations in DW mode or PW mode, wherein DW mode supports depthwise convolution calculations and PW mode supports both normal convolution calculations and pointwise convolution calculations. When performing convolution calculations in DW mode, the data port in the multiplexer used to transmit data between the upper and lower rows is closed; When performing convolution calculations in PW mode, the data port in the path selector used to transfer data between the adder and the data register in the same row is closed.

5. The flexible convolutional network accelerator based on a RISC-V processor according to claim 4, characterized in that, The MAT unit (220) is configured to perform ordinary convolution calculation and / or pointwise convolution calculation on the data to be convolved, and the MAT unit (220) and the reconfigurable processing unit (210) perform convolution calculation simultaneously or separately.

6. The flexible convolutional network accelerator based on a RISC-V processor according to claim 5, characterized in that, The convolutional processing kernel (200) further includes a data cache unit A (230), a data cache unit B (240), a data cache unit C (250), and an auxiliary cache unit (260). All the convolutional processing kernels (200) share the data cache unit A (230) and the data cache unit C (250). Each convolutional processing kernel (200) includes the corresponding data cache unit B (240) and the auxiliary cache unit (260). The auxiliary cache unit (260) is configured to cache the calculation results of the reconfigurable processing unit (210) and provide input data for the MAT unit (220) and the reconfigurable processing unit (210); When performing the depthwise convolution calculation, the data cache unit A (230) is configured to store the input feature map data of the data to be convolved, and the data cache unit B (240) is configured to store the convolution kernel weight data of the data to be convolved; During the pointwise convolution calculation, the data cache unit A (230) and the data cache unit C (250) are configured to store the convolution kernel weight data of the data to be convolved, and the data cache unit B (240) is configured to store the input feature map data of the data to be convolved; The convolution processing kernel (200) is also configured to perform convolution calculations on the data to be convolved using a computation data stream, the computation data stream including time-division multiplexing computation and pipelined computation.

7. A flexible convolutional network accelerator based on a RISC-V processor according to claim 6, characterized in that, The time-division multiplexing calculation includes: When performing the pointwise convolution calculation or the ordinary convolution calculation, the convolution calculation is performed jointly by the reconfigurable processing unit (210) and the MAT unit (220); When performing the depthwise convolution calculation, the convolution calculation is performed by the reconfigurable processing unit (210); When performing the depthwise convolution calculation, the pointwise convolution calculation, and the ordinary convolution calculation, the MAT unit (220) performs them first, and after the reconfigurable processing unit (210) completes the depthwise convolution calculation, the MAT unit (220) and the reconfigurable processing unit (210) jointly perform the pointwise convolution calculation and the ordinary convolution calculation.

8. A flexible convolutional network accelerator based on a RISC-V processor according to claim 6, characterized in that, The pipelined computation includes, when the depthwise convolution computation is performed on the reconfigurable processing unit (210), caching the computation result of the reconfigurable processing unit (210) as the input data for the pointwise convolution computation in the auxiliary cache unit (260), and transmitting the input data cached in the auxiliary cache unit (260) to the MAT unit (220).

9. A flexible convolutional network accelerator based on a RISC-V processor according to claim 6, characterized in that, The convolutional processing kernel (200) also includes: An intermediate processing unit (270) is disposed between the reconfigurable processing unit (210) and the auxiliary cache unit (260), and the intermediate processing unit (270) is configured to shift, truncate and rearrange the calculation results of the reconfigurable processing unit (210); A post-processing unit (280) is disposed after the MAT unit (220), and the post-processing unit (280) is configured to perform shifting, truncation and activation processing on the calculation results of the MAT unit (220); An output buffer unit (290) is configured to cache the final calculation result of the convolution calculation on the data to be convolved.

10. A flexible convolutional network accelerator based on a RISC-V processor according to claim 9, characterized in that, The accelerator also includes: BootRom module (400), which is configured to store a firmware bootloader for power-on startup of the convolutional processing core (200); An interrupt controller (500) is configured to handle interrupt requests from external devices or software programs to the convolutional processing core (200); A DDR controller (600) configured as an interface for connecting to the memory of an external storage device; PCIe controller (700), the PCIe controller (700) being configured as a redundant interface for connecting external device memory; The AXI module (800) is configured to arbitrate and route the data to be convolved and the final computation result, respectively.

Images