A (2+1)D convolution hardware structure of an FPGA-based R(2+1)D network hardware accelerator

Abstract

The present invention discloses a (2+1)D convolution hardware structure of an R(2+1)D network hardware accelerator based on FPGA. The hardware structure comprises an accelerator core deployed on the FPGA; the accelerator core is connected to a DDR via an AXI4 bus and interacts with the DDR for feature maps and weight data; the accelerator core is also connected to a host via an AXI4-lite bus, and the host reads and writes configuration registers in the accelerator core via the AXI4-lite bus; the accelerator core comprises a convolution module, a pooling module and a fully connected module; the convolution module is responsible for more than 90% of the computational load and storage capacity of the entire accelerator core and is responsible for performing convolution calculations and element-by-element addition operations; the pooling module and the fully connected module are responsible for the final average pooling and fully connected layer calculations of the network; and the convolution module adopts a pipelined (2+1)D convolution hardware architecture.

Description

Technical Field

[0001] The present invention relates to the field of computer recognition technology, and more specifically, to a (2+1)D convolution hardware structure of an R(2+1)D network hardware accelerator based on FPGA. Background Art

[0002] In the fields of video action recognition, 3D medical image diagnosis, etc., 3D convolutional neural network (3D CNN) has been widely used and achieved considerable recognition efficiency and accuracy. The convolution calculation method in 3D CNN is as follows: Figure 1 The three-dimensional convolution (3D Conv) shown in the figure is different from the two-dimensional convolution (2DConv) in traditional two-dimensional convolutional neural networks used in fields such as image recognition. The input and output feature maps and convolution kernels of 3D Conv are both three-dimensional, which can extract the temporal dimension feature information between different frames. The standard 3DCNN (C3D) uses this convolution method.

[0003] However, the standard 3D Conv has a large amount of computation and consumes a lot of storage resources. In addition, 3D Conv does not distinguish between temporal and spatial features, but performs 3D Conv in a general way, which creates unnecessary computational burden. Therefore, a series of 3D CNN variants have been produced, among which R(2+1)D CNN solves this problem well and shows excellent accuracy while significantly reducing the number of parameters. ResNet(2+1)D network (R(2+1)D) adopts the following method: Figure 2 The (2+1)D convolution ((2+1)D Conv) shown in the figure decomposes the 3D Conv into two independent and continuous 2D spatial convolutions and 1D temporal convolutions. (2+1)D Conv has two main advantages over 3D Conv: the first advantage is the additional nonlinear correction between the two operations. Compared with the network using full 3D convolution for the same number of parameters, this effectively doubles the number of nonlinearities, enabling the model to represent more complex functions. The second potential benefit is that the decomposition helps with optimization, resulting in lower training loss and lower test loss in practice. R(2+1)DCNN achieves state-of-the-art performance on video datasets including UCF101, Kinetics, and Sports-1M.

[0004] like Figure 3 The network structure is R(2+1)D, and the main body consists of a stem layer, four layer layers, an average pooling layer, and a fully connected layer. The stem layer contains a 1×7×7 2D convolution and a 3×1×1 1D convolution. The layer consists of two blocks, such as Figure 4Each block contains two (2+1)D convolutions, a shortcut connection, and a shortcut connection that may contain a 1×1×1 convolution. Table 1 shows the parameter information of the R(2+1)D network.

[0005] Table 1 Structure and parameter information of R(2+1)D network

[0006]

[0007] Currently, 3D CNN accelerators are primarily designed for 3D convolution. This convolution acceleration method is computationally inefficient when applied to (2+1)D convolutions. Furthermore, many designs use the 3D Winograd algorithm, which is ineffective when applied to R(2+1)D CNNs, which have large variations in dimension, size, and step size. Therefore, accelerators for R(2+1)D networks have not yet been fully researched and explored. Summary of the Invention

[0008] In order to solve the problems of the huge computational load and storage requirements of 3D CNN and the shortcomings and defects caused by the existing technology of placing the entire network on the FPGA chip and adopting a fully pipelined architecture, the present invention provides an FPGA-based R(2+1)D network hardware accelerator (2+1)D convolution hardware structure.

[0009] In order to achieve the above-mentioned purpose of the present invention, the technical solutions adopted are as follows:

[0010] A (2+1)D convolution hardware structure of an R(2+1)D network hardware accelerator based on FPGA, the hardware structure comprising an accelerator core deployed on the FPGA;

[0011] The accelerator core is connected to the DDR via the AXI4 bus, and interacts with the DDR for feature maps and weight data;

[0012] The accelerator core is also connected to the host via an AXI4-lite bus, and the host reads and writes configuration registers in the accelerator core via the AXI4-lite bus;

[0013] The accelerator core includes a convolution module, a pooling module and a fully connected module;

[0014] The convolution module is responsible for more than 90% of the computation and storage capacity of the entire accelerator core, and is responsible for performing convolution calculations and element-by-element addition operations;

[0015] The pooling module and the fully connected module are responsible for the final average pooling and fully connected layer calculations of the network;

[0016] The convolution module adopts a pipelined (2+1)D convolution hardware architecture.

[0017] Preferably, the convolution module includes a 2D convolution processing unit, a 2D weight cache unit, a feature map cache unit, a 2D output cache unit, a 1D convolution processing unit, a 1D weight cache unit, a 0D output cache unit, an element-by-element addition unit, a first selector, a second selector, a third selector, and a fourth selector;

[0018] The first input end of the first selector is connected to the DDR, and the second input end of the first selector is connected to the output end of the fourth selector; the output end of the first selector is connected to the 2D convolution processing unit and the first input end of the second selector respectively;

[0019] The input end of the 2D weight cache unit is connected to the DDR; the output end of the 2D weight cache unit is connected to the 2D convolution processing unit;

[0020] The output end of the 2D convolution processing unit is connected to the input end of the 2D output buffer unit;

[0021] The output end of the 2D output buffer unit is connected to the second input end of the second selector;

[0022] The output end of the second selector is connected to the 1D convolution processing unit;

[0023] The input end of the 1D weight cache unit is connected to the DDR, and the output end of the 1D weight cache unit is connected to the 1D convolution processing unit;

[0024] The output end of the 1D convolution processing unit is connected to the first input end of the third selector, the element-by-element addition unit, and the first input end of the fourth selector respectively;

[0025] The output end of the 0D output buffer unit is connected to the element-by-element addition unit;

[0026] The output end of the element-by-element adding unit is connected to the second input end of the third selector and the second input end of the fourth selector respectively.

[0027] Furthermore, the 1D convolution processing unit is used to perform the 1*1*1 convolution calculation, and the calculation results are stored in the 0D output buffer unit for subsequent element-by-element addition operations.

[0028] The feature map data is only written from DDR to the on-chip feature map cache unit at the beginning and written back to DDR at the end; the feature map data of the middle layer are all stored in the on-chip feature map cache unit;

[0029] The feature map cache unit is used to store the input feature map of the 2D convolution processing unit and the output feature map of the 1D convolution processing unit. After calculating a (2+1)D convolution, the result of the 1D convolution processing unit does not need to be written back to the DDR, and the 2D convolution processing unit does not need to read the input feature map data of the next layer from the DDR.

[0030] Furthermore, the working mode of the convolution module is as follows:

[0031] Working mode 1: (2+1)D convolution in the stem layer

[0032] After DDR writes the input feature map data into the feature map cache unit, it starts pipeline (2+1)D convolution. The result of the 2D convolution processing unit is cached in the 2D output cache unit. The 1D convolution processing unit reads the data in the 2D output cache unit as the input feature map, and the calculation result is directly written back to the feature map cache unit. At the same time, in order to ensure the normal element-by-element addition of the next block, the calculation result of the 1D convolution processing unit is also written into the 0D output cache unit.

[0033] Working mode 2: There is no 1*1*1 convolution in the shortcut connection in the block, and the first (2+1)D convolution in the block is calculated.

[0034] 2D convolution reads the output feature map of the previous round of 1D convolution from the feature map cache unit as the input feature map, convolution, and the results processed by the 2D convolution processing unit are cached in the 2D output cache unit; the 1D convolution processing unit reads the data in the 2D output cache unit as the input feature map, and the calculation results are directly written back to the feature map cache unit;

[0035] Since the input feature map data required for the element-by-element addition of the block has been stored in the 0D output buffer unit in the previous (2+1) convolution, there is no need to read or write it;

[0036] Working mode 3: The shortcut connection in the block contains a 1*1*1 convolution, and the first (2+1)D convolution in the block is calculated.

[0037] The 2D convolution processing unit reads data from the feature map cache unit, and the 1D convolution processing unit also reads data through the second selector for 1*1*1 convolution; since the computational complexity of 1*1*1 convolution is less than that of 2D convolution, all 1*1*1 convolution results are stored in the 0D output cache unit before the 2D convolution processing unit outputs a completed output feature map channel slice;

[0038] Working mode 4: The second (2+1)D convolution in the block, if there is a 1*1*1 convolution in the block

[0039] After performing the element-by-element addition operation, the data read from the 0D output buffer unit is added to the output result of the 1D convolution processing unit and then written back to the 0D output buffer unit;

[0040] Working mode 5: The second (2+1)D convolution in the block, if the block does not have 1*1*1 convolution

[0041] 1) If the next block contains a 1*1*1 convolution or is the last block of all convolutional layers, the 0D output buffer unit remains unchanged;

[0042] 2) If the next block does not have a 1*1*1 convolution, the data read from the 0D output buffer unit is added to the output result of the 1D convolution processing unit and then written into the 0D output buffer unit again.

[0043] Furthermore, the number of DSPs consumed by the 2D convolution processing unit is 64*(128 / 2)=

[0044] 4096; the number of DSPs consumed by the 1D convolution processing unit is 128*(64 / 2)=4096.

[0045] Furthermore, the feature map cache unit adopts an input-output switchable buffer structure, which has one write port and two read ports.

[0046] According to the slicing strategy, the feature map cache allocates 24.5Mb of storage space, with a cache depth of 49K and a width of 512. Taking into account the switchable structure of the input and output feature maps, the total storage space consumed is 49Mb, consuming a total of 208 URAMs.

[0047] Furthermore, the 2D weight cache unit and the 1D weight cache unit both adopt a ping-pong structure;

[0048] The 2D weight cache unit buffer stores the weight data corresponding to the current input slice channel slice and output channel slice, and consumes the following storage space:

[0049] N weight_buffer_2D =N ic ×T c ×kernel_size_2D×DW

[0050] According to the slicing strategy, the storage space required for the 2D convolution weights is 10.125Mb, the buffer depth is 20.25K, the bit width is 512, and the storage space consumed by the ping-pong structure is 20.5Mb, consuming a total of 96 URAM resources;

[0051] The 1D weight cache unit, considering storing the current input channel slice and the weight data of all corresponding output channels, consumes the following storage space:

[0052] N weight_buffer_1D =T c ×N oc ×kernel_size_1D×DW

[0053] According to the slicing strategy, the storage space required is 6.75Mb, the buffer depth is 6.75K, and the bit width is 1024. Taking into account the ping-pong structure, the storage space consumed is 13.5Mb, consuming a total of 64 URAM resources.

[0054] Furthermore, the storage space consumption of the output channel slice buffer of the 2D convolution processing unit is:

[0055] N output_buffer_2D =N ic ×T c ×of_map_size*DW

[0056] According to the slicing strategy, the storage space required by the 2D output buffer unit is 55.125Mb, the buffer depth is 55.125K, the bit width is 1024, and a total of 210 URAM resources are consumed;

[0057] The storage space required by the 0D output buffer unit is 24.5Mb, the buffer depth is 24.5K, the bit width is 1024, and a total of 105 URAM resources are consumed.

[0058] Furthermore, the hardware structure adopts the following input and output channel expansion method: the convolution calculation of the convolution module parallelly calculates the multiplication of uic input feature maps and uic*uoc weights in each cycle and obtains the partial sum of uoc output feature maps;

[0059] The slicing method adopted by the hardware structure is as follows: for 2D convolution, the input feature map data of all input channels are directly stored in the feature map cache unit, and the output channel is sliced, each slice has Tc output channels, and is stored in the 2D output cache unit, for a total of N slices; for 1D convolution, the output channel slice of the 2D convolution is its input channel slice, and the output feature map data of all its output channels are directly stored in the feature map buffer;

[0060] Furthermore, the loop sequence between slices is as follows:

[0061] For 2D convolution, all input feature maps are in the on-chip buffer. Each time, a 2D weight slice is read from the DDR, with a total of Nic*Tc*kernel_size_2D weight data. The output feature map data of Tc output channels can be calculated and stored in the 2D output cache unit.

[0062] For 1D convolution, the Tc channel output feature map data of the 2D convolution output is used as the input data, combined with the 1D weight slices read from DDR, a total of Noc*Tc*kernel_size_1D weight data, and the partial sum of the output feature map of Noc channels can be calculated;

[0063] The order of loops within a slice is as follows:

[0064] Assuming that the bit width of the data in the weight and input feature map buffers is uic*DW, the convolution module requires uoc cycles to load the weights, and only one cycle to load the input feature map; in order to improve computational efficiency, the 2D weight cache unit and the 1D weight cache unit adopt a ping-pong structure, and the time for weight reuse is set to be no less than the time it takes to load the weights into the convolution module.

[0065] The beneficial effects of the present invention are as follows:

[0066] This paper adopts a centralized, local pipelined architecture, performing convolution calculations. All convolutional layer calculations are performed using convolution modules. Optimization methods tailored to the specific characteristics of the 2D and 1D convolution processing units in the (2+1)D convolution are implemented, improving overall computational throughput and local computational hardware utilization.

[0067] Aiming at the time reuse characteristics of weight data in 2D convolution and 1D convolution in (2+1)D convolution, a weight data time reuse method that adapts to their respective characteristics is proposed. The present invention improves the computational efficiency of the 2D convolution processing unit and the 1D convolution processing unit, making the computational efficiency close to 100%.

[0068] The present invention reuses 1D convolution to calculate the 1*1*1 convolution in the branch connection (shortcut connection) of the residual building block, and proposes a data flow optimization method for element-by-element addition after the shortcut connection. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is the 3D convolution operation of the prior art.

[0070] Figure 2 is a 3D convolution calculation, where Figure 2 (a) Performing a full 3D convolution using a kernel of size t×d×d, where t represents the temporal extent and d is the spatial width and height. Figure 2 (b) The (2+1)D convolution block splits the computation into spatial 2D convolution followed by temporal 1D convolution.

[0071] Figure 3 It is the ResNet(2+1)D network structure.

[0072] Figure 4 It is the Block structure in the R(2+1)D network.

[0073] Figure 5 This is a diagram of the hardware structure of the (2+1)D convolution accelerator described in the present invention.

[0074] Figure 6 It is a schematic diagram of 2D convolution in R(2+1)D in the present invention.

[0075] Figure 7 It is a schematic diagram of 1D convolution in R(2+1)D in the present invention.

[0076] Figure 8 It is a hardware structure diagram of the convolution module described in the present invention.

[0077] Figure 9 It is a hardware structure diagram of the convolution processing unit of the present invention.

[0078] Figure 10 It is a hardware structure diagram of the feature map cache unit of the present invention.

[0079] Figure 11 It is a schematic diagram of the ping-pong structure of the present invention.

[0080] Figure 12 It is a schematic diagram of the expansion method of the input and output channels of the present invention.

[0081] Figure 13 It is a (2+1)D convolutional slice loop.

[0082] Figure 14 It is a timing diagram of the 2D convolution processing unit of the present invention.

[0083] Figure 15 It is a timing diagram of the 1D convolution processing unit of the present invention. DETAILED DESCRIPTION

[0084] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0085] Example 1

[0086] Due to the huge computational and storage requirements of 3D CNN, it is definitely unrealistic to put the ResNet (2+1)D network on an FPGA chip and use a fully pipelined architecture. Therefore, this invention adopts a centralized architecture for convolution calculations, that is, all convolution layer calculations use the convolution module as the computing core, and all convolution layers are calculated sequentially on this computing core. The details are as follows:

[0087] like Figure 5 As shown, a (2+1)D convolution hardware structure of an R(2+1)D network hardware accelerator based on FPGA, the hardware structure includes an accelerator core deployed on the FPGA;

[0088] The accelerator core is connected to the DDR via the AXI4 bus, and interacts with the DDR for feature maps and weight data;

[0089] The accelerator core is also connected to the host via an AXI4-lite bus, and the host reads and writes configuration registers in the accelerator core via the AXI4-lite bus;

[0090] The accelerator core includes a convolution module, a pooling module and a fully connected module;

[0091] The convolution module is responsible for more than 90% of the computation and storage of the entire accelerator core, and is responsible for performing convolution calculations and element-by-element addition operations;

[0092] The pooling module and the fully connected module (fc module) are responsible for the final average pooling and fully connected layer calculations of the network;

[0093] The convolution module adopts a pipelined (2+1)D convolution hardware architecture.

[0094] In a specific embodiment, the R(2+1)D network splits the 3D convolution into a 2D convolution ( Figure 6 ) and 1D convolution ( Figure 7 ), but the input and output feature maps of these two convolutions are both three-dimensional. Both 2D and 1D convolutions can be considered special cases of 3D Conv. In 2D convolution, the temporal dimension of the convolution kernel is 1, while in 1D convolution, the spatial dimension of the convolution kernel is 1×1. A purely centralized design—implementing a general-purpose 3D Conv on an FPGA and using it to compute 2D and 1D convolutions—would inevitably lead to inefficient data arrangement and addressing.

[0095] Due to the difference in convolution kernel dimensions between 2D convolution and 1D convolution, this embodiment proposes the following Figure 8Pipeline (2+1)D convolution hardware architecture.

[0096] The convolution module includes a 2D convolution processing unit (2D PE), a 2D weight buffer unit (weight buffer2D), a feature map buffer unit (feature map buffer), a 2D output buffer unit (output buffer 2D), a 1D convolution processing unit (1D PE), and a 1D weight buffer unit.

[0097] (weight buffer 1D), 0D output buffer unit (output buffer 0D), element-wise addition unit (element-wise addition), first selector, second selector, third selector, fourth selector;

[0098] The first input end of the first selector is connected to the DDR, and the second input end of the first selector is connected to the output end of the fourth selector; the output end of the first selector is connected to the 2D convolution processing unit and the first input end of the second selector respectively;

[0099] The input end of the 2D weight cache unit is connected to the DDR; the output end of the 2D weight cache unit is connected to the 2D convolution processing unit;

[0100] The output end of the 2D convolution processing unit is connected to the input end of the 2D output buffer unit;

[0101] The output end of the 2D output buffer unit is connected to the second input end of the second selector;

[0102] The output end of the second selector is connected to the 1D convolution processing unit;

[0103] The input end of the 1D weight cache unit is connected to the DDR, and the output end of the 1D weight cache unit is connected to the 1D convolution processing unit;

[0104] The output end of the 1D convolution processing unit is connected to the first input end of the third selector, the element-by-element addition unit, and the first input end of the fourth selector respectively;

[0105] The output end of the 0D output buffer unit is connected to the element-by-element addition unit;

[0106] The output end of the element-by-element adding unit is connected to the second input end of the third selector and the second input end of the fourth selector respectively.

[0107] In a specific embodiment, the convolution module can perform three functions: 2D convolution, 1D convolution, and 1*1*1 convolution, and has the following characteristics:

[0108] 1) The 1D convolution processing unit is used to reuse the 1*1*1 convolution calculation, and the calculation results are stored in the 0D output buffer unit for subsequent element-by-element addition operations.

[0109] 2) Feature map data is only initially written from DDR to the on-chip feature map cache unit and then written back to DDR. All feature map data in the middle layer is stored in the on-chip feature map cache unit, which reduces the amount of data access to DDR, improves memory access efficiency, and reduces power consumption.

[0110] 3) The feature map cache unit is used to store the input feature map of the 2D convolution processing unit and the output feature map of the 1D convolution processing unit. After calculating a (2+1)D convolution, the result of the 1D convolution processing unit does not need to be written back to the DDR, and the 2D convolution processing unit does not need to read the input feature map data of the next layer from the DDR.

[0111] 4) Weight data is large and varies from layer to layer, so it needs to be frequently read from the DDR. To improve computational efficiency, both the 2D and 1D weight cache units adopt a ping-pong structure. While data in one buffer is being used for computation, the other buffer reads data from the DDR to prepare for the next computation.

[0112] In a specific embodiment, the convolution module operates as follows:

[0113] Working mode 1: (2+1)D convolution in the stem layer

[0114] The first two convolutions of the R(2+1)D network are not in the residual building block, so there is no shortcut connection and no element-by-element addition process. After DDR writes the input feature map data into the feature map cache unit, it starts the pipeline (2+1)D convolution. The result of the 2D convolution processing unit is cached in the 2D output cache unit; the 1D convolution processing unit reads the data in the 2D output cache unit as the input feature map, and the calculation result is directly written back to the feature map cache unit; at the same time, in order to ensure the normal element-by-element addition of the next block, the calculation result of the 1D convolution processing unit is also written into the 0D output cache unit;

[0115] Working mode 2: There is no 1*1*1 convolution in the shortcut connection in the block, and the first (2+1)D convolution in the block is calculated.

[0116] The 2D convolution reads the output feature map of the previous round of 1D convolution from the feature map cache unit as the input feature map, and the result processed by the 2D convolution processing unit is cached in the 2D output cache unit; the 1D convolution processing unit reads the data in the 2D output cache unit as the input feature map, and the calculation result is directly written back to the feature map cache unit;

[0117] Since the input feature map data required for the element-by-element addition of the block has been stored in the 0D output buffer unit in the previous (2+1) convolution, there is no need to read or write it;

[0118] Working mode 3: The shortcut connection in the block contains a 1*1*1 convolution, and the first (2+1)D convolution in the block is calculated.

[0119] The 2D convolution processing unit reads data from the feature map cache unit, and the 1D convolution processing unit also reads data through the second selector for 1*1*1 convolution; since the computational complexity of 1*1*1 convolution is less than that of 2D convolution, all 1*1*1 convolution results are stored in the 0D output cache unit before the 2D convolution processing unit outputs a completed output feature map channel slice;

[0120] Working mode 4: The second (2+1)D convolution in the block, if there is a 1*1*1 convolution in the block

[0121] The addressing mode of element-by-element addition is relatively simple, and the data will not be reused after the addition. After the element-by-element addition operation, the data read from the 0D output buffer unit is added to the output result of the 1D convolution processing unit and then written back to the 0D output buffer unit;

[0122] Working mode 5: The second (2+1)D convolution in the block, if the block does not have 1*1*1 convolution

[0123] 1) If the next block contains a 1*1*1 convolution or is the last block of all convolutional layers, the 0D output buffer unit remains unchanged;

[0124] 2) If the next block does not have a 1*1*1 convolution, the data read from the 0D output buffer unit is added to the output result of the 1D convolution processing unit and then written into the 0D output buffer unit again.

[0125] Example 2

[0126] Based on the hardware structure described in Example 1, the specific implementation of the hardware structure of the convolution module is discussed below, taking the Xilinx Alveo U250 data center accelerator card as an example.

[0127] Table 1 Statistics of important hardware resources on the Alevo U250 accelerator card

[0128] Resource Name BRAM URAM DSP Number of resources 2688 1280 12288 Storage space 96.868Mb 368.64Mb -

[0129] like Figure 9 As shown, in this embodiment, the feature map data and weight data are quantized into 8 bits, and the multiplication in the convolution module is implemented by the DSP in the FPGA.

[0130] A DSP can implement multiplication with a bit width of 27*18, so a DSP can perform two 8-bit multiplications of the form a(b+c).

[0131] The calculation formula for the number of DSPs consumed by a convolution processing unit is

[0132] N DSP =uic×(uoc / 2)

[0133] The number of DSPs consumed by the 2D convolution processing unit described in this embodiment is 64*(128 / 2)=4096; the number of DSPs consumed by the 1D convolution processing unit described in this embodiment is 128*(64 / 2)=4096.

[0134] In a specific embodiment, the on-chip buffering in this embodiment is implemented using URAM resources on the FPGA. The advantage of URAM is that it is deeper than BRAM, and when cascaded into a larger RAM, it has dedicated address selection and wiring resources, which can build a large-capacity and high-speed on-chip storage space.

[0135] In a specific embodiment, Figure 10 As shown, the feature map cache unit adopts an input-output switchable buffer structure, which has one write port and two read ports. The thick arrow represents the input feature map data required for 2D convolution written to the feature map cache unit, and the arrow represents the reading of the input feature map data from the feature map cache unit to the 2D convolution processing unit. Since the input feature maps of 2D convolution and 1*1*1 convolution are the same, the dotted line represents the reading of the input feature map data to the 1D convolution processing unit for 1*1*1 convolution calculation. There are two buffers inside the feature map cache, one for reading and one for writing. The reading and writing modes are exchanged after switching the state.

[0136] Based on the slicing strategy, the storage space consumed by the 2D convolution input feature map slice and the 1D convolution output feature map is 24.5Mb, so the total storage space of the feature map cache unit is 49Mb. The depth of both buffers in the feature map cache unit is 49K, and the data bit width is 512, consuming a total of 208URAM resources.

[0137] In a specific embodiment, in order to improve the overall computing efficiency, the 2D weight cache unit and the 1D weight cache unit both adopt a ping-pong structure, that is, when the buffered data in the current buffer is used for PE calculation, the other one reads the weight data required for the next calculation from the DDR.

[0138] The 2D weight cache unit buffers and stores the weight data corresponding to the current and output channel slices, and consumes the following storage space:

[0139] N weight_buffer_2D =N ic ×T c ×Kernel)siz_2D×DW

[0140] According to the slicing strategy, the storage space allocated to the 2D convolution weights is 10.125 Mb, the buffer depth is 20.25 KB, the bit width is 512, and the storage space consumed by the ping-pong structure is 20.5 Mb, consuming a total of 96 URAM resources.

[0141] The 1D weight cache unit, considering storing the current input channel slice and the weight data of all corresponding output channels, consumes the following storage space:

[0142] N weight_buffer_1D =T c ×N oc ×Kernel_siz_1D×DW

[0143] According to the previous slicing strategy, the storage space required is 6.75Mb, the buffer depth is 6.75K, and the bit width is 1024. Taking into account the ping-pong structure, the storage space consumed is 13.5Mb, consuming a total of 64 URAM resources.

[0144] In a specific embodiment, the storage space consumption of the output channel slice buffer of the 2D convolution processing unit is:

[0145] N output_buffer_2D =N ic ×T c ×of_map_size*DW

[0146] According to the previous slicing strategy, the storage space required by the 2D output buffer unit is 55.125Mb, the buffer depth is 55.125K, the bit width is 1024, and a total of 210 URAM resources are consumed;

[0147] Because the 0D output buffer unit not only stores the calculation results of 1*1*1 convolution, but also may store the calculation results of 1D PE or the results of element-by-element addition unit, the storage space required by the 0D output buffer unit is 24.5Mb, the buffer depth is 24.5K, and the bit width is 1024, consuming a total of 105 URAM resources.

[0148] Example 3

[0149] Based on the hardware structure described in Example 2, this embodiment also provides a convolution calculation method of a (2+1)D convolution hardware structure of an R(2+1)D network hardware accelerator based on FPGA.

[0150] The hardware structure adopts the input and output channel expansion method, such as Figure 12 As shown, each cycle of the convolution calculation of the convolution module parallelly calculates the multiplication of uic input feature maps and uic*uoc weights and obtains the partial sum of uoc output feature maps.

[0151] This embodiment utilizes input and output channel expansion to effectively accommodate convolution kernels of varying sizes. Because the input and output channels are typically large, this approach achieves high parallelism and multiplier utilization. Furthermore, the input and output channels exhibit a clear pattern of change, resulting in good versatility. The addressing logic for on-chip buffer data is also relatively simple.

[0152] The selection of expansion coefficients is mainly based on the following considerations:

[0153] 1. If the DSP resources on the FPGA chip allow, try to use a larger expansion coefficient.

[0154] 2. The uic of 1D convolution is equal to the uoc of 2D convolution.

[0155] 3. The uoc of 1D convolution is equal to the uic of 2D convolution, or a multiple of uic, because the result of 1D convolution is written to the feature map cache and will be read into the convolution module as the input feature map of the next 2D convolution. The bit width of the 2D convolution input feature map cache is uic*DW, and the bit width of the 1D convolution output feature map cache is uoc*DW. When uoc is an integer multiple of uic, it is convenient for the next round of 2D convolution to read the result of 1D convolution.

[0156] For (2+1)D convolution, combined with the distribution of input and output channels of the R(2+1)D network at each layer and the number of computing resources of the U250 accelerator card, the present invention uses the following expansion coefficients:

[0157] 1.2D convolution input channel expansion coefficient: Uic=64

[0158] 2.2D convolution output channel and 1D convolution input channel expansion coefficient: Uc=128

[0159] 3.1D convolution output channel expansion coefficient: Uoc=64

[0160] In this embodiment, Nic is the number of channels of the 2D convolution input feature map, Nc is the number of output channels of the 2D convolution and the number of input channels of the 1D convolution, and Noc is the number of output channels of the 1D convolution. Due to the storage advantages of the UltraScale+ series FPGAs and the pipelined (2+1)D convolution structure, the slicing strategy adopted by the hardware structure is as follows:

[0161] 1. For 2D convolution, directly store the input feature map data of all input channels into the on-chip buffer, slice the output channels, and each slice has Tc output channels, for a total of N slices;

[0162] 2. For 1D convolution, the output channel slice of 2D convolution is its input channel slice, and the output feature map data of all its output channels is directly stored in the on-chip buffer;

[0163] The on-chip buffer includes a feature map cache unit, a 2D output cache unit, and a 0D output cache unit.

[0164] The slicing strategy is set based on the following considerations:

[0165] 1. The input feature maps of all channels of 2D convolution and all output feature maps of 1D convolution are all stored in the on-chip buffer. Firstly, the storage resources of the on-chip buffer can be fully utilized. Secondly, the output feature maps of 1D convolution are stored on the chip and can be directly used as input feature map data for the next round of 2D convolution, avoiding unnecessary read and write DDR operations.

[0166] 2. The reason for slicing the output channels of the 2D convolution is to alleviate the storage pressure caused by the weight data, so that the method and architecture of the present invention can be applied to more FPGA devices with different resource counts.

[0167] 2D convolution output channel slice coefficients:

[0168] <![CDATA[N c ]]> 45 144 144 144 144 230 230 288 288 460 460 576 576 921 921 1152 1152 Tc 45 144 144 144 144 230 230 288 288 460 460 576 576 576 288 288 288

[0169] In a specific embodiment, Figure 13 As shown, the loop sequence between slices is as follows:

[0170] For 2D convolution, all input feature maps are in the on-chip buffer. Each time, a 2D weight slice is read from the DDR, with a total of Nic*Tc*kernel_size_2D weight data. The output feature map data of Tc output channels can be calculated and stored in the 2D output cache unit.

[0171] For 1D convolution, the Tc channel output feature map data of the 2D convolution output is used as the input data, combined with the 1D weight slices read from DDR, a total of Noc*Tc*kernel_size_1D weight data, and the partial sum of the output feature maps of Noc channels can be calculated.

[0172] The order of loops within a slice is as follows:

[0173] The intra-slice loop involves the addressing method and computational efficiency of the on-chip buffer. Taking 2D convolution as an example, assuming that the bit width of the data in the weight and input feature map buffers is Uic*DW, then the convolution module requires Uc cycles to load the weights, while loading the input feature map only requires one cycle. Because the time it takes to load weights is much longer than the time it takes to input feature map data, it is necessary to improve the time utilization of the weight data already loaded into the PE. To improve computational efficiency, the 2D weight cache unit and the 1D weight cache unit adopt a ping-pong structure, and the time it takes to reuse weights is set to be no less than the time it takes to load the weights into the convolution module.

[0174] For 2D convolution, the following methods are used to improve the reuse time of weight data on the convolution module:

[0175] 1. Since 2D convolution is a convolution in the spatial dimension, and the convolution kernel is two-dimensional, the 3D input feature map can be reused Nid times in the temporal dimension. Therefore, the convolution kernel size of 2D convolution is cyclic.

[0176] 2. The minimum value of Nid is 2, which is much smaller than Uc. It can be seen that reuse in the time dimension alone is not enough, so the loop order needs to be adjusted to improve the time reuse of the weights. The implementation method of the present invention is to give priority to the execution of the partial output feature map loop, that is, to divide the output feature map size into a group of Uc / 2. The calculation of these Uc / 2 pixels requires the weight data in PE. Because Nid*Uc / 2≥Uc, the calculation efficiency can be close to 100% through the structure of the ping-pong buffer register.

[0177] Summarizing the above parts, we can get the timing diagram of the 2D convolution processing unit (as shown in Figure 14) and the loop sequence, where the pseudo code of the loop sequence is as follows:

[0178]

[0179] The following figure shows the pseudo code of the 2D convolution slice loop. The specific loop process is:

[0180] 1. For the output feature map of Tc channels to be calculated on the chip, since the expansion degree is Uc, each time Uc channels are processed in parallel, a loop is required Second-rate.

[0181] 2. When processing the to-th output channel expansion, since the input channel expansion degree is Uic, each time Uic channels are processed in parallel, it is necessary to loop through the input channel Second-rate.

[0182] 3. After determining that the to-th output channel is expanded and the ti-th input channel is expanded, the specific output feature map is processed. Due to the strategy of output feature map sub-loop to improve computational efficiency, Noh*

[0183] Now's output feature map is divided into vertical and horizontal blocks, and the size of each block is Use th and tw as the coordinates of the vertical and horizontal double loop to traverse these sub-blocks.

[0184] 4. After determining the block size, a double loop of the convolution kernel size is performed to obtain the Uic*Uc weight data required for subsequent calculations. These weights will be reused in subsequent loops and will also be used in parallel during expansion.

[0185] 5. Traverse the output feature map pixels in the output feature map sub-block through a double loop with sth and stw as coordinates.

[0186] 6. Use od to traverse the time dimension of these pixels.

[0187] 7. Finally, there is the loop within the expanded input and output channels.

[0188] For 1D convolution, since the convolution calculation is performed on the time dimension of the input feature map, the weights can reuse the spatial dimension of the input feature map. Since the size of the spatial dimension of the input feature map is greater than Uoc=64 in most cases, the computational utilization rate can reach 100%. Even when the size of the spatial dimension of the input feature map is 49, the computational efficiency can reach 49 / 64=76.6%.

[0189] So we can get the following timing diagram of 1D convolution processing unit (such as Figure 15 As shown) and the pseudo code of the slice inner loop of the 1D convolution processing unit are as follows:

[0190]

[0191] The following figure shows the pseudo code of the 1D convolution slice loop. The specific loop process is:

[0192] 1. Traverse the output channels. Since the expansion degree is Uoc, Uoc output channels are calculated in parallel, and a total of loops are required Second-rate.

[0193] 2. When the to-th output channel is expanded, the input channel of the 1D convolution is expanded to Uic, and Uc input channels are calculated in parallel. After the 2D volume outputs the feature map data of Tc output channels, it is necessary to traverse the data of these Tc channels in a loop. Second-rate.

[0194] 3. After determining the position of the input and output channels, start traversing the time dimension of the output feature map, because the convolution operation of 1D convolution is performed in this dimension.

[0195] 4. Then traverse the convolution kernel and determine the Uc*Uoc weight data, which will be reused in subsequent loops.

[0196] 5. Traverse the spatial dimensions oh and ow of the output feature map.

[0197] 6. Finally, the calculations within the input and output channels are expanded, which can be completed at once after expansion.

[0198] Obviously, the above embodiments of the present invention are merely examples for the purpose of illustrating the present invention, and are not intended to limit the embodiments of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A (2+1)D convolution hardware device for an R(2+1)D network hardware accelerator based on FPGA, characterized by: The hardware device includes an accelerator core deployed on an FPGA; The accelerator core is connected to the DDR via the AXI4 bus, and interacts with the DDR for feature maps and weight data; The accelerator core is also connected to the host via an AXI4-lite bus, and the host reads and writes configuration registers in the accelerator core via the AXI4-lite bus; The accelerator core includes a convolution module, a pooling module and a fully connected module; The convolution module is responsible for more than 90% of the computation and storage capacity of the entire accelerator core, and is responsible for performing convolution calculations and element-by-element addition operations; The pooling module and the fully connected module are responsible for the final average pooling and fully connected layer calculations of the network; The convolution module adopts a pipelined (2+1)D convolution hardware architecture; The convolution module includes a 2D convolution processing unit, a 2D weight cache unit, a feature map cache unit, a 2D output cache unit, a 1D convolution processing unit, a 1D weight cache unit, a 0D output cache unit, an element-by-element addition unit, a first selector, a second selector, a third selector, and a fourth selector. The first input end of the first selector is connected to the DDR, and the second input end of the first selector is connected to the output end of the fourth selector; the output end of the first selector is connected to the 2D convolution processing unit and the first input end of the second selector respectively; The input end of the 2D weight cache unit is connected to the DDR; the output end of the 2D weight cache unit is connected to the 2D convolution processing unit; The output end of the 2D convolution processing unit is connected to the input end of the 2D output buffer unit; The output end of the 2D output buffer unit is connected to the second input end of the second selector; The output end of the second selector is connected to the 1D convolution processing unit; The input end of the 1D weight cache unit is connected to the DDR, and the output end of the 1D weight cache unit is connected to the 1D convolution processing unit; The output end of the 1D convolution processing unit is connected to the first input end of the third selector, the element-by-element addition unit, and the first input end of the fourth selector respectively; The output end of the 0D output buffer unit is connected to the element-by-element addition unit; The output end of the element-by-element adding unit is connected to the second input end of the third selector and the second input end of the fourth selector respectively.

2. The FPGA-based R(2+1)D network hardware accelerator (2+1)D convolution hardware device according to claim 1, characterized in that: The 1D convolution processing unit is used to perform 1*1*1 convolution calculation reuse, and the calculation results are stored in the 0D output buffer unit for subsequent element-by-element addition operations. The feature map data is only written from DDR to the on-chip feature map cache unit at the beginning and written back to DDR at the end; the feature map data of the middle layer are all stored in the on-chip feature map cache unit; The feature map cache unit is used to store the input feature map of the 2D convolution processing unit and the output feature map of the 1D convolution processing unit. After calculating a (2+1)D convolution, the result of the 1D convolution processing unit does not need to be written back to the DDR, and the 2D convolution processing unit does not need to read the input feature map data of the next layer from the DDR.

3. The FPGA-based R(2+1)D network hardware accelerator (2+1)D convolution hardware device according to claim 1, characterized in that: The working mode of the convolution module is as follows: Working mode 1: (2+1)D convolution in the stem layer After DDR writes the input feature map data into the feature map cache unit, it starts pipeline (2+1)D convolution. The result of the 2D convolution processing unit is cached in the 2D output cache unit. The 1D convolution processing unit reads the data in the 2D output cache unit as the input feature map, and the calculation result is directly written back to the feature map cache unit. At the same time, in order to ensure the normal element-by-element addition of the next block, the calculation result of the 1D convolution processing unit is also written into the 0D output cache unit. Working mode 2: There is no 1*1*1 convolution in the shortcut connection in the block, and the first (2+1)D convolution in the block is calculated. The 2D convolution reads the output feature map of the previous round of 1D convolution from the feature map cache unit as the input feature map, and the result processed by the 2D convolution processing unit is cached in the 2D output cache unit; the 1D convolution processing unit reads the data in the 2D output cache unit as the input feature map, and the calculation result is directly written back to the feature map cache unit; Since the input feature map data required for the element-by-element addition of the block has been stored in the 0D output buffer unit in the previous (2+1) convolution, there is no need to read or write it; Working mode 3: The shortcut connection in the block contains a 1*1*1 convolution, and the first (2+1)D convolution in the block is calculated; The 2D convolution processing unit reads data from the feature map cache unit, and the 1D convolution processing unit also reads data through the second selector for 1*1*1 convolution; since the computational complexity of 1*1*1 convolution is less than that of 2D convolution, all 1*1*1 convolution results are stored in the 0D output cache unit before the 2D convolution processing unit outputs a completed output feature map channel slice; Working mode 4: The second (2+1)D convolution in the block, if there is a 1*1*1 convolution in the block After performing the element-by-element addition operation, the data read from the 0D output buffer unit is added to the output result of the 1D convolution processing unit and then written back to the 0D output buffer unit; Working mode 5: The second (2+1)D convolution in the block, if the block does not have 1*1*1 convolution 1) If the next block contains a 1*1*1 convolution or is the last block of all convolutional layers, the 0D output buffer unit remains unchanged; 2) If the next block does not have a 1*1*1 convolution, the data read from the 0D output buffer unit is added to the output result of the 1D convolution processing unit and then written into the 0D output buffer unit again.

4. The FPGA-based R(2+1)D network hardware accelerator (2+1)D convolution hardware device according to claim 1, characterized in that: The number of DSPs consumed by the 2D convolution processing unit is 64*(128 / 2)=4096; the number of DSPs consumed by the 1D convolution processing unit is 128*(64 / 2)=4096.

5. The FPGA-based R(2+1)D network hardware accelerator (2+1)D convolution hardware device according to claim 1, characterized in that: The feature map cache unit adopts an input-output switchable buffer structure, which has one write port and two read ports; According to the slicing strategy, the storage space allocated to the feature map cache is 24.5Mb, the cache depth is 49K, and the bit width is 512; considering the switchable structure of the input and output feature maps, the storage space consumed is 49Mb, and a total of 208 URAMs are consumed.

6. The FPGA-based R(2+1)D network hardware accelerator (2+1)D convolution hardware device according to claim 1, characterized in that: The 2D weight cache unit and the 1D weight cache unit both adopt a ping-pong structure; The 2D weight cache unit buffer stores the weight data corresponding to the current output channel slice, and consumes the following storage space: N weight_buffer_2D =N ic ×T c ×kernel_size_2D×DW According to the slicing strategy, the storage space required for the 2D convolution weights is 10.125Mb, the buffer depth is 20.25K, the bit width is 512, and the storage space consumed by the ping-pong structure is 20.5Mb, consuming a total of 96 URAM resources; The 1D weight cache unit, considering storing the current input channel slice and the weight data of all corresponding output channels, consumes the following storage space: N weight_buffer_1D =T c ×N oc ×kernel_size_1D×DW According to the slicing strategy, the storage space required to be allocated is 6.75Mb, the buffer depth is 6.75K, and the bit width is 1024; considering the ping-pong structure, the storage space consumed is 13..5Mb, and a total of 64 URAM resources are consumed.

7. The FPGA-based R(2+1)D network hardware accelerator (2+1)D convolution hardware device according to claim 1, characterized in that: Since the storage space consumption of the output channel slice buffer of the 2D convolution processing unit is: N output_buffer_2D =N ic ×T c ×of_map_size*DW According to the slicing strategy, the storage space required by the 2D output buffer unit is 55.125Mb, the buffer depth is 55.125K, the bit width is 1024, and a total of 210 URAM resources are consumed; The storage space required by the 0D output buffer unit is 24.5Mb, the buffer depth is 24.5K, the bit width is 1024, and a total of 105 URAM resources are consumed.

8. The FPGA-based R(2+1)D network hardware accelerator (2+1)D convolution hardware device according to any one of claims 1 to 7, characterized in that: The hardware device adopts the following input and output channel expansion method: each cycle of the convolution calculation of the convolution module parallelly calculates the multiplication of uic input feature maps and uic*uoc weights and obtains the partial sum of uoc output feature maps; The slicing method adopted by the hardware device is as follows: for 2D convolution, the input feature map data of all input channels are directly stored in the feature map cache unit, and the output channel is sliced. Each slice T c Output channels are stored in the 2D output buffer unit, with a total of N slices. For 1D convolution, the output channel slice of the 2D convolution is its input channel slice, and the output feature map data of all its output channels is directly stored in the feature map buffer.

9. The FPGA-based R(2+1)D network hardware accelerator (2+1)D convolution hardware device according to claim 8, characterized in that: The order of loops between slices is as follows: For 2D convolution, all input feature maps are in the on-chip buffer, and a 2D weight slice is read from DDR each time, for a total of N ic *T c *kernel_size_2D weight data, calculated to get T c The output feature map data of each output channel is stored in the 2D output buffer unit; For 1D convolution, the T of the 2D convolution output c The channel output feature map data is used as input data, combined with the 1D weight slices read from DDR, a total of N oc *T c *kernel_size_1D weight data, calculation can get N oc The partial sum of the output feature maps of the channels; The order of loops within a slice is as follows: Assuming that the bit width of the data in the weight and input feature map buffers is uic*DW, the convolution module requires uoc cycles to load the weights, and only one cycle to load the input feature map; in order to improve computational efficiency, the 2D weight cache unit and the 1D weight cache unit adopt a ping-pong structure, and the time for weight reuse is set to be no less than the time it takes to load the weights into the convolution module.

Images