FPGA-oriented lightweight dimension separable convolution method

Abstract

The invention discloses a lightweight dimension separable convolution method for an FPGA (Field Programmable Gate Array). The method comprises the following steps: obtaining a four-dimensional feature tensor to be convolved; dividing the four-dimensional feature tensor to be convolved into a plurality of first three-dimensional feature tensors; performing second-dimension and third-dimension convolution operation on each first three-dimensional feature tensor to obtain a plurality of second three-dimensional tensors; performing first dimension convolution operation on each second three-dimensional tensor to obtain a plurality of third three-dimensional tensors; combining the plurality of third three-dimensional tensors to form a first four-dimensional tensor; and performing fourth dimension fusion operation on the first four-dimensional tensor to obtain a final four-dimensional tensor. The method has the characteristics of short operation time and low power consumption.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and hardware architecture technology, and more specifically, to a lightweight, dimensionally separable convolution method for FPGAs. Background Technology

[0002] Three-dimensional convolutional networks (3D CNNs), as a powerful feature extraction tool, have achieved excellent performance in a variety of tasks. 3D CNNs can simultaneously process the spatial and temporal dimensions of data, thus possessing significant advantages in spatiotemporal feature learning. For example, in fields such as millimeter-wave radar data processing, video stream analysis, and medical imaging, 3D CNNs can effectively extract spatiotemporal information, providing crucial support for tasks such as object detection, behavior recognition, and scene analysis.

[0003] Traditional 3D convolutional networks face significant computational and resource challenges in deep learning applications. Standard 3D convolutional kernels typically require simultaneous convolution operations in both spatial (width (W) and height (H)) and temporal (frames (T)) dimensions. The kernel size is K_t × K_h × K_w, causing the number of parameters and computational cost to increase rapidly with increasing K_t, K_h, and K_w. While this increase in computational cost and number of parameters can improve the model's expressive power, it also makes its deployment in practical applications more complex and expensive. Especially on low-power computing platforms such as FPGAs, standard 3D convolutions often require massively parallel computing units and frequent external memory accesses, making their deployment in real-time applications extremely difficult and failing to meet the requirements of real-time performance (long computation time) and low power consumption.

[0004] The prior art discloses a 3DCNN implementation method based on a systolic array FPGA-overlayed convolutional neural network (CNN) accelerator. The accelerator includes an FPGA chip and an external memory. The on-chip storage resources in the FPGA chip are used for data caching and streaming, and the external memory is used as the main data storage device. The 3D CNN implementation method includes the following steps: (1) Introducing a depth counting and data flow control mechanism in the DMA read and input feed path, loading feature data sequentially along the depth dimension and completing the accumulation of intermediate results, thereby realizing three-dimensional convolution calculation on a two-dimensional systolic array; (2) Using a three-dimensional folding and vectorized mapping strategy, rearranging the data blocks corresponding to the 3D convolution window according to the step size relationship of depth / height / width and mapping them to multi-channel vector input, so as to improve the utilization of the PE array and reduce the repeated loading of filters; (3) Performing operator decomposition and mapping on 3D pooling, making 3D average pooling equivalent to convolution operation, and decomposing 3D max pooling into two-level processing of two-dimensional planar pooling and depth direction pooling, so as to reuse existing two-dimensional pooling units. This method essentially relies on a two-dimensional array to perform time-based expansion of the depth dimension. Furthermore, the three-dimensional folding and pooling decomposition are strongly coupled with the kernel size and step size configuration, resulting in limited support for more complex or morphologically varied 3D networks and operators. Its versatility and scalability remain constrained. Summary of the Invention

[0005] This invention addresses the shortcomings of existing convolution techniques, such as high power consumption and long computation time on FPGA platforms, by providing a lightweight, dimensionally separable convolution method for FPGAs. This method features short computation time and low power consumption.

[0006] The primary objective of this invention is to solve the aforementioned technical problems. The technical solution of this invention is as follows: A lightweight, dimensionally separable convolution method for FPGAs includes: S1: Obtain the four-dimensional feature tensor to be convolved; S2: Divide the four-dimensional feature tensor to be convolved into multiple first three-dimensional feature tensors; S3: Perform a second-dimensional and third-dimensional convolution operation on each first three-dimensional feature tensor to obtain multiple second three-dimensional tensors; S4: Perform a first-dimensional convolution operation on each second three-dimensional tensor to obtain multiple third three-dimensional tensors; merge the multiple third three-dimensional tensors to form a first four-dimensional tensor; S5: Perform a fourth-dimensional fusion operation on the first four-dimensional tensor to obtain the final four-dimensional tensor.

[0007] Furthermore, in step S3, the convolution operation in the second and third dimensions includes: S301: Perform two-dimensional depth convolution on the second and third dimensions of each of the first three-dimensional feature tensors to obtain multiple first intermediate three-dimensional tensors; S302: Perform nonlinear activation on each of the first intermediate three-dimensional tensors to obtain multiple second three-dimensional tensors.

[0008] Furthermore, a two-dimensional depthwise convolution is performed on the second and third dimensions of the first three-dimensional feature tensor to obtain the first intermediate three-dimensional tensor, which includes: S30101: Set the two-dimensional convolution kernel; use multiple random access memories on the FPGA chip as row buffers; S30102: Based on the first three-dimensional feature tensor, continuously cache feature tensors of more than a first preset number of rows in the row buffer; S30103: Obtain the regions to be convolved in the multiple row buffers and expand them to form multiple first vectors; S30104: Unfold the two-dimensional convolution kernel to form a second vector; S30105: Perform vector dot product operation between the multiple first vectors and the second vector respectively to obtain the first intermediate three-dimensional tensor.

[0009] Furthermore, in step S3, the convolution operation in the second and third dimensions includes: S311: Perform a one-dimensional convolution on the second dimension of each of the first three-dimensional feature tensors to obtain multiple second intermediate three-dimensional tensors; S312: Perform a one-dimensional convolution on the third dimension of each second intermediate three-dimensional tensor to obtain multiple third intermediate three-dimensional tensors; S313: Perform nonlinear activation on each third intermediate three-dimensional tensor to obtain multiple second three-dimensional tensors.

[0010] Furthermore, a one-dimensional convolution is performed on the second dimension of the first three-dimensional feature tensor to obtain a second intermediate three-dimensional tensor, including: S31101: Set a one-dimensional convolution kernel; S31102: Based on the first three-dimensional feature tensor, cache the second preset number of feature tensors in the on-chip row buffer structure of the FPGA; S31103: The feature tensors in the row buffer structure are divided using a sliding window to obtain multiple third vectors; S31104: Perform a vector dot product operation between each third vector and a one-dimensional convolution kernel to obtain multiple second intermediate three-dimensional tensors.

[0011] Further, in step S4, the first-dimensional convolution operation includes: S401: Perform a one-dimensional convolution on the first dimension of each second three-dimensional tensor to obtain multiple fourth intermediate three-dimensional tensors; S402: Perform nonlinear activation operations on each fourth intermediate three-dimensional tensor to obtain multiple third three-dimensional tensors.

[0012] Furthermore, in step S5, the fourth-dimensional fusion operation includes: S501: Divide the first four-dimensional tensor into multiple fourth-dimensional vectors according to the fourth dimension; S502: Perform vector-matrix multiplication on each fourth-dimensional vector to obtain multiple fourth-dimensional vectors after multiplication. S503: Integrate multiple fourth-dimensional vectors that have undergone multiplication operations to obtain the first intermediate four-dimensional tensor; S504: Perform a nonlinear activation operation on the first intermediate four-dimensional tensor to obtain the final four-dimensional tensor.

[0013] Furthermore, the formula for vector-matrix multiplication is as follows:

[0014] Indicates the first dimension. Indicates the second dimension. Indicates the third dimension. Represents the fourth-dimensional vector. This represents the preset weight matrix. This represents the preset bias matrix.

[0015] An FPGA-based convolution execution method includes: S01: Obtain the four-dimensional feature tensor to be convolved; write the four-dimensional feature tensor to be convolved into the large-capacity block random access memory resource pool on the FPGA chip; S02: Construct a sliding window using the shift register logic on the FPGA chip, and use the sliding window to read data from the large-capacity block random access memory resource pool to obtain the read data; S03: Input the read data into the depth convolution calculation array on the FPGA chip for calculation to obtain convolutional data; S04: Input the convolutional data into the point convolution calculation engine on the FPGA chip to obtain the final four-dimensional tensor.

[0016] A lightweight, dimensionally separable convolutional system for FPGAs, comprising: Tensor Acquisition Module: Acquires the four-dimensional feature tensor to be convolved; Tensor partitioning module: partitions the four-dimensional feature tensor to be convolved into multiple first three-dimensional feature tensors; First convolution module: Performs second-dimensional and third-dimensional convolution operations on each first three-dimensional feature tensor to obtain multiple second three-dimensional tensors; The second convolution module performs a first-dimensional convolution operation on each second three-dimensional tensor to obtain multiple third three-dimensional tensors; the multiple third three-dimensional tensors are merged to form a first four-dimensional tensor. Dimension fusion module: Performs a fourth-dimensional fusion operation on the first four-dimensional tensor to obtain the final four-dimensional tensor.

[0017] Compared with the prior art, the beneficial effects of the present invention are: This invention is based on a lightweight, dimensionally separable convolution method for FPGAs. It splits and merges traditional four-dimensional convolution into multiple convolutional branches of different dimensions, ensuring that the input and output data of each convolutional branch are in the same order and facilitating unified scheduling. This avoids the difficulties in data interoperability and the problem of repeated construction of large-scale parallel multiply-accumulate units caused by the complex calculation order and data layout of traditional multi-dimensional convolutions. While maintaining the multi-dimensional receptive field, it significantly reduces the number of parameters and multiply-accumulate operations, reduces the occupation of hardware resources such as LUTs, DSPs and on-chip storage, and further improves the implementation efficiency and adaptability on FPGAs, realizing lightweight hardware acceleration of multi-dimensional convolutional networks. Attached Figure Description

[0018] Figure 1 The flowchart is provided for a lightweight, dimensionally separable convolution method for FPGAs, as shown in Example 1.

[0019] Figure 2 This is a schematic diagram illustrating the computational decomposition of spatiotemporally separable convolution provided in Example 1.

[0020] Figure 3 This is a schematic diagram illustrating the execution of the double buffering mechanism provided in Example 1.

[0021] Figure 4 A schematic diagram of the quantization mechanism provided in Example 1.

[0022] Figure 5 This is a schematic diagram of a lightweight, dimensionally separable convolutional system for FPGA provided in Example 1.

[0023] Figure 6 This is a schematic diagram of the FPGA structure provided in Example 2. Detailed Implementation

[0024] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions; It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0025] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0026] Example 1 like Figure 1 As shown, a lightweight, dimensionally separable convolution method for FPGAs includes: S1: Obtain the four-dimensional feature tensor to be convolved; S2: Divide the four-dimensional feature tensor to be convolved into multiple first three-dimensional feature tensors; S3: Perform a second-dimensional and third-dimensional convolution operation on each first three-dimensional feature tensor to obtain multiple second three-dimensional tensors; S4: Perform a first-dimensional convolution operation on each second three-dimensional tensor to obtain multiple third three-dimensional tensors; merge the multiple third three-dimensional tensors to form a first four-dimensional tensor; S5: Perform a fourth-dimensional fusion operation on the first four-dimensional tensor to obtain the final four-dimensional tensor.

[0027] In one specific embodiment, step S2 is performed as follows: On the FPGA chip, the four-dimensional input feature tensor and the convolution weights of each layer are received from external memory via the AXI bus, and the input feature tensor is represented as... Where T is the number of time frames, and H and W are the spatial dimensions. Input the number of channels; The input features are stored and rearranged in the FPGA's on-chip cache according to a preset line buffer and time buffer structure. The FPGA forms a series of BRAMs.

[0028] (Total n items) The feature map of depth is taken sequentially for each... Calculations are performed using (three-dimensional) channels, letting This provides a data stream for subsequent spatiotemporally separable convolution calculations. Furthermore, such as Figure 2 As shown, in step S3, the convolution operation in the second and third dimensions includes: S301: Perform two-dimensional depth convolution on the second and third dimensions of each of the first three-dimensional feature tensors to obtain multiple first intermediate three-dimensional tensors; S302: Perform nonlinear activation on each of the first intermediate three-dimensional tensors to obtain multiple second three-dimensional tensors.

[0029] In a specific embodiment, the convolution operation of the second and third dimensions is performed as follows: On the FPGA chip, for the input feature tensor Each channel (3D) is executed with a size of Spatial (only in the H and W directions) 2D depthwise convolution yields intermediate feature tensors. Convolution is performed only within the same channel, and the kernel size is [size missing]. Step size is Boundary fill is The output space size meets

[0030] Formed in FPGA through row buffer A sliding window with indexes at each time point. Output space location ,aisle Extract the corresponding A local region is converted into a region of length by data tiling. The spatial feature vector is then multiplied by the convolution kernel vector in the same channel to generate a spatial convolution output.

[0031] Spatial convolution is calculated as follows:

[0032] For each input channel Configure independent 2D convolution kernels Convolution is performed only within the spatial dimension of the channel, without inter-channel mixing; on the FPGA chip, a row buffer is formed by several BRAMs, with continuous caching at least... Line input features, such that each output position Required A local region can be directly obtained from the row buffer; expand this local region in row or column order to a length of... Spatial feature vectors and convolution kernel Expand into vectors Performed through a multiply-accumulate array The vector dot product operation is used to achieve spatial depth convolution in a unified vector form.

[0033] The activation function is expressed as:

[0034] Perform pointwise nonlinear activation operations on each element of the intermediate feature tensor Y obtained in the above steps to obtain the activated intermediate feature tensor Y'. The nonlinear activation includes at least one of ReLU, Leaky ReLU, Sigmoid or Tanh, in order to enhance the nonlinear expressive power of spatial features.

[0035] Furthermore, a two-dimensional depthwise convolution is performed on the second and third dimensions of the first three-dimensional feature tensor to obtain the first intermediate three-dimensional tensor, which includes: S30101: Set the two-dimensional convolution kernel; use multiple random access memories on the FPGA chip as row buffers; S30102: Based on the first three-dimensional feature tensor, continuously cache feature tensors of more than a first preset number of rows in the row buffer; S30103: Obtain the regions to be convolved in the multiple row buffers and expand them to form multiple first vectors; S30104: Unfold the two-dimensional convolution kernel to form a second vector; S30105: Perform vector dot product operation between the multiple first vectors and the second vector respectively to obtain the first intermediate three-dimensional tensor.

[0036] Furthermore, in step S3, the convolution operation in the second and third dimensions includes: S311: Perform a one-dimensional convolution on the second dimension of each of the first three-dimensional feature tensors to obtain multiple second intermediate three-dimensional tensors; S312: Perform a one-dimensional convolution on the third dimension of each second intermediate three-dimensional tensor to obtain multiple third intermediate three-dimensional tensors; S313: Perform nonlinear activation on each third intermediate three-dimensional tensor to obtain multiple second three-dimensional tensors.

[0037] In a specific embodiment, the convolution operation of the second and third dimensions is performed as follows: On the FPGA chip, for the input feature tensor Each channel is subjected to one-dimensional depthwise convolution along the height direction H and the width direction W for spatial feature extraction, replacing the traditional two-dimensional spatial convolution operation. This effectively extracts spatial structure features while reducing computational complexity and storage overhead, resulting in an intermediate spatial feature tensor. ).

[0038] While maintaining the time dimension T, width dimension W, and number of channels Under the premise of not changing, for each channel of the input feature tensor, only perform one-dimensional depthwise convolution operation along the spatial height direction H; Each input channel is configured with an independent one-dimensional convolution kernel. The convolution stride is Boundary fill is Convolution is performed only within the same channel, without inter-channel mixing. On the FPGA chip, adjacent rows at the same width position are continuously cached using a row buffer structure. The row feature data forms a one-dimensional sliding window in the height direction; the feature values ​​within this sliding window are expanded in height order into a length of... The height feature vector is then used to perform a vector dot product operation with the corresponding one-dimensional convolution kernel vector of the channel, generating a one-dimensional spatial feature output in the H direction, thus obtaining the intermediate feature tensor. .

[0039] While maintaining the time dimension T, height dimension H′, and number of channels Under the premise of not changing, for the intermediate feature tensor For each channel, a one-dimensional depthwise convolution operation is performed only along the spatial width direction W; Each input channel is configured with an independent one-dimensional convolution kernel. The convolution stride is Boundary fill is Convolution is also performed only within the channels; On the FPGA chip, adjacent elements at the same height position are continuously cached using a column buffer structure. The feature data is used to form a one-dimensional sliding window along the width direction; the feature values ​​within this sliding window are then expanded in width order to form a window of length. The width feature vector is obtained and then a vector dot product operation is performed with the corresponding one-dimensional convolution kernel vector of the channel to generate a one-dimensional spatial feature output in the W direction, ultimately yielding the spatial feature tensor. .

[0040] For the obtained spatial feature tensor Each element in the tensor undergoes a pointwise nonlinear activation operation to obtain the activated spatial feature tensor. The nonlinear activation function includes, but is not limited to, at least one of ReLU, Leaky ReLU, Sigmoid, or Tanh, to enhance the spatial feature representation capability extracted by one-dimensional convolution.

[0041] Furthermore, a one-dimensional convolution is performed on the second dimension of the first three-dimensional feature tensor to obtain a second intermediate three-dimensional tensor, including: S31101: Set a one-dimensional convolution kernel; S31102: Based on the first three-dimensional feature tensor, cache the second preset number of feature tensors in the on-chip row buffer structure of the FPGA; S31103: The feature tensors in the row buffer structure are divided using a sliding window to obtain multiple third vectors; S31104: Perform a vector dot product operation between each third vector and a one-dimensional convolution kernel to obtain multiple second intermediate three-dimensional tensors.

[0042] Further, in step S4, the first-dimensional convolution operation includes: S401: Perform a one-dimensional convolution on the first dimension of each second three-dimensional tensor to obtain multiple fourth intermediate three-dimensional tensors; S402: Perform nonlinear activation operations on each fourth intermediate three-dimensional tensor to obtain multiple third three-dimensional tensors.

[0043] In a specific embodiment, the specific operation of the first-dimensional convolution is as follows: Maintaining space dimensions , Without changing the underlying structure, perform on-chip processing of the activated intermediate feature tensor on the FPGA chip. Each channel performs an execution size along the time dimension of Temporal one-dimensional depthwise convolution yields the spatiotemporal feature tensor. ,in, The number of channels after temporal convolution (without changing the total number of channels) is [value], and the time step is [time step]. Time fill The output time length satisfies ; In FPGAs, time buffering is used to maintain the same spatial location in adjacent spaces. The feature values ​​in each time frame are used to transform the time window into a length of [length missing]. The temporal feature vector is then multiplied by the corresponding channel's temporal convolution kernel vector to generate the temporal convolution output.

[0044] Temporal convolution is calculated as follows:

[0045] For each channel Configure temporal convolution kernels Convolution is performed only along the time dimension; on the FPGA chip, for each spatial location... Set a time buffer, continuously caching at least... The feature values ​​of each time frame at this location constitute a time window; the feature values ​​within the time window are then... The eigenvalues ​​are expanded in chronological order to form a time feature vector. And the temporal convolution kernel is expanded into a vector. Executed via multiply-accumulate array The vector dot product operation is used to implement temporal depthwise convolution in a unified vector form.

[0046] Each element in the obtained spatiotemporal feature tensor Z undergoes pointwise nonlinear activation to obtain the activated spatiotemporal feature tensor Z'. The nonlinear activation function may be the same as or different from the activation functions in other steps to further enhance the modeling capability of temporal dynamics. The nonlinear activation includes at least one of ReLU, Leaky ReLU, Sigmoid, or Tanh to enhance the nonlinear expressive power of spatial features.

[0047] The activation function is expressed as:

[0048] Furthermore, in step S5, the fourth-dimensional fusion operation includes: S501: Divide the first four-dimensional tensor into multiple fourth-dimensional vectors according to the fourth dimension; S502: Perform vector-matrix multiplication on each fourth-dimensional vector to obtain multiple fourth-dimensional vectors after multiplication. S503: Integrate multiple fourth-dimensional vectors that have undergone multiplication operations to obtain the first intermediate four-dimensional tensor; S504: Perform a nonlinear activation operation on the first intermediate four-dimensional tensor to obtain the final four-dimensional tensor.

[0049] In a specific embodiment, the fourth-dimensional fusion operation is performed as follows: On the FPGA chip, the activated spatiotemporal feature tensor Execute size in the channel dimension. The point convolution operation converts the number of channels to... The feature mapping is for a channel number of Output feature tensor Specifically, for any output position The feature values ​​of all channels at this location are tiled to form a data structure of length [length missing]. The channel feature vector is obtained and a vector-matrix multiplication operation is performed with the point convolution weight matrix stored on the FPGA chip or in external memory to obtain all output channel values ​​at that position, thus realizing cross-channel information fusion.

[0050] Point convolution is represented as:

[0051] spatiotemporal feature tensor same position All channel values ​​are tiled to form a data structure of length [length missing]. Channel feature vectors Store the point convolution weights of all output channels in matrix form. The bias vector is Performed on the FPGA chip via vector-matrix multiplication array. The operation generates all output channel values ​​at that position at once, thus completing channel fusion in a unified vector-matrix multiplication form.

[0052] Pointwise nonlinear activation operations are performed on each element of 0 in the obtained output feature tensor to obtain the final output feature tensor. The final output features are then written back to external memory or transferred to subsequent network layers or post-processing modules via the AXI interface module, realizing the inference process of a spatiotemporally separable convolutional 3D convolutional network based on FPGA. The final network output is... It is four-dimensional data.

[0053] The activation function is expressed as:

[0054] Furthermore, the formula for vector-matrix multiplication is as follows:

[0055] Indicates the first dimension. Indicates the second dimension. Indicates the third dimension. Represents the fourth-dimensional vector. This represents the preset weight matrix. This represents the preset bias matrix.

[0056] In one specific embodiment, the multiply-accumulate array is a single set of multiply-accumulate processing unit arrays on the FPGA chip. By configuring different weight loading methods and address generation strategies, time-division multiplexing is performed in the order of spatial convolution, temporal convolution, and point convolution.

[0057] The on-chip cache adopts a ping-pong double buffer structure, corresponding to the current computation data block and the next computation data block respectively. When one buffer is used for convolution operation, the other buffer performs data prefetching or result write-back in parallel. After the convolution operation is completed, the roles of the two buffers are switched to achieve pipeline parallelism of input, computation and output, and reduce external storage access latency.

[0058] The convolution weights and feature data are represented in a fixed-point format, preferably 8-bit or 16-bit fixed-point numbers. The multiplication and addition operations are performed in the FPGA's DSP resources. The convolution output is truncated or saturated on-chip before being used as the input for the next stage. Compared to the 32-bit floating-point implementation, the fixed-point implementation can significantly reduce the area of ​​a single multiplication and addition unit and the on-chip storage capacity.

[0059] The FPGA is connected to the external processing system via an AXI4 or AXI4-Stream bus. The input feature tensor and convolution weights are streamed from the external memory via the AXI bus, rearranged by the on-chip cache and data tiling module, and then input to the multiply-accumulate array. After activation and quantization processing, the convolution result is written back to the external memory or provided to the subsequent network layer via the AXI bus until all network layers have been calculated.

[0060] The multiply-accumulate arrays used for the spatial two-dimensional depthwise convolution, temporal one-dimensional depthwise convolution, and pointwise convolution steps are the same set of multiply-accumulate processing unit arrays on the FPGA chip. By configuring different weight loading methods and address generation strategies, time-division multiplexing is performed in the order of spatial convolution, temporal convolution, and pointwise convolution. In order to time-division multiplex spatial convolution, temporal convolution, and pointwise convolution in the same set of multiply-accumulate arrays, the different stages of spatiotemporal convolution are first defined: Spatial convolution: through The spatial characteristics of the local window are calculated; Temporal convolution: through The convolution kernel is used to calculate time series features; Point convolution: through The convolutional kernels fuse channel features.

[0061] Each stage uses the same array of multiply-accumulate processing units on the FPGA chip, but their computation order and weight loading methods differ. Assume the output of the multiply-accumulate unit is... Therefore, the time-division multiplexing calculation order can be written as follows:

[0062]

[0063]

[0064] in: The feature vectors of spatial convolution; The vector of the spatial convolution kernel; The feature vector of the temporal convolution; The vector of the temporal convolution kernel; For channel feature vectors; The point convolution weight matrix; This is a bias term.

[0065] As the clock cycle progresses, different parts of the data are loaded into the same set of multiply-accumulate units for processing in turn, thereby achieving time-division multiplexing.

[0066] The on-chip cache employs a ping-pong dual-buffering structure, corresponding to the current computation data block and the next computation data block respectively. When one buffer is used for convolution operations, the other buffer performs data prefetching or result write-back in parallel. After the convolution operation is completed, the roles of the two buffers are switched to achieve pipelined parallelism of input, computation, and output, reducing external storage access latency. Specifically, as follows... Figure 3 As shown, one buffer is used for the current computation, and the other buffer is used for data prefetching or storing the output results. Assume there are two buffers. and Within each clock cycle: when During convolution calculation, Used to store the next batch of input features; After the calculation is completed and Switch roles and continue convolution operations to maintain the parallelism between computation and data flow.

[0067] The double buffering mechanism for data streams can be represented by the following formula:

[0068]

[0069] After each convolution calculation is completed, data exchange and computation are performed in parallel.

[0070] like Figure 4 As shown, the convolution weights and feature data used are represented in a fixed-point format, preferably quantized as 8-bit numbers. The multiplication and addition operations are performed in the DSP resources of the FPGA. The convolution output is truncated or saturated on-chip and then used as the input for the next stage.

[0071] like Figure 5 As shown, a lightweight, dimensionally separable convolutional system for FPGAs includes: Tensor Acquisition Module: Acquires the four-dimensional feature tensor to be convolved; Tensor partitioning module: partitions the four-dimensional feature tensor to be convolved into multiple first three-dimensional feature tensors; First convolution module: Performs second-dimensional and third-dimensional convolution operations on each first three-dimensional feature tensor to obtain multiple second three-dimensional tensors; The second convolution module performs a first-dimensional convolution operation on each second three-dimensional tensor to obtain multiple third three-dimensional tensors; the multiple third three-dimensional tensors are merged to form a first four-dimensional tensor. Dimension fusion module: Performs a fourth-dimensional fusion operation on the first four-dimensional tensor to obtain the final four-dimensional tensor.

[0072] It should be noted that this invention proposes a method for computing three-dimensional convolutional networks based on spatiotemporally separable convolution, reducing the number of parameters in traditional three-dimensional convolution from... Reduced to This method significantly reduces the number of parameters and multiply-accumulate operations in convolutional layers. By using data tiling and vector-matrix multiplication, it simplifies the traditional complex 3D convolution calculation process into multiple small-scale matrix multiplication operations. Compared with traditional 3D convolution implementations, spatiotemporally separable convolution uses a unified multiply-accumulate array for time-division multiplexing, avoiding the construction of large-scale parallel computing units and significantly reducing FPGA resource consumption. Through this optimization, spatiotemporally separable convolution not only effectively reduces computational load but also significantly reduces the resource requirements and power consumption of FPGA designs, thus achieving a more lightweight computing solution.

[0073] Example 2 Based on the lightweight dimensionality-separable convolution method for FPGA described in Example 1, this example uses the same lightweight dimensionality-separable convolution method for FPGA as in Example 1.

[0074] An FPGA-based convolution execution method includes: S01: Obtain the four-dimensional feature tensor to be convolved; write the four-dimensional feature tensor to be convolved into the large-capacity block random access memory resource pool on the FPGA chip; S02: Construct a sliding window using the shift register logic on the FPGA chip, and use the sliding window to read data from the large-capacity block random access memory resource pool to obtain the read data; S03: Input the read data into the depth convolution calculation array on the FPGA chip for calculation to obtain convolutional data; S04: Input the convolutional data into the point convolution calculation engine on the FPGA chip to obtain the final four-dimensional tensor.

[0075] The various parts of an FPGA chip, such as Figure 6 As shown in the diagram. In the overall architecture, the left side represents external memory (DDR), and the area within the dashed box on the right represents the internal logic of the FPGA chip.

[0076] Data inflow: Data enters the FPGA's DMA read channel from DDR via the AXI read bus, and then is stored in the input FIFO.

[0077] Data preparation: The data input to the FIFO enters the data rearrangement unit (a row / time buffer composed of BRAM), where the data sliding and format adjustment are completed.

[0078] Core computation (spatiotemporal convolution): The rearranged feature data stream and the data in the weight cache are fed into the DSP computing array to complete the convolution operation in the spatial and temporal dimensions.

[0079] Post-processing and caching: The calculation results are processed by activation and post-processing units (LUT / logic implementation) and then temporarily stored in an intermediate buffer (BRAM / register file).

[0080] Core computation (channel fusion): The intermediate results and another set of weight data are fed into the point convolution computation engine (DSP + addition tree) to complete the final channel fusion.

[0081] Data write-back: The final calculation result is stored in the output FIFO, and then written back to the external DDR via the DMA write channel and AXI write bus.

[0082] Global Control: The Global Controller (FSM) is responsible for coordinating the operation of various modules and data transmission.

[0083] In this invention, the hardware processing structure of the FPGA includes: Storage control layer: includes the MIG controller that connects to the external DDR, the AXI interconnect matrix, and the DMA controller responsible for data transfer; Data caching layer: includes input / output FIFOs for isolating the clock domain, and a BRAM resource pool (Ping-Pong Buffer) for building sliding windows and rearranging data. Computation acceleration layer: Contains a multiply-accumulate array (PE Array) composed of FPGA DSP48 resources, used to perform specific fixed-point multiply-accumulate operations; Post-processing layer: Contains quantization, activation, and pooling modules implemented by FPGA logic slices and lookup tables (LUTs); The data flow between layers is as follows: external memory flows to DMA read, input FIFO flows to BRAM buffer, DSP computing array flows to post-processing module, output FIFO flows to DMA write, and external memory flows to DMA write.

[0084] The hardware accelerator of this invention is deployed on an FPGA chip, and its overall data processing framework and flow are as follows: The entire processing flow begins with external memory (such as DDR4 SDRAM) outside the FPGA chip, which stores the raw input feature map data to be processed and the pre-trained network weight parameters.

[0085] 1. Data Input and On-Chip Buffering: The Direct Memory Access Controller (DMAController) configured inside the FPGA acts as the master device, initiating read transaction requests to external memory via the high-performance AXI4 interconnect bus interface. The high-bandwidth data stream read first undergoes cross-clock domain synchronization and rate matching through an asynchronous input FIFO (First-In-First-Out queue), and is then written to the FPGA's on-chip large-capacity block random access memory (BRAM) resource pool. These BRAMs are logically divided into multi-level line buffers and time buffers for storing image feature data, and a weight buffer for storing convolution weights, providing a low-latency parallel access path for subsequent calculations.

[0086] 2. Spatiotemporal Depth Convolution Computation Pipeline: The data scheduling logic reads data in parallel from the row buffer and time buffer of the BRAM, and dynamically constructs a sliding window data stream that meets the requirements of the convolution kernel size through on-chip shift register logic (SRL). This data stream, along with the weight stream synchronously read from the weight buffer, is fed into the depth convolution computation array. This array consists of multiple parallel DSP slices (Digital Signal Processing slices, such as DSP48), each DSP slice efficiently performing multiply-accumulate operations. The computation results are accumulated internally by the DSP or in a tightly coupled CLB (Configurable Logic Block) accumulator. The accumulated intermediate feature stream undergoes nonlinear processing via an activation function module built based on a lookup table (LUT).

[0087] 3. Channel Fusion Computation Pipeline: The activated spatiotemporal feature data is temporarily stored in another set of on-chip intermediate BRAM buffers or distributed RAM (LUTRAM) for data rearrangement. Subsequently, this intermediate data is read out in parallel and fed into the point convolution computation engine along with a second set of point convolution weights. This engine is also built on a high-concurrency DSP slice array, and the multiplication results are output in parallel to an adder tree constructed from CLB carry chain resources, quickly completing the summation operation of multi-channel data to generate the final output feature data.

[0088] 4. Result Write-back and Global Control: The final calculated output data stream is first written to the output FIFO buffer for data accumulation and bit width adaptation. When the transmission conditions are met, the DMA controller initiates a write transaction again through the AXI4 bus interface, transferring the data burst in the FIFO back to the external memory on-chip, awaiting read by the host. The entire data flow, buffer read / write, and start / stop of the computing array are uniformly scheduled and synchronized by a global control unit based on a finite state machine (FSM), ensuring efficient pipeline operation.

[0089] The same or similar labels correspond to the same or similar parts; The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent. Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A lightweight, dimensionally separable convolution method for FPGAs, characterized in that, include: S1: Obtain the four-dimensional feature tensor to be convolved; S2: Divide the four-dimensional feature tensor to be convolved into multiple first three-dimensional feature tensors; S3: Perform a second-dimensional and third-dimensional convolution operation on each first three-dimensional feature tensor to obtain multiple second three-dimensional tensors; S4: Perform a first-dimensional convolution operation on each second three-dimensional tensor to obtain multiple third three-dimensional tensors; merge the multiple third three-dimensional tensors to form a first four-dimensional tensor; S5: Perform a fourth-dimensional fusion operation on the first four-dimensional tensor to obtain the final four-dimensional tensor.

2. The lightweight, dimensionally separable convolution method for FPGAs according to claim 1, characterized in that, In step S3, the convolution operations in the second and third dimensions include: S301: Perform two-dimensional depth convolution on the second and third dimensions of each of the first three-dimensional feature tensors to obtain multiple first intermediate three-dimensional tensors; S302: Perform nonlinear activation on each of the first intermediate three-dimensional tensors to obtain multiple second three-dimensional tensors.

3. The lightweight, dimensionally separable convolution method for FPGAs according to claim 2, characterized in that, Performing a two-dimensional depthwise convolution on the second and third dimensions of the first three-dimensional feature tensor yields a first intermediate three-dimensional tensor, including: S30101: Set the two-dimensional convolution kernel; use multiple random access memories on the FPGA chip as row buffers; S30102: Based on the first three-dimensional feature tensor, continuously cache feature tensors of more than a first preset number of rows in the row buffer; S30103: Obtain the regions to be convolved in the multiple row buffers and expand them to form multiple first vectors; S30104: Unfold the two-dimensional convolution kernel to form a second vector; S30105: Perform vector dot product operation between the multiple first vectors and the second vector respectively to obtain the first intermediate three-dimensional tensor.

4. The lightweight, dimensionally separable convolution method for FPGAs according to claim 1, characterized in that, In step S3, the convolution operation in the second and third dimensions includes: S311: Perform a one-dimensional convolution on the second dimension of each of the first three-dimensional feature tensors to obtain multiple second intermediate three-dimensional tensors; S312: Perform a one-dimensional convolution on the third dimension of each second intermediate three-dimensional tensor to obtain multiple third intermediate three-dimensional tensors; S313: Perform nonlinear activation on each third intermediate three-dimensional tensor to obtain multiple second three-dimensional tensors.

5. The lightweight, dimensionally separable convolution method for FPGAs according to claim 4, characterized in that, Performing a one-dimensional convolution on the second dimension of the first three-dimensional feature tensor yields a second intermediate three-dimensional tensor, including: S31101: Set a one-dimensional convolution kernel; S31102: Based on the first three-dimensional feature tensor, cache the second preset number of feature tensors in the on-chip row buffer structure of the FPGA; S31103: The feature tensors in the row buffer structure are divided using a sliding window to obtain multiple third vectors; S31104: Perform a vector dot product operation between each third vector and a one-dimensional convolution kernel to obtain multiple second intermediate three-dimensional tensors.

6. The lightweight, dimensionally separable convolution method for FPGAs according to claim 1, characterized in that, In step S4, the first-dimensional convolution operation includes: S401: Perform a one-dimensional convolution on the first dimension of each second three-dimensional tensor to obtain multiple fourth intermediate three-dimensional tensors; S402: Perform nonlinear activation operations on each fourth intermediate three-dimensional tensor to obtain multiple third three-dimensional tensors.

7. The lightweight, dimensionally separable convolution method for FPGAs according to claim 1, characterized in that, In step S5, the fourth-dimensional fusion operation includes: S501: Divide the first four-dimensional tensor into multiple fourth-dimensional vectors according to the fourth dimension; S502: Perform vector-matrix multiplication on each fourth-dimensional vector to obtain multiple fourth-dimensional vectors after multiplication. S503: Integrate multiple fourth-dimensional vectors that have undergone multiplication operations to obtain the first intermediate four-dimensional tensor; S504: Perform a nonlinear activation operation on the first intermediate four-dimensional tensor to obtain the final four-dimensional tensor.

8. The lightweight, dimensionally separable convolution method for FPGAs according to claim 7, characterized in that, The formula for vector-matrix multiplication is as follows: Indicates the first dimension. Indicates the second dimension. Indicates the third dimension. Represents the fourth-dimensional vector. This represents the preset weight matrix. This represents the preset bias matrix.

9. An FPGA-based convolution execution method, applied to the convolution method according to any one of claims 1 to 8, characterized in that, include: S01: Obtain the four-dimensional feature tensor to be convolved; The four-dimensional feature tensor to be convolved is written into the large-capacity block random access memory resource pool on the FPGA chip; S02: Construct a sliding window using the shift register logic on the FPGA chip, and use the sliding window to read data from the large-capacity block random access memory resource pool to obtain the read data; S03: Input the read data into the depth convolution calculation array on the FPGA chip for calculation to obtain convolutional data; S04: Input the convolutional data into the point convolution calculation engine on the FPGA chip to obtain the final four-dimensional tensor.

10. A lightweight, dimensionally separable convolution system for FPGAs, applied to the convolution method according to any one of claims 1 to 8, characterized in that, include: Tensor Acquisition Module: Acquires the four-dimensional feature tensor to be convolved; Tensor partitioning module: partitions the four-dimensional feature tensor to be convolved into multiple first three-dimensional feature tensors; First convolution module: Performs second-dimensional and third-dimensional convolution operations on each first three-dimensional feature tensor to obtain multiple second three-dimensional tensors; The second convolution module performs a first-dimensional convolution operation on each second three-dimensional tensor to obtain multiple third three-dimensional tensors; the multiple third three-dimensional tensors are merged to form a first four-dimensional tensor. Dimension fusion module: Performs a fourth-dimensional fusion operation on the first four-dimensional tensor to obtain the final four-dimensional tensor.

Images