Low-coupling transform target detection method and system based on FPGA acceleration

Abstract

This invention provides a loosely coupled Transformer object detection method and system based on FPGA acceleration, comprising: constructing a Transformer object detection network model based on a loosely coupled optimization method; and constructing a multi-framework general-purpose FPGA accelerator based on the Winograd fast algorithm to simultaneously support the convolutional neural network used for feature extraction and the Transformer codec based on a multi-head attention mechanism in the Transformer object detection network model. This invention also proposes an efficient memory access transformation scheme for special types of operators, enabling the accelerator to achieve high computational efficiency for all operators while obtaining higher convolution operation performance; for the various special functions included in the attention mechanism, a high-precision approximation scheme and corresponding data flow optimization are proposed, and a difference fitting method is proposed for GELU to improve approximation accuracy. Furthermore, the attention scaling operation is fused with the Softmax operator, saving runtime scaling operation overhead.

Description

Technical Field

[0001] This invention relates to the field of computer technology, specifically to a loosely coupled Transformer target detection method and system based on FPGA acceleration. More particularly, it relates to a method for loosely coupling optimization of the Transformer target detection algorithm. Background Technology

[0002] In recent years, deep learning technology has developed rapidly, and neural network models based on the Transformer framework have demonstrated outstanding performance in many fields, especially in object detection, a core task in computer vision. By introducing the Transformer framework, which can model global dependencies between features, the accuracy of object detection has been significantly improved, and algorithm design has also ushered in new directions. However, with the continuous increase in the demand for model accuracy, the complexity, computational cost, and scale of object detection algorithms have all increased significantly, posing a severe challenge to their deployment on edge devices.

[0003] Currently, the forward inference process of neural networks typically runs on hardware platforms such as Graphics Processing Units (GPUs), Central Processing Units (CPUs), Application-Specific Integrated Circuits (ASICs), or Field-Programmable Gate Arrays (FPGAs). Among these, GPUs possess a large number of parallel computing units and efficient floating-point matrix operation capabilities, offering advantages in high throughput and high memory bandwidth, but their power consumption is relatively high. While CPUs are highly versatile, their number of computing units is limited, and their control logic is complex, resulting in lower efficiency for neural network inference. ASICs can achieve performance improvements or power reductions through dedicated circuit optimization, but their high development costs, poor flexibility, and long development cycles limit their widespread application. In contrast, FPGAs have significant advantages in power consumption, parallelism, and flexibility, while also having a shorter development cycle, making them the preferred hardware platform for accelerating lightweight Transformer / CNN model inference.

[0004] Current mainstream Transformer object detection frameworks, such as the DETR series, mostly employ an architecture combining a CNN backbone network with a Transformer encoder-decoder. While this architecture improves performance, it also complicates the coupling between layers in the network, making it difficult to deploy the algorithm model on resource-constrained edge devices. Furthermore, some existing Transformer accelerator architectures only optimize for operators such as matrix multiplication, while offloading nonlinear operators, normalization operators, and other operations to other platforms. This architecture struggles to achieve complete hardware acceleration. Therefore, hardware accelerator design for Transformer object detection frameworks needs to optimize the Generalized Matrix Multiplication (GEMM) operator while also considering the hardware acceleration requirements for nonlinear operators, normalization operators, and convolutional operators in feature extraction networks to achieve complete forward inference acceleration capabilities.

[0005] Patent document CN111459877A discloses a Winograd YOLOv2 target detection model method based on FPGA acceleration. It uses a PYNQ board, and the main control chip of the PYNQ board includes a processing system terminal (PS) and a programmable logic terminal (PL). The PS caches the YOLO model and feature map data of the image to be detected. The PL caches the parameters of the YOLO model and the image to be detected in the on-chip RAM, deploys a YOLO accelerator with the Winograd algorithm, completes the accelerated calculation of the model, forms the data path of the hardware accelerator, and realizes target detection of the image to be detected. It can also read out the calculation results of the acceleration circuit and perform image preprocessing and display.

[0006] However, patent document CN111459877A cannot be applied to Transformer object detection. Transformer object detection involves numerous matrix multiplications and matrix transpose operations, as well as complex nonlinear layers such as Softmax and LayerNorm layers. The FPGA accelerator in this patent document only supports convolution operations and cannot support the aforementioned Transformer-related operations (matrix multiplication and nonlinear layers). In other words, the types of operators supported by this patent document are relatively limited. It can support convolution operators in models with fewer operator types, such as YOLOv2, but for Transformer object detection models containing a large number of complex operators, a significant portion of the computation cannot be supported.

[0007] In summary, from the perspective of algorithm and hardware co-design, this study investigates a target detection algorithm model and its hardware implementation scheme that combines high efficiency and ease of deployment through algorithm simplification and FPGA-based hardware accelerator design. This has significant theoretical value and application prospects. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the purpose of this invention is to provide a loosely coupled Transformer target detection method and system based on FPGA acceleration.

[0009] The present invention provides a loosely coupled Transformer target detection method based on FPGA acceleration, comprising:

[0010] Step S1: Construct a Transformer object detection network model based on a low-coupling optimization method;

[0011] Step S2: Construct a multi-framework general-purpose FPGA accelerator based on the Winograd fast algorithm to simultaneously support the convolutional neural network used for feature extraction and the Transformer codec based on the multi-head attention mechanism in the Transformer object detection network model.

[0012] Preferably, the target detection network model includes a first target detection model and a second target detection model;

[0013] The first target detection model includes a feature fusion module for low-coupling optimization and a low-coupling encoder structure. The feature fusion module uses multi-gradient feature concatenation, introduces reparameterization operations into the convolutional structure, and uses a single 3×3 convolution during inference. The low-coupling encoder structure includes a trainable DECONV4 deconvolution operator. The DECONV4 deconvolution operator is inserted into the feature recovery path, and there are two aggregation paths to obtain the optimized feature fusion module in the low-coupling encoder.

[0014] The second target detection model is a simplified model with a straight-through structure and no lateral path. It only uses deconvolution layers as upsampling paths and directly concatenates the output of the Transformer encoder with the result of the upsampling path as the decoder input.

[0015] Preferably, the FPGA accelerator includes a configurable multi-type computationally intensive operator operation module, an input buffer module, a weight buffer module, a layer bias buffer module, an accumulation module, a data rearrangement module, an overall main control unit, and a special function processing unit (SFU) for computing nonlinear operators in the attention mechanism;

[0016] The FPGA accelerator can process both matrix multiplication and multi-type convolution, two types of computationally intensive operators, contained in the Transformer object detection network model. It performs memory access processing for each operator in the input cache module and weight cache module to generate a unified data stream and send it to the configurable computing array to maximize the utilization of the configurable computing units. In the output accumulation part, it also needs to process different data streams based on different operator types.

[0017] Preferably, generating a unified data stream includes: dividing the left-multiplied matrix into rows and blocks to transform them into input channels of the input feature map; dividing the right-multiplied matrix into columns and blocks to transform them into input channels of the convolution kernel; traversing the column direction of the output matrix to transform it into traversing the pixels of the input feature map; and transforming each row and column of the output matrix into the direction of the output channel.

[0018] Preferably, the configurable multi-type computationally intensive operator operation module includes multiple configurable processing units (CPEs).

[0019] The configurable processing unit includes an input transformation module, a weight transformation module, and a dot product and output transformation module, which correspond to the Winograd domain transformation of input data, the Winograd domain transformation of weight data, and the Winograd domain transformation of dot product results in different convolution types, respectively.

[0020] Preferably, the configurable processing units constitute a configurable processing unit group and a configurable computing array through the parallelism of the output channel dimension and the input channel dimension;

[0021] The configurable computing array has three computing modes: CONV2 mode, GEMM / CONV1 mode, and CONV3 mode.

[0022] Preferably, the input cache module, weight cache module, and accumulation cache module all adopt a ping-pong cache structure;

[0023] The input buffer is involved in the main memory access control, and completes the correct memory access operation and the generation of the computing array configuration signal based on different output pixel parallelism.

[0024] The accumulation cache module integrates a residual cache, a bias cache, quantization and dequantization units, an activation module with multiple activations, an accumulation logic unit, and an accumulation control unit.

[0025] Preferably, the special function processing unit (SFU) includes a ReLU section, a GeLU section, and a SiLU section;

[0026] For the ReLU part, activation is achieved directly through a selector;

[0027] For the GeLU part, the difference fitting method based on piecewise linear fitting makes the piecewise error as small as the error magnitude of direct fitting.

[0028] For the SiLU part, a direct fitting method is used for nonlinear calculation. The range of the input data is determined by range decoding, and the corresponding slope and intercept are found by k-LUT and b-LUT according to the range, and then the multiplication and addition operation is completed to obtain the output.

[0029] Preferably, the special function calculation unit further includes a multi-stage pipelined Softmax calculation module and a two-stage LayerNorm calculation module;

[0030] In the multi-stage pipeline Softmax calculation module, the scaling operation of attention weights is integrated. The scaling operation is decomposed so that part of it becomes a shift operation and part of it is integrated into the constant operation based on the selector.

[0031] Within a single stage, the multi-stage pipelined Softmax dataflow traverses the entire sequence based on fine-grained pipeline, while between multiple stages, it distributes multiple cached sequences to the corresponding computation modules of each stage based on coarse-grained pipeline, so that the operations between different pipeline stages overlap.

[0032] A loosely coupled Transformer target detection system based on FPGA acceleration according to the present invention includes:

[0033] Step M1: Construct a Transformer object detection network model based on a loosely coupled optimization method;

[0034] Step M2: Construct a multi-framework general-purpose FPGA accelerator based on the Winograd fast algorithm to simultaneously support the convolutional neural network used for feature extraction and the Transformer codec based on the multi-head attention mechanism in the Transformer object detection network model.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. This invention constructs a multi-framework general-purpose FPGA accelerator based on the Winograd fast algorithm. It unifies the data flow of two major types of computationally intensive operators, namely convolution and matrix multiplication, under the Winograd domain and proposes an efficient padding strategy that is adapted to them. At the same time, it proposes an efficient memory access transformation scheme for special types of operators, so that the accelerator has high computational efficiency for all operators and achieves higher convolution operation performance.

[0037] 2. This invention proposes a high-precision approximation scheme and corresponding data flow optimization for the various special functions contained in the attention mechanism, and proposes a difference fitting method for GeLU to improve the approximation accuracy. At the same time, it integrates the attention scaling operation with the Softmax operator to save the scaling operation overhead at runtime.

[0038] 3. The FPGA hardware accelerator of the present invention has finely divided the ping-pong buffer, which can realize effective ping-pong control. By caching most of the accumulated data on the chip, the data transfer time is greatly reduced, and it conforms to the processing mode of special functions.

[0039] 4. The hardware accelerator of this invention for the Transformer object detection framework needs to optimize the general matrix multiplication (GEMM) operator while taking into account the hardware acceleration requirements of nonlinear operators, normalization operators and convolution operators in the feature extraction network, and has stronger inference capabilities. Attached Figure Description

[0040] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0041] Figure 1 Schematic diagrams of two types of loosely coupled network structures based on the DETR framework;

[0042] Figure 2 A schematic diagram of the overall structure of a multi-framework general-purpose FPGA accelerator for DETR;

[0043] Figure 3 A unified schematic diagram of the data flow for matrix multiplication and convolution;

[0044] Figure 4 This is a schematic diagram of a configurable processing unit structure based on Winograd;

[0045] Figure 5 A schematic diagram of a configurable processing unit group and a configurable computing array structure;

[0046] Figure 6 This is a schematic diagram illustrating the processing of two special operators, CONV3_2 and DECONV4.

[0047] Figure 7 A schematic diagram illustrating the memory access process for different types of operators;

[0048] Figure 8 Example diagram of the filling process in the column direction;

[0049] Figure 9 This is a schematic diagram of the multi-type activation function calculation module structure;

[0050] Figure 10 This is a schematic diagram of a multi-stage pipelined Softmax data flow. Detailed Implementation

[0051] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0052] This invention, starting from a hardware-software co-design approach, provides an FPGA-based hardware accelerator implementation method for the DETR framework. Through low-coupling optimization algorithm design and combined with the Winograd fast algorithm, this invention proposes an efficient and concise Transformer object detection network model and hardware accelerator, solving the acceleration and deployment problems of various operators in the DETR framework. This accelerator is a general-purpose hardware accelerator that supports the DETR framework and is compatible with multiple framework structures. It achieves FPGA acceleration of the low-coupling Transformer object detection algorithm, optimizes convolution operation performance, reduces computational overhead, and ensures high detection accuracy and real-time detection speed. In the software design, this invention provides a low-coupling optimized Transformer object detection network model, employing a CSP-Rep Fusion feature fusion module and a low-coupling encoder structure, reducing computational requirements and improving inference efficiency. Furthermore, this invention provides a low-coupling model with a through-structure, further reducing coupling while maintaining detection performance. In the hardware design, based on the Winograd fast algorithm, this invention provides a multi-framework general-purpose FPGA accelerator, improving hardware acceleration performance by processing different types of convolution and matrix multiplication operators through a unified data flow. For special functions such as GeLU, Softmax, and LayerNorm, high-precision approximation methods and multi-stage pipelined computation modules are provided to optimize computational efficiency. The FPGA accelerator adopts a ping-pong cache structure, supports memory access transformations for different operator types, optimizes padding and memory access operations, and decomposes scaling operations in the attention mechanism, improving hardware utilization and acceleration efficiency. Based on the above, this FPGA-accelerated, loosely coupled Transformer object detection method can be flexibly applied to FPGA acceleration scenarios for the DETR framework, and the loosely coupled optimization method and its structure can be used to meet the simplified model requirements in the field of edge computing.

[0053] Example 1

[0054] The present invention provides a loosely coupled Transformer target detection method based on FPGA acceleration, comprising:

[0055] Step S1: Construct a Transformer object detection network model based on a loosely coupled optimization method. For example... Figure 1 As shown, this includes two loosely coupled network structures based on the DETR framework: a first object detection model and a second object detection model. Figure 1As shown on the left, the first object detection model includes a feature fusion module and a low-coupling encoder structure for low-coupling optimization. The CSP-Rep Fusion feature fusion module employs multi-gradient feature concatenation, enabling more efficient feature aggregation and distributing the computational load evenly across each layer, significantly reducing computational requirements. Simultaneously, the convolutional structure introduces reparameterization operations, decoupling the inference and training processes. During inference, the convolutional structure is a single 3×3 convolution, reducing deployment difficulty while improving algorithm inference speed. The low-coupling encoder structure introduces a trainable DECONV4 deconvolution operator and inserts it into the feature recovery path, reducing the aggregation paths from three to two, resulting in the optimized LC-PAN feature fusion module in the low-coupling encoder, reducing feature aggregation to two times. The introduced learnable upsampling operator helps recover more accurate feature and localization information. Furthermore, discarding shallow feature aggregation paths significantly reduces the length of the input sequence sent to the decoder, further improving inference efficiency.

[0056] The CSP-Rep Fusion feature fusion module achieves more efficient feature aggregation through multi-gradient feature concatenation and distributes computational load evenly, significantly reducing computational requirements. Reparameterization operations are introduced into convolutions to decouple inference from training; inference is performed using a single 3×3 convolution, improving speed and reducing deployment complexity. The loosely coupled encoder structure introduces a trainable DECONV4 deconvolution operator and simplifies the aggregation path to two, further reducing the number of feature aggregations in the optimized LC-PAN feature fusion module. The introduced upsampling operator helps to accurately recover feature information, and discarding shallow feature aggregation paths further improves inference efficiency.

[0057] In addition, such as Figure 1 As shown on the right, the present invention also includes a second target detection model. The second target detection model is a simplified model with a straight-through structure and no lateral path. It only uses deconvolution layers as upsampling paths, which achieves more accurate feature recovery. It also fully considers that the low-resolution feature maps after intra-scale interaction contain richer semantic information. Based on the introduction of upsampling paths, it proposes to use the sequence input characteristics of the Transformer decoder to directly concatenate the encoder output with the result of the upsampling path as the decoder input, thereby obtaining a straight-through structure algorithm model with lower coupling and guaranteed detection performance.

[0058] Using only deconvolutional layers as the upsampling path, more accurate feature recovery is obtained. Considering that low-resolution feature maps contain rich semantic information, we propose directly concatenating the Transformer encoder output and the upsampling result into the decoder, further reducing coupling while ensuring detection performance.

[0059] Step S2: Construct a multi-framework general-purpose FPGA accelerator based on the Winograd fast algorithm to simultaneously support the convolutional neural network used for feature extraction and the Transformer codec based on the multi-head attention mechanism in the Transformer object detection network model. For example... Figure 2 As shown, the accelerator mainly includes a configurable multi-type computationally intensive operator operation module, an input cache module, a weight cache module, a layer bias cache module, an accumulation module, a data rearrangement module, an overall main control unit, and a Special Function Unit (SFU) for computing nonlinear operators in the attention mechanism. It can process both matrix multiplication and multi-type convolution, two types of computationally intensive operators simultaneously included in the Transformer object detection network model. Memory access processing for each operator is performed in the input cache module and the weight cache module to generate a unified data stream sent to the configurable computing array, maximizing the utilization of the configurable computing unit. The output accumulation part also needs to process different data streams based on different operator types. To achieve resource reuse and efficient mapping of different operators, this invention is based on... Figure 3 The transformation process shown unifies the data flow of the two main types of operators: left multiplication of the matrix by row blocks transforms into input channels of the input feature map; right multiplication of the matrix by column blocks transforms into input channels of the convolution kernel; traversal of the output matrix by column direction transforms into traversal of the input feature map pixels; and each row and column of the output matrix transforms into the direction of the output channels. Therefore, matrix multiplication can be transformed into 1×1 convolution during the data arrangement stage, and matrix block division can be achieved through operations such as input channel traversal and input feature map block division. Thus, the unification of the data flow of multiple types of operators is transformed into the unification of the data flow of multiple types of convolution.

[0060] To further achieve data flow unification across multiple convolution types, this invention provides, based on the Winograd fast algorithm, such as... Figure 4 The Configurable Process Element (CPE) shown includes an input transformation module, a weight transformation module, and a dot product and output transformation module. These correspond to the Winograd domain transformations of the input data, the weight data, and the dot product result in different convolution types, respectively. According to the Winograd fast algorithm, the CPE's configuration transformation operation requires an input transformation matrix B. T The convolution kernel weight transformation matrix G and the output transformation matrix A TThe system transforms the input data, weights, and outputs to the Winograd domain. The input transformation module, weight transformation module, dot product and output transformation module are implemented by adding adders and selectors. In the Winograd domain, matrix multiplication, 1×1 convolution, 2×2 convolution and 3×3 convolution are unified in data flow, so that the computing array can flexibly calculate different types of operators based on the operation mode control signal.

[0061] Furthermore, this invention introduces parallelism in both the output channel dimension and the input channel dimension, providing, for example... Figure 5 The configurable processing unit group and configurable computing array are shown. Figure 5 In the diagram, CPE1, CPE2, ..., CPEpout represent several configurable processing units, which together form a CPEGroup. A CPEGroup extracts and integrates the input data transformation modules of multiple CPEs into one, and the input data transformation results are broadcast and shared. A single input value vector d is shared by Pout CPEs, and each Pout weight data transformation module receives a weight value vector from the Pout group, achieving parallel processing of Pout along the output channel dimension. In actual operation, a single input value vector d corresponds to the four input feature data of a certain row in the input channel, and the Pout group weight value vector corresponds to the weight data of different convolutional kernels at the corresponding input channel positions, thus obtaining the intermediate convolution results corresponding to Pout channels on the output feature map. These channels are independent of each other, so the parallel output results in a single CPE Group do not need to be accumulated. The structure of the configurable computing array includes multiple CPEGroups and an accumulator. In the diagram, d1 to dPin represent the input data of Pin channels, g11 to gPinPout represent the calculated weights, and r0, r1, ..., rPout represent the output data of Pout channels. Each CPE Group's output value vector corresponds to four sets of intermediate output results. Accumulation requires a four-parallel pipelined addition tree structure; therefore, the total number of pipelined addition trees in the multi-dimensional parallel computing array is 4 × Pin. The configurable computing array's computing modes include CONV2 mode, GEMM / CONV1 mode, and CONV3 mode.

[0062] For two special types of operators—downsampling convolution with a stride of 2 and 4×4 deconvolution with a stride of 1—this invention provides, as follows: Figure 6 The targeted processing is shown below. Taking a 3×3 convolution with a stride of 2 as an example, as follows... Figure 6On the upper side, the odd-to-even pixels of the downsampling convolution are split, and the CONV2 mode and GEMM / CONV1 mode in the configurable computation array are used for split-accumulation operations. Data flow unification is achieved through memory access transformation. For a 5×5 convolution with a stride of 2, the CONV3 mode and CONV2 mode can be split and accumulated to complete the operator transformation in a similar way. For a 4×4 deconvolution with a stride of 1, this invention is based on Figure 6 The method in the lower part breaks down the 4×4 deconvolution process into four 2×2 convolutions, which avoids all invalid zero value calculations and eliminates the zero-insertion operation. At the same time, it can realize array reuse and improve hardware efficiency.

[0063] The FPGA hardware accelerator of this invention includes an input buffer module, a weight buffer module, and an accumulation buffer module, all of which adopt a ping-pong buffer structure. The input buffer is involved in the main memory access control; the memory access process for different types of operators is as follows... Figure 7 As shown, the correct memory access operations and the generation of the computation array configuration signal can be completed based on different output pixel parallelisms. Furthermore, the input buffer has corresponding memory access optimizations for padding; zero values ​​in an entire row can be skipped on a row-by-row basis, while zero values ​​in the column direction can be skipped... Figure 8 The filling method simplifies the memory access and filling process. The accumulation cache module integrates residual cache, bias cache, quantization and dequantization units, activation module with multiple activation types, accumulation logic unit, and accumulation control unit.

[0064] The FPGA hardware accelerator of this invention can compute various types of operators that cannot be computed by comparison files by flexibly setting multiple computing modes. In addition, during the data transfer process, this invention can support input feature maps of arbitrary size based on configuration information and perform corresponding large-size input feature map block division, which has better adaptability. At the same time, the block division process can also be fully adapted to the pipeline design of this invention.

[0065] The FPGA-accelerated, loosely coupled Transformer target detection method of this invention also includes a Transformer special function calculation unit, such as... Figure 9 As shown, the Transformer special function calculation unit includes a ReLU part, a GeLU part, and a SiLU part. For ReLU, activation is directly achieved through a selector. For GeLU, a difference fitting method based on piecewise linear fitting is used, so that the piecewise error can reach the same magnitude as direct fitting, requiring only about half the number of segments. Figure 9The GeLU part reuses the ReLU activation result and calculates the difference between the ReLU and the piecewise linear fitting result to obtain an approximate value for GeLU. This difference has an even function property, and the piecewise linear fitting process fully utilizes this characteristic to achieve higher fitting accuracy. For the SiLU part, a direct fitting method is used for nonlinear calculation. The range of the input data is determined by range decoding, and the corresponding slope and intercept are found using k-LUT and b-LUT based on the range, thus completing the multiplication and addition operations to obtain the output.

[0066] The Transformer special function computation unit also includes a multi-stage pipelined Softmax computation module and a two-stage LayerNorm computation module. In the multi-stage pipelined Softmax computation module, scaling operations with attention weights are integrated. These scaling operations are decomposed, with some becoming hardware-friendly shift operations and others integrated into selector-based constant operations, thus saving the overhead of scaling operations during computation. Specifically, the scaling operations with attention weights are integrated, and the scaling operations are decomposed into two... s And the scaling factor may contain 2 0.5 Or 1.2 s In hardware, this can be directly achieved through a right shift operation. This invention integrates the latter's calculation process into the approximation process of the exp exponent, through a transformation process as shown in formula (1):

[0067]

[0068] Simultaneously, the subsequent calculations of the scaling operation and the process of calculating the exp exponent are completed. The domain of the input x is... Chinese x i The range corresponding to -max(x), It can then be obtained directly by multiplying the input fixed-point number by the fixed-point constant configured based on the square root, as shown in formula (2):

[0069]

[0070] Taking the Softmax calculation module as an example, the multi-stage pipelined Softmax data flow is as follows: Figure 10 As shown, the Softmax computation process has four stages, including data loading. In this invention, the multi-stage pipelined Softmax data flow traverses the entire sequence within a single stage based on fine-grained pipeline, while between multiple stages, multiple cache sequences are distributed to the corresponding computation modules of each stage based on coarse-grained pipeline, so that the operations between different pipeline stages overlap, thereby improving hardware utilization and acceleration efficiency.

[0071] This invention aims to address the hardware-software coordination problem in existing technologies where various operators within the DETR framework cannot be effectively accelerated and efficiently deployed. It provides an efficient and easily deployable object detection algorithm model and a general-purpose hardware accelerator that supports DETR framework deployment mapping and is compatible with multiple framework structures, achieving FPGA acceleration of a loosely coupled Transformer object detection algorithm. Through a loosely coupled optimization algorithm design, an efficient and simplified network model is provided, exhibiting high detection accuracy and real-time detection speed. Based on the Winograd fast algorithm, a general-purpose FPGA accelerator is provided, improving convolution operation performance. An efficient unified data flow strategy is designed to address the acceleration requirements of different operators. Simultaneously, for special functions in the Transformer, high-precision approximation and data flow optimization schemes are provided, reducing computational overhead.

[0072] Example 2

[0073] The present invention also provides an FPGA-accelerated low-coupling Transformer target detection system. The FPGA-accelerated low-coupling Transformer target detection system can be implemented by executing the process steps of the FPGA-accelerated low-coupling Transformer target detection method. That is, those skilled in the art can understand the FPGA-accelerated low-coupling Transformer target detection method as a preferred embodiment of the FPGA-accelerated low-coupling Transformer target detection system.

[0074] A loosely coupled Transformer target detection system based on FPGA acceleration according to the present invention includes:

[0075] Module M1: Constructs a Transformer object detection network model based on a low-coupling optimization method. The object detection network model includes a first object detection model and a second object detection model. The first object detection model includes a feature fusion module and a low-coupling encoder structure for low-coupling optimization. The feature fusion module uses multi-gradient feature concatenation, introduces reparameterization operations into the convolutional structure, and uses a single 3×3 convolution during inference. The low-coupling encoder structure includes a trainable DECONV4 deconvolution operator. The DECONV4 deconvolution operator is inserted into the feature recovery path, resulting in two aggregation paths, thus obtaining the optimized feature fusion module in the low-coupling encoder. The second object detection model is a simplified model with a direct-pass structure and no lateral paths. It only uses deconvolution layers as upsampling paths, and directly concatenates the output of the Transformer encoder with the result of the upsampling path as the decoder input.

[0076] Module M2: Constructs a multi-framework general-purpose FPGA accelerator based on the Winograd fast algorithm, used to simultaneously support the convolutional neural network for feature extraction and the Transformer codec based on the multi-head attention mechanism in the Transformer object detection network model. The FPGA accelerator includes configurable multi-type computationally intensive operator operation modules, an input buffer module, a weight buffer module, a layer bias buffer module, an accumulation module, a data rearrangement module, an overall main control unit, and a special function processing unit (SFU) for computing nonlinear operators in the attention mechanism. The input buffer module, weight buffer module, and accumulation buffer module all adopt a ping-pong buffer structure. The input buffer involves the main memory access control, completing the correct memory access operation and computation array configuration signal generation operation based on different output pixel parallelism. The accumulation buffer module integrates a residual buffer, a bias buffer, quantization and dequantization units, an activation module with multiple activations, an accumulation logic unit, and an accumulation control unit. The FPGA accelerator can process both matrix multiplication and multi-type convolution, two computationally intensive operators, simultaneously included in the Transformer object detection network model. It performs memory access processing for each operator in the input buffer module and weight buffer module to generate a unified data stream sent to the configurable computing array, maximizing the utilization of the configurable computing units. The output accumulation section also processes different data streams based on different operator types. Generating a unified data stream includes: transforming left-multiplication matrix row blocks into input channels of the input feature map, right-multiplication matrix column blocks into input channels of the convolution kernel, transforming column traversal of the output matrix into pixel traversal of the input feature map, and transforming each row and column of the output matrix into the output channel direction. The configurable multi-type computationally intensive operator operation module includes multiple configurable processing units (CPEs). Each configurable processing unit includes an input transformation module, a weight transformation module, and a dot product and output transformation module, corresponding to the Winograd domain transformation of input data, the Winograd domain transformation of weight data, and the Winograd domain transformation of the dot product result in different convolution types, respectively. Configurable processing units, through their parallelism in the output and input channel dimensions, constitute configurable processing unit groups and configurable computing arrays. The computing modes of the configurable computing array include CONV2 mode, GEMM / CONV1 mode, and CONV3 mode.

[0077] The Special Function Processing Unit (SFU) includes ReLU, GeLU, and SiLU parts. For the ReLU part, activation is directly achieved through a selector. For the GeLU part, a piecewise linear fitting difference fitting method is used to minimize the piecewise error to the same magnitude as direct fitting. For the SiLU part, a direct fitting method is used for nonlinear calculation. The range of the input data is determined by range decoding, and the corresponding slope and intercept are found using k-LUT and b-LUT based on the range, thus performing multiplication and addition operations to obtain the output. The Special Function Calculation Unit also includes a multi-stage pipelined Softmax calculation module and a two-stage LayerNorm calculation module. In the multi-stage pipelined Softmax calculation module, attention weight scaling operations are integrated. The scaling operations are decomposed, with one part becoming a shift operation and the other part fused into a selector-based constant operation. Within a single stage, the multi-stage pipelined Softmax dataflow traverses the entire sequence based on fine-grained pipeline, while between multiple stages, it distributes multiple cached sequences to the corresponding computation modules of each stage based on coarse-grained pipeline, so that the operations between different pipeline stages overlap.

[0078] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0079] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A loosely coupled Transformer target detection method based on FPGA acceleration, characterized in that, include: Step S1: Construct a Transformer object detection network model based on a low-coupling optimization method; Step S2: Construct a multi-framework general-purpose FPGA accelerator based on the Winograd fast algorithm to simultaneously support the convolutional neural network used for feature extraction and the Transformer codec based on the multi-head attention mechanism in the Transformer object detection network model; The target detection network model includes a first target detection model and a second target detection model; The first target detection model includes a feature fusion module for low-coupling optimization and a low-coupling encoder structure. The feature fusion module uses multi-gradient feature concatenation, and the convolutional structure introduces reparameterization operations. During inference, the convolutional structure is a single 3×3 convolution. The low-coupling encoder structure includes a trainable DECONV4 deconvolution operator. The DECONV4 deconvolution operator is inserted into the feature recovery path, and two aggregation paths are formed to obtain the optimized feature fusion module in the low-coupling encoder. The second target detection model is a simplified model with a straight-through structure and no lateral path. It only uses deconvolution layers as upsampling paths and directly concatenates the output of the Transformer encoder with the result of the upsampling path as the decoder input. The targets of the detection are defects on the surface of products, ships on the sea, or moving objects in the monitored area; The FPGA accelerator includes a configurable multi-type computationally intensive operator operation module, an input cache module, a weight cache module, a layer bias cache module, an accumulation module, a data rearrangement module, an overall main control unit, and a special function processing unit (SFU) for computing nonlinear operators in the attention mechanism. The FPGA accelerator can process both matrix multiplication and multi-type convolution, two types of computationally intensive operators, contained in the Transformer object detection network model. It performs memory access processing for each operator in the input cache module and weight cache module to generate a unified data stream and send it to the configurable computing array to maximize the utilization of the configurable computing units. In the output accumulation part, it also needs to process different data streams based on different operator types.

2. The FPGA-accelerated, loosely coupled Transformer target detection method according to claim 1, characterized in that, The generation of a unified data stream includes: dividing the left-multiplied matrix into rows and blocks to transform them into input channels of the input feature map; dividing the right-multiplied matrix into columns and blocks to transform them into input channels of the convolution kernel; traversing the column direction of the output matrix to transform it into traversing the pixels of the input feature map; and transforming each row and column of the output matrix into the direction of the output channel.

3. The FPGA-accelerated, loosely coupled Transformer target detection method according to claim 1, characterized in that, The configurable multi-type computationally intensive operator operation module includes multiple configurable processing units (CPEs); The configurable processing unit includes an input transformation module, a weight transformation module, and a dot product and output transformation module, which correspond to the Winograd domain transformation of input data, the Winograd domain transformation of weight data, and the Winograd domain transformation of dot product results in different convolution types, respectively.

4. The FPGA-accelerated, loosely coupled Transformer target detection method according to claim 3, characterized in that, Configurable processing units can form configurable processing unit groups and configurable computing arrays through the parallelism of output channel dimension and input channel dimension; The configurable computing array has three computing modes: CONV2 mode, GEMM / CONV1 mode, and CONV3 mode.

5. The FPGA-accelerated, loosely coupled Transformer target detection method according to claim 1, characterized in that, The input cache module, weight cache module, and accumulation cache module all adopt a ping-pong cache structure. The input buffer is involved in the main memory access control, and completes the correct memory access operation and the generation of the computing array configuration signal based on different output pixel parallelism. The accumulation cache module integrates a residual cache, a bias cache, quantization and dequantization units, an activation module with multiple activations, an accumulation logic unit, and an accumulation control unit.

6. The FPGA-accelerated, loosely coupled Transformer target detection method according to claim 1, characterized in that, The Special Function Processing Unit (SFU) includes a ReLU section, a GeLU section, and a SiLU section. For the ReLU part, activation is achieved directly through a selector; For the GeLU part, the difference fitting method based on piecewise linear fitting makes the piecewise error as small as the error magnitude of direct fitting. For the SiLU part, a direct fitting method is used for nonlinear calculation. The range of the input data is determined by range decoding, and the corresponding slope and intercept are found by k-LUT and b-LUT according to the range, and then the multiplication and addition operation is completed to obtain the output.

7. The FPGA-accelerated, loosely coupled Transformer target detection method according to claim 6, characterized in that, The special function calculation unit also includes a multi-stage pipelined Softmax calculation module and a two-stage LayerNorm calculation module; In the multi-stage pipeline Softmax calculation module, the scaling operation of attention weights is integrated. The scaling operation is decomposed so that part of it becomes a shift operation and part of it is integrated into the constant operation based on the selector. Within a single stage, the multi-stage pipelined Softmax dataflow traverses the entire sequence based on fine-grained pipeline, while between multiple stages, it distributes multiple cached sequences to the corresponding computation modules of each stage based on coarse-grained pipeline, so that the operations between different pipeline stages overlap.

8. A loosely coupled Transformer target detection system based on FPGA acceleration, characterized in that, include: Step M1: Construct a Transformer object detection network model based on a loosely coupled optimization method; Step M2: Construct a multi-framework general-purpose FPGA accelerator based on the Winograd fast algorithm to simultaneously support the convolutional neural network used for feature extraction and the Transformer codec based on the multi-head attention mechanism in the Transformer object detection network model; The target detection network model includes a first target detection model and a second target detection model; The first target detection model includes a feature fusion module for low-coupling optimization and a low-coupling encoder structure. The feature fusion module uses multi-gradient feature concatenation, and the convolutional structure introduces reparameterization operations. During inference, the convolutional structure is a single 3×3 convolution. The low-coupling encoder structure includes a trainable DECONV4 deconvolution operator. The DECONV4 deconvolution operator is inserted into the feature recovery path, and two aggregation paths are formed to obtain the optimized feature fusion module in the low-coupling encoder. The second target detection model is a simplified model with a straight-through structure and no lateral path. It only uses deconvolution layers as upsampling paths and directly concatenates the output of the Transformer encoder with the result of the upsampling path as the decoder input. The targets of the detection are defects on the surface of products, ships on the sea, or moving objects in the monitored area; The FPGA accelerator includes a configurable multi-type computationally intensive operator operation module, an input cache module, a weight cache module, a layer bias cache module, an accumulation module, a data rearrangement module, an overall main control unit, and a special function processing unit (SFU) for computing nonlinear operators in the attention mechanism. The FPGA accelerator can process both matrix multiplication and multi-type convolution, two types of computationally intensive operators, contained in the Transformer object detection network model. It performs memory access processing for each operator in the input cache module and weight cache module to generate a unified data stream and send it to the configurable computing array to maximize the utilization of the configurable computing units. In the output accumulation part, it also needs to process different data streams based on different operator types.

Images