An implementation method of an artificial intelligence algorithm for AI on an FPGA

Abstract

The application discloses an implementation method of an artificial intelligence algorithm for AI on an FPGA, and comprises the following steps: analyzing the artificial intelligence algorithm and adapting FPGA resource characteristics; performing format conversion, normalization and block preprocessing on input data; optimizing key modules such as a convolution layer, a pooling layer and a full connection layer, wherein a convolution calculation unit is constructed by using LUT and DSP, and an adaptive LUT structure is adopted to dynamically adjust logic, the pooling layer adopts a comparison network hardware structure, and the full connection layer optimizes weight storage and multiply-add operation; a plurality of parallel calculation units are constructed on the FPGA, and a double-buffer queue and a pipeline technology are adopted to perform task scheduling and parallel calculation; and finally, intermediate results are merged and post-processing output is performed. The application fully combines the parallelism and reconfigurable characteristics of the FPGA, significantly improves the calculation efficiency of the AI algorithm, reduces power consumption, and optimizes the FPGA resource utilization rate.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence (AI) and field-programmable gate array (FPGA) technology, and specifically to a method for implementing AI-oriented artificial intelligence algorithms on an FPGA. Background Technology

[0002] With the rapid development of artificial intelligence technology, deep learning algorithms, especially convolutional neural networks, have made breakthroughs in fields such as image classification, object detection, and speech recognition. However, these algorithms typically require a large amount of computing resources and storage bandwidth, making efficient inference on resource-constrained embedded platforms a key focus of the industry.

[0003] Existing AI algorithm deployment solutions mainly include four technical routes: CPU general-purpose computing, GPU parallel acceleration, ASIC embedded implementation, and FPGA reconfigurable computing. CPU solutions offer strong versatility but relatively low computational efficiency; GPU solutions offer high parallel processing capabilities but consume a lot of power and have uncontrollable latency; ASIC solutions offer excellent performance and power consumption but have long development cycles and lack flexibility. FPGAs, combining parallel computing capabilities, low power consumption, and programmable flexibility, are gradually becoming an important choice for embedded deployment of AI algorithms.

[0004] However, existing FPGA implementations generally suffer from the following problems: excessive resource consumption of general-purpose multipliers; lack of efficient collaboration between data preprocessing and core computing modules; low utilization of on-chip storage resources; and the need for redesign to adapt to different convolution kernel sizes and accuracies. Therefore, there is an urgent need for a method that can efficiently and flexibly implement AI algorithms on FPGAs. Summary of the Invention

[0005] The purpose of this invention is to provide a method for implementing AI-oriented artificial intelligence algorithms on FPGAs. By adopting an adaptive LUT structure multiplication unit, a double-buffered data scheduling mechanism, and a pipelined parallel computing architecture, it effectively solves the problems of high resource consumption, low computing efficiency, and insufficient versatility in the existing technology.

[0006] Compared to traditional implementations based on DSP hard-core multipliers, the LUT-based multiplication matrix structure proposed in this invention can significantly reduce the occupation of limited DSP resources, so that the size of the multiplication matrix is ​​no longer limited by the number of on-chip DSPs. At the same time, the adaptive LUT configuration mechanism allows the same hardware architecture to be adapted to different convolution kernel sizes and data precisions through software configuration, which significantly improves the versatility and portability of the FPGA implementation scheme.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A method for implementing an AI-oriented artificial intelligence algorithm on an FPGA includes the following steps:

[0009] Algorithm analysis and adaptation steps: Conduct a detailed analysis of the AI-oriented artificial intelligence algorithm to identify the key computing modules and their computing requirements, including data types, data volume, and computational complexity; Based on the resource characteristics of the FPGA, including the number of LUTs, the number of DSP blocks, on-chip memory capacity, and external memory bandwidth, perform preliminary adaptation of the AI ​​algorithm to determine the approximate architecture and resource allocation scheme when implementing it on the FPGA, including the parallelism of each computing module, memory hierarchy partitioning, and data path design.

[0010] Data preprocessing steps: For the input data of the AI ​​algorithm, perform format conversion and normalization operations; according to the FPGA memory resources and the computational requirements of the algorithm, divide the input data into blocks and determine an appropriate boundary padding strategy to ensure the accuracy of the calculation after block division and avoid the loss of boundary information or the accumulation of calculation errors caused by block division.

[0011] Key module optimization steps: Convolution calculation unit is constructed on FPGA using LUT, automatic LUT structure adaptation module and DSP unit; The automatic LUT structure adaptation module dynamically adjusts the internal logic connection and lookup table content of LUT according to the current convolution kernel size, data type and expected calculation precision before the convolution operation starts, so that the same hardware multiplication unit can adapt to the multiplication operation requirements of different bit widths and precisions without resynthesizing the hardware circuit.

[0012] Parallel computing and scheduling steps: For the pooling layer, a dedicated hardware circuit structure is used for optimization, and a comparison tree network is used to find local maxima within a single cycle; in the fully connected layer, the weight storage and multiply-accumulate operation methods are optimized. The weight data is stored in the on-chip memory of the FPGA, and multiply-accumulate operations are performed using pipelined technology.

[0013] This allows the weight reading, multiplication, and accumulation operations to be executed concurrently in time. Multiple parallel computing units are built on the FPGA, with each unit responsible for executing the computational tasks of one or more key modules in the AI ​​algorithm. A task scheduling mechanism is designed to rationally allocate computational tasks to each computing unit based on data dependencies and the idle state of the computing units, maximizing resource utilization. During parallel computing, pipeline technology is used to further improve computational efficiency and reduce the waiting time of data between processing stages.

[0014] Results merging and post-processing steps: Merge the intermediate results obtained from each computation unit, stitch and adjust the partial feature maps generated by block computation according to the algorithm requirements, and eliminate the boundary discontinuities introduced by block processing; perform post-processing operations on the merged results, including activation function calculation and final classification or regression output, and output the final result after normalization.

[0015] An AI-oriented artificial intelligence algorithm implementation system on FPGA, comprising:

[0016] The algorithm analysis and adaptation unit is used to perform algorithm analysis and adaptation operations as described above, receive the description of the AI ​​model to be deployed, parse the configuration parameters of each layer, and generate resource allocation suggestions and hardware implementation schemes.

[0017] The data preprocessing unit is responsible for performing format conversion, normalization, and block preprocessing operations on the input data. It receives raw data from external memory via the AXI bus, processes it, and then sends the data blocks to the downstream computing unit via the on-chip FIFO.

[0018] The key module optimization unit includes sub-units for optimizing convolutional layers, pooling layers, and fully connected layers. These include a module for automatically adapting LUT structures, a pooling layer optimization circuit, and a fully connected layer weight storage and computation optimization unit. The convolutional layer sub-unit uses a LUT multiplication matrix based on a divide-and-conquer strategy, the pooling layer sub-unit uses a dedicated hardware circuit based on a comparison network, and the fully connected layer sub-unit uses on-chip BRAM to store weights and combines them with a pipelined multiply-accumulator.

[0019] The parallel computing and scheduling unit constructs multiple parallel computing units and is responsible for task scheduling and pipeline management. It uses a double-buffered queue to achieve the overlap of data supply and computation execution time, keeping each computing unit continuously busy.

[0020] The result merging and post-processing unit is used to merge intermediate results and perform post-processing operations. It splices the partial feature maps output by each calculation unit into a complete result according to their spatial location, performs activation function calculation and softmax normalization, and outputs the final classification or regression result.

[0021] Furthermore, in the algorithm analysis and adaptation step, the key computing module includes a convolutional layer, a pooling layer, and a fully connected layer.

[0022] Furthermore, in the data preprocessing step, the pixel values ​​of the image data are converted from 8-bit unsigned integers to fixed-point numbers suitable for FPGA operations, and normalized to the range of [0, 1] or [-1, 1].

[0023] Furthermore, the module that automatically adapts to the LUT structure uses a 6-input LUT structure to complete 3x3 multiplication, and achieves multiplication operations with larger-size convolutional kernels by combining and accumulating the results of multiple 3x3 multiplication calls.

[0024] Furthermore, the dedicated hardware circuit structure of the pooling layer is a max pooling hardware structure based on a comparison network, which can quickly find the maximum value in a local region.

[0025] Furthermore, the task scheduling mechanism employs a double-buffered queue to manage the data blocks to be processed, ensuring that the computing unit remains continuously busy and reducing waiting time.

[0026] Furthermore, in the convolution calculation unit, the convolution calculation result of the previous sub-block is used as part of the input for the calculation of the next sub-block, and the convolution calculation of different sub-blocks is performed in parallel at different stages.

[0027] Furthermore, the post-processing operations include activation function calculation and classification or regression result output.

[0028] Furthermore, the FPGA is a Xilinx Zynq UltraScale+ MPSoC chip.

[0029] Furthermore, the artificial intelligence algorithm is a convolutional neural network algorithm. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of the hardware structure of the convolution calculation module on the FPGA of the present invention;

[0032] Figure 2 This is a schematic diagram of the architecture of the parallel computing and scheduling unit of the present invention;

[0033] Figure 3 This is the overall flowchart of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] A method for implementing AI-oriented artificial intelligence algorithms on FPGA, comprising:

[0036] The following section uses Convolutional Neural Networks (CNNs) for image classification as an example to illustrate the specific implementation of this invention.

[0037] First, algorithm analysis and adaptation are performed. For this CNN model, key modules such as convolutional layers, pooling layers, and fully connected layers are analyzed to clarify the computational characteristics and data flow of each layer. At this stage, the input and output feature map sizes, convolutional kernel sizes and numbers, stride, padding methods, parameter count, and computational cost of each layer need to be statistically analyzed. Based on this, the computational resources, storage bandwidth, and on-chip cache capacity required for FPGA implementation of each layer are evaluated. For a typical CNN classification network using 256x256 pixel images as input, its convolutional layers typically account for more than 80% of the total computation and are the key areas requiring optimization. Pooling layers are mainly used for downsampling, with relatively low computational cost but high execution frequency. Fully connected layers have a large number of parameters but a regular computational structure, making them suitable for pipelined processing. Based on the above analysis results and the resource limits of the target FPGA device, a resource allocation scheme for each module is formulated.

[0038] In the data type conversion stage, the pixel values ​​of the input data are converted from 8-bit unsigned integers to fixed-point numbers. This invention adopts the Qmn format, which converts image pixel values ​​into 12-bit fixed-point numbers (1 sign bit, 3 integer bits, and 8 decimal bits) in specific implementations. This representation can cover the normalized range [0, 1] while maintaining a quantization precision of approximately 0.004. Compared to floating-point arithmetic, fixed-point arithmetic has a smaller logic area and higher processing speed on FPGAs, and by reasonably selecting the number of decimal places, resource consumption can be significantly reduced while meeting precision requirements.

[0039] The normalization operation maps the transformed pixel values ​​to the interval [0, 1] or [-1, 1]. The specific interval selection depends on the preprocessing method used during training and is accomplished by implementing a fixed-point divider in the FPGA. For [0, 1] normalization, the fixed-point pixel value is divided by 255; for [-1, 1] normalization, it is first divided by 128 and then subtracted by 1. The selection of normalization parameters directly affects the model's inference accuracy, therefore it needs to be strictly consistent with the preprocessing during the training phase.

[0040] Data block processing divides a large input image into several smaller sub-blocks based on the FPGA's on-chip storage capacity, processing them sequentially. Taking a 256x256 pixel input image as an example, it is divided into 16x16 pixel blocks, resulting in 256 sub-blocks. For edge sub-blocks, mirror padding is used, reflecting pixel values ​​outwards along the image edge as a symmetrical axis. For a convolution kernel size of K, the padding width is (K-1) / 2 pixels. Mirror padding, compared to zero padding, better preserves boundary information and reduces edge artifacts.

[0041] During the key module optimization phase, the focus is on in-depth optimization of the convolution calculation module.

[0042] Design an int8 multiplier based on a divide-and-conquer strategy. Divide two 8-bit numbers A and B into three high-order bits, three middle-order bits, and two low-order bits respectively. Obtain all cross-product terms by expanding the product. A total of 9 small multipliers are required, specifically 4 3×3, 2 3×2, 2 2×3, and 1 2×2 multipliers.

[0043] These small multipliers are implemented on the FPGA using LUTs: each 3×3 multiplier requires one 6-input LUT, each 3×2 or 2×3 multiplier requires one 5-input LUT, and each 2×2 multiplier requires one 4-input LUT. Therefore, a single int8 multiplier consumes a total of 4×LUT6 + 4×LUT5 + 1×LUT4 = 9 LUTs.

[0044] Based on this, a 64×64 multiplication matrix (i.e., 4096 int8 multiplication units) similar to Google TPU is constructed. The multiplication part requires 4096×9 = 36,864 LUTs. With additional logic such as control and storage (additional overhead of about 20% to 50%), the total LUT requirement is about 44,000 to 55,000.

[0045] Unlike the strategy of TPU's pulsating array to reduce the number of computing units through data flow multiplexing, FPGA adopts a fully parallel implementation to match the throughput. The 64×64 multiplication matrix can complete 4096 multiplications per cycle. If the operating frequency is 1GHz, the peak computing power can reach 4.096 TOPS (8 TOPS if multiply-add are combined). In practice, 8 TOPS can be achieved through pipeline or dual-issue design.

[0046] The module that automatically adapts the LUT structure dynamically adjusts the lookup table content based on the current configuration during runtime: when the convolutional kernel size switches from 3x3 to 5x5 or 7x7, the control logic reconfigures the internal connections of the LUT, combining and accumulating multiple 3x3 multiplication results using a divide-and-conquer strategy, effectively achieving multiplication operations with larger convolutional kernels without resynthesizing the hardware circuitry. This reconfigurable feature allows the same hardware architecture to adapt to different network models, significantly shortening the model deployment cycle.

[0047] In addition, for the pooling layer, a dedicated hardware circuit based on a comparison network is designed to quickly find the maximum value in a local region. For example, a three-level comparator tree structure is used to achieve single-cycle output for 2×2 window max pooling.

[0048] For the fully connected layer, the weight data is stored in the FPGA's on-chip memory (BRAM) and multiply-accumulate operations are performed using pipelined technology, which consists of three stages: weight read, multiplication, and addition accumulation. In addition, ping-pong buffers are used to prefetch weights to avoid computational pauses.

[0049] In terms of parallel computing and scheduling, four parallel computing units are built on the FPGA, each containing the aforementioned convolution, pooling, and fully connected processing sub-modules.

[0050] A double-buffered queue is used to manage the image data blocks to be processed. One buffer is used to write the sub-blocks to be processed, and the other is used to read the current sub-block. When any computing unit is idle, the next data block is obtained from the head of the queue for computation.

[0051] Inside the convolutional computation unit, a pipeline technique is used so that the convolutional computation result of the previous 16×16 sub-block can be used as part of the input for the computation of the adjacent sub-block. Overlapping areas do not need to be re-computed, thereby reducing the overhead of repeated reading and computation.

[0052] The four computing units execute tasks independently and in parallel. The task scheduling mechanism dynamically allocates tasks based on data dependencies and unit idle status to ensure that each unit is constantly busy.

[0053] Finally, the results are merged and post-processed. The feature maps of the convolutional and pooling layers output by the four computing units are stitched together into a complete feature map according to their original spatial positions. Then, the activation function (such as ReLU) is calculated, and the result is fed into a fully connected layer to obtain the final classification score. After softmax normalization, the class with the highest probability is output as the image classification result.

[0054] Using the above method, this CNN image classification application was implemented on a Xilinx Zynq UltraScale+ FPGA. After synthesis, the resource usage was approximately 52,000 LUTs, 800 DSPs, and 512KB of BRAM. The total latency for processing a single 256×256 image frame was approximately 8ms (excluding image transmission), which is about 15 times faster than the pure software implementation on an ARM Cortex-A53, with a power consumption of only 4.5W. Furthermore, by modifying the LUT content or loading different model weights, it can be quickly adapted to other AI algorithms, demonstrating good versatility and flexibility.

Claims

1. A method for implementing an artificial intelligence algorithm for AI on an FPGA, characterized in that, include: Algorithm analysis and adaptation steps: Analyze the artificial intelligence algorithm and initially adapt it to FPGA resources; The data preprocessing step involves format conversion, normalization, and block processing of the input data; The key module optimization steps involve optimizing key modules (such as convolutional layers, pooling layers, and fully connected layers) on the FPGA; the parallel computing and scheduling steps involve building parallel computing units on the FPGA and performing task scheduling and pipeline management; and the result merging and post-processing steps involve merging intermediate results and performing post-processing operations.

2. The method for implementing AI-oriented artificial intelligence algorithms on FPGA according to claim 1, characterized in that, In the algorithm analysis and adaptation step, the approximate architecture and resource allocation scheme on the FPGA are determined based on the computational requirements of the key computing modules of the artificial intelligence algorithm and the resource characteristics of the FPGA.

3. The method for implementing AI-oriented artificial intelligence algorithms on FPGA according to claim 1, characterized in that, In the data preprocessing step, the pixel values ​​of the input data are converted into a fixed-point number form suitable for FPGA operations and normalized to a specific range.

4. The method for implementing AI-oriented artificial intelligence algorithms on FPGA according to claim 1, characterized in that, In the key module optimization step: In the convolution calculation module, a convolution calculation unit is constructed using a LUT, an automatic LUT structure adaptation module, and a DSP unit. The automatic LUT structure adaptation module dynamically adjusts the internal logical connections and lookup table contents of the LUT based on factors such as the size of the convolution kernel. For the pooling layer, a hardware structure based on a comparison network is used for optimization; In the fully connected layer, the weight storage and multiply-accumulate operation methods are optimized and pipelined technology is adopted.

5. The method for implementing AI-oriented artificial intelligence algorithms on FPGA according to claim 1, characterized in that, In the parallel computing and scheduling steps, a double-buffered queue is used to manage the data blocks to be processed, and pipeline technology is used in the parallel computing process.

6. The method for implementing AI-oriented artificial intelligence algorithms on FPGA according to claim 1, characterized in that, In the result merging and post-processing step, the intermediate results are spliced ​​and adjusted according to the requirements of artificial intelligence algorithms, and then the activation function is calculated and the final result is output.

Images