High-throughput hardware acceleration architecture of ultra-large data volume quantitative multiplication network

Abstract

The invention belongs to the field of FPGAs (Field Programmable Gate Array), and particularly relates to a method for deploying a convolution neural network on an FPGA, which is suitable for carrying out FPGA deployment on various convolution networks. By means of the implementation mode, high throughput and high frame number can be achieved while a large number of hardware resources are saved, compared with a traditional FPGA deployment mode, a large number of hardware resources are saved, the operation speed is increased, power consumption is reduced, meanwhile, the deployment mode has extremely high flexibility, code parallelism can be changed according to different hardware resource configurations, and the implementation efficiency is improved. In addition, the deployment mode enables mass data to be compressed on the chip, avoids power consumption and time consumption increased by multiple times of access to off-chip storage, and has the advantages of being low in delay, low in power consumption, high in operation processing speed and the like. The problems of large resource consumption, high power consumption, low operation speed, inflexible structure and the like caused when a traditional convolutional neural network is deployed on the FPGA are effectively avoided.

Description

Technical Field

[0001] The present invention relates to a method for deploying a convolutional classification neural network on an FPGA, belonging to the technical field of scene classification and recognition, and mainly relates to a method for implementing the deployment of a classification neural network on an FPGA. Background Art

[0002] With the development of autonomous driving technology, intelligent detection and recognition of road scenes are playing an increasingly important role in many application scenarios. Road scene classification is a key classification task, widely used in tasks such as autonomous driving, intelligent transportation, and high-precision navigation. In recent years, deep learning-based methods have been widely used in road scene classification due to their powerful feature abstraction and generalization capabilities. However, deep learning-based methods often require significant computational costs and high power consumption. Convolutional neural networks (CNNs) hold great potential in road scene recognition applications. However, typical CNN networks, such as AlexNet, VggNet, and ResNet, typically require hundreds of MB of memory. Directly deploying neural networks on edge devices faces many challenges. First, the multiplication and addition operations in neural networks are very intensive. Direct deployment without proper system design results in significant resource overhead, making network deployment difficult on small and medium-sized boards. Second, directly deploying neural networks using floating-point precision puts significant pressure on edge device hardware resources, making them hardware-inefficient. Furthermore, neural networks contain multiple types of operators, and customizing hardware architectures for each type of operator consumes significant hardware resources. However, completing convolutional neural network operations requires a large number of weight parameters, which means that a considerable amount of on-chip storage space is required. The storage of edge devices such as field-programmable gate arrays (FPGAs) and application-specific integrated circuits (ASICs) is usually unable to support the high-energy-efficiency deployment of convolutional neural networks. The present invention aims to solve these problems. Summary of the Invention

[0003] The technical problem solved by the present invention is: to overcome the shortcomings of the existing technology and propose a method for deploying a convolutional classification neural network on an FPGA. This method uses 8-bit quantization technology and fusion operators to reduce the consumption of computing resources and storage resources on the FPGA board, effectively avoiding the inability to directly deploy on the FPGA due to limitations of computing resources and storage resources. Deployment on the FPGA through this method has multiple advantages such as low latency, low power consumption, and high throughput.

[0004] The technical solution of the present invention is:

[0005] A method for deploying a convolutional classification neural network on an FPGA includes the following steps:

[0006] Step 1: Input hyperparameters into the convolutional neural network for parameter training to obtain a floating-point model .PTH file. Then, perform 8-bit quantization training on the obtained floating-point model .PTH file to obtain a fixed-point quantization model .PTH file. Then, extract the quantization step data of the fixed-point quantization model .PTH file. Based on the obtained quantization step data, obtain the 8-bit fixed-point data of the weight parameter and the floating-point data of the normalization parameter.

[0007] The obtained floating-point model .PTH file contains floating-point numbers of weight parameters and floating-point numbers of normalization parameters;

[0008] Step 2: Rearrange the 8-bit fixed-point data of the weight parameters obtained in step 1 according to the six-level convolution cycle order to obtain rearranged 8-bit fixed-point data, and then generate an 8-bit fixed-point data .coe file from the rearranged 8-bit fixed-point data.

[0009] The floating-point data of the normalized parameters of the weight parameters obtained in step 1 are rearranged in the data stream according to the six-level convolution cycle order to obtain the rearranged floating-point data of the normalized parameters;

[0010] Then, the rearranged floating-point data of the normalized parameters are merged with the quantization step data in step 1 to generate a normalized .coe file;

[0011] Step 3: Modify the sample script published on the Xilinx official website according to the parallelism configuration and the path of the 8-bit fixed-point data .coe file to obtain the TCL instantiation script A;

[0012] Modify the sample script published on the Xilinx official website according to the parallelism configuration and the path of the normalized .coe file to obtain TCL instantiation script B;

[0013] Step 4: Copy the instructions in the TCL instantiation script A to Vivado to batch generate storage IP core A. Copy the instructions in the TCL instantiation script B to Vivado to batch generate storage IP core B.

[0014] In step 5, the feature map to be processed is stored in the SD card, and the SD card is inserted into the board with IP core A and IP core B to perform group convolution calculations to obtain classification results, thus completing the deployment of the convolution classification neural network on the FPGA.

[0015] In step 1, the method for performing 8-bit quantization training is:

[0016] Import the floating-point model .PTH file into the fixed-point quantized model, perform the set number of training rounds, and save the state of the quantized model to obtain the fixed-point quantized model .PTH file.

[0017] In step 2, the method for generating an 8-bit fixed-point data .coe file is:

[0018] Import the 8-bit fixed-point data of the weight parameters, then input the key value of each weight layer according to the dictionary operation to obtain the corresponding weight data value. Rearrange the weight data value according to the six-level convolution cycle order, and import it into a .txt file according to the .coe file format. After the data import is completed, change the suffix attribute from .txt to .coe;

[0019] The second step is to generate script instructions and corresponding calculation parallelism configurations based on the TCL of the IP core provided by Xilinx's official website. Use Python to batch generate the required TCL instructions under different parallelisms and put them in a .txt file. Then copy this instruction and input it into the TCL controller of Vivado.

[0020] Compared with the existing technology, the present invention has the following advantages:

[0021] (1) The present invention makes full use of quantization technology to quantize floating-point 32-bit width data to 8-bit width data. Compared with the existing method of deploying floating-point models on FPGA, it can greatly reduce the demand for computing resources and storage resources, so that the saved resources can be used to further improve the performance of convolutional neural networks on FPGA.

[0022] (2) The present invention utilizes operator fusion to fuse four operators at the mathematical level. Four mathematical operations can be completed through only two floating-point multiplications and one floating-point addition, which greatly saves computing resources. At the same time, the quantization operator is placed before the pooling operator. Without affecting the correctness, the data is stored with an 8-bit fixed-point width instead of the traditional floating-point data storage, which can greatly reduce the consumption of storage resources.

[0023] (4) The present invention adopts the AXI4 handshake protocol as the communication protocol for interaction between PS-DDR and PL, which not only avoids data loss but also has an ultra-high data stream transmission speed, which can perfectly match the processing speed of the PL on the FPGA board.

[0024] The present invention belongs to the field of FPGAs, and specifically relates to a method for implementing a dedicated convolutional neural network on an FPGA. The method is suitable for deploying various convolutional neural networks on an FPGA. This method can achieve high throughput and high frame rates while saving a large amount of hardware resources. Compared with traditional FPGA deployment methods, this method saves a large amount of hardware resources, improves computing speed, and reduces power consumption. At the same time, this deployment method is extremely flexible and can adjust code parallelism according to different hardware resource configurations to achieve high resource utilization. Moreover, this deployment method compresses large amounts of data on-chip, avoiding the increased power and time consumption of multiple accesses to off-chip storage. It has multiple advantages such as low latency, low power consumption, and high computing processing speed. It effectively avoids the various problems caused by traditional convolutional neural networks deployed on FPGAs, such as high resource consumption, high power consumption, slow computing speed, and inflexible structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A data transmission interface and operation flow chart of an embodiment of the present invention;

[0026] Figure 2 The hardware deployment of the algorithm reconstruction of the embodiment of the present invention shows the digital precision of the basic block operation and the fusion diagram of the inverse quantization-BN-ReLU-quantization layer;

[0027] Figure 3 This is a flow chart of convolution operation in an embodiment of the present invention;

[0028] Figure 4 This is a diagram showing the principle of group convolution in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0030] This paper proposes a full-pipeline deployment method for a neural network. The proposed method can be deployed on an FPGA board consisting of an ARM-based processing system (PS) and programmable logic (PL). The PS includes DDR, an Ethernet interface, and other components. The PL includes the following components: a DDR interface controller, a parameter buffer, data flow control, data flow calculation, an output control module, and an output calculation module. The PS-DDR is primarily used to store feature images and transmit image data to the PL via the HP port of the AXI4 protocol. The image data in the PS-DDR comes from images pre-stored on the host computer via the network port or by reading an SD card.

[0031] In this embodiment, the multiplication convolution classification neural network Vgg11 model is deployed on the AMD-Xilinx KV260 processing platform for hardware acceleration processing. The platform is equipped with an xck26-sfvc784-2LV-c FPGA chip, which has 117K lookup tables (LUTs), 234K flip-flops (FFs), 144 block random access memories (BRAMs), 64 ultra-large flexible random access memories (Ultra RAMs), and 1248 digital signal processing units (DSPs). The following example will be used to introduce this FPGA deployment implementation method in detail. The flowchart is as follows: Figure 1 As shown, the following steps are included:

[0032] Step 1: Run the Vgg11 multiplication classification neural network floating-point model (with grouped convolution) on the Python interpreter. By continuously changing the hyperparameters, find the floating-point model file (PTH) with the highest test accuracy and save the corresponding floating-point model .PTH file. Then, perform 10 rounds of 8-bit quantization training on the floating-point model .PTH file and save the final round of fixed-point quantization model .PTH file. Then, extract the quantization step data of the fixed-point quantization model .PTH file. Based on the obtained quantization step data, obtain the 8-bit fixed-point data of the weight parameters and the floating-point data of the normalization parameters.

[0033] Step 2: Rearrange the 8-bit fixed-point data of the weight parameters obtained in step 1 according to the six-level convolution cycle order to obtain rearranged 8-bit fixed-point data, and then generate an 8-bit fixed-point data .coe file from the rearranged 8-bit fixed-point data.

[0034] The floating-point data of the normalized parameters of the weight parameters obtained in step 1 are rearranged in the data stream according to the six-level convolution cycle order to obtain the rearranged floating-point data of the normalized parameters;

[0035] Then, the rearranged floating-point data of the normalized parameters are merged with the quantization step data in step 1 to generate a normalized .coe file;

[0036] Step 3. Modify the sample script released on the Xilinx official website according to the input parallelism [32, 32, 32, 32, 32, 32, 32, 32], output parallelism [4, 4, 4, 4, 2, 2, 1, 1] and the path of the 8-bit fixed-point data .coe file \home\coe\8bit.coe to obtain the TCL instantiation script A.

[0037] Modify the sample script released on the Xilinx official website to obtain TCL instantiation script B according to the output parallelism configuration [4, 4, 4, 4, 2, 2, 1, 1] and the normalized .coe file path \home\coe\bn.coe;

[0038] Step 4: Copy the instructions in TCL instantiation script A to Vivado to batch generate storage IP core A. Copy the instructions in TCL instantiation script B to Vivado to batch generate storage IP core B.

[0039] Step 5: Store 2100 road scene feature maps to be processed into an SD card and insert the SD card into the board with IP core A and IP core B. The specific calculation process is as follows: Figure 2 As shown, group convolution calculation is performed in the convolution layer, and the group convolution calculation method is as follows Figure 3 and Figure 4 As shown in the figure, the classification results are finally obtained and compared with the classification labels of the actual 2100 road scene feature maps to be processed.

[0040] After the above optimization method is deployed on the board, the final resource consumption is as follows:

[0041]

[0042] For hardware accelerators, throughput, resource utilization, and power consumption are key performance metrics. During model inference, giga-operations (GOPs) are used to measure the number of operations in the network. One MAC operation is a multiplication and an addition, which is considered two operations. Throughput is typically determined by the number of operations. Resource utilization of a hardware accelerator is the number of LUTs, FFs, BRAMs, and DSPs consumed by the deployed acceleration network. Accelerator power consumption can be assessed using the ReportPower tool in Vivado.

[0043] After multiple simulation tests and experiments, the key performance indicators of the present invention were obtained. Among them, at a clock frequency of 100M, the total power consumption of the present invention is 2.750W, and the computing power reaches 104.76GOP. Taking scene classification as an example, the frame rate can reach 225fps;

[0044] Of course, the present invention may have many other application examples. Without departing from the spirit and essence of the present invention, technicians familiar with this field may make various corresponding modifications based on the present invention, but these corresponding modifications should all fall within the scope of protection of the claims attached to the present invention.

[0045] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. The deployment method of convolutional classification neural network on FPGA is characterized by The steps of the method include: Step 1: Input hyperparameters into the convolutional neural network for parameter training to obtain a floating-point model .PTH file. Then, perform 8-bit quantization training on the obtained floating-point model .PTH file to obtain a fixed-point quantization model .PTH file. Then, extract the quantization step data of the fixed-point quantization model .PTH file. Based on the obtained quantization step data, obtain the 8-bit fixed-point data of the weight parameter and the floating-point data of the normalization parameter. Step 2: Rearrange the 8-bit fixed-point data of the weight parameters obtained in step 1 according to the six-level convolution cycle order to obtain rearranged 8-bit fixed-point data, and then generate an 8-bit fixed-point data .coe file from the rearranged 8-bit fixed-point data. The floating-point data of the normalized parameters of the weight parameters obtained in step 1 are rearranged in the data stream according to the six-level convolution cycle order to obtain the rearranged floating-point data of the normalized parameters; Merge the rearranged floating-point data of the normalized parameters with the quantization step data in step 1 to generate a normalized .coe file; Step 3: Modify the sample script published on the Xilinx official website according to the parallelism configuration and the path of the 8-bit fixed-point data .coe file to obtain the TCL instantiation script A; Modify the sample script published on the Xilinx official website according to the parallelism configuration and the path of the normalized .coe file to obtain TCL instantiation script B; Step 4: Copy the instructions in the TCL instantiation script A to Vivado to batch generate storage IP core A. Copy the instructions in the TCL instantiation script B to Vivado to batch generate storage IP core B. In step 5, the feature map to be processed is stored in the SD card, and the SD card is inserted into the board with IP core A and IP core B to perform convolution calculations to obtain classification results, thus completing the deployment of the convolution classification neural network on the FPGA.

2. The method for deploying a convolutional neural network on an FPGA according to claim 1, characterized in that: In step 1, the obtained floating-point model .PTH file contains floating-point numbers of weight parameters and floating-point numbers of normalization parameters.

3. The method for deploying a convolutional neural network on an FPGA according to claim 1, wherein: In step 1, the method for performing 8-bit quantization training is: Import the floating-point model .PTH file into the fixed-point quantized model, perform the set number of training rounds, and save the state of the quantized model to obtain the fixed-point quantized model .PTH file.

4. The method for deploying a convolutional neural network on an FPGA according to claim 1, wherein: In step 1, the method for obtaining the 8-bit fixed-point data of the weight parameter is: Divide the weight data in the fixed-point quantization model .PTH file by the quantization step data and round it off to get the 8-bit fixed-point data of the weight parameter.

5. The method for deploying a convolutional neural network on an FPGA according to claim 1, wherein: In step 2, the method for generating an 8-bit fixed-point data .coe file is: The 8-bit fixed-point data of the weight parameters are re-indexed according to the six-level convolution cycle order, and imported into a .txt file according to the .coe file format. After the data import is completed, the suffix attribute is changed from .txt to .coe.

6. The method for deploying a convolutional neural network on an FPGA according to claim 1, wherein: In step 2, the method for generating a normalized .coe file is: The floating-point data of the normalized parameters are re-indexed according to the six-level convolution cycle order of the weight data values, and imported into the .txt file according to the .coe file format. After the data import is completed, the suffix attribute is changed from .txt to .coe.

7. The method for deploying a convolutional neural network on an FPGA according to claim 1, wherein: In step 3, the method for generating the TCL instantiation script A is: Use Python to generate the TCL instructions required for different parallelism levels based on the sample script released on the Xilinx official website, the path of the 8-bit fixed-point data .coe file, and the corresponding calculation parallelism configuration. Put them in a .txt file to obtain TCL instantiation script A.

8. The method for deploying a convolutional neural network on an FPGA according to claim 1, wherein: In step 3, the method for generating the TCL instantiation script B is: Use Python to generate the TCL instructions required for different parallelism levels based on the sample script released on the Xilinx official website, the normalized .coe file path, and the corresponding calculation parallelism configuration. Put them in a .txt file to obtain TCL instantiation script B.

9. The method for deploying a convolutional neural network on an FPGA according to claim 1, wherein: In step 5, the method for performing convolution calculation on the data in the SD card and the data in the IP core is: The data in the SD card is stored in the DDR of the PS side of the board using the storage tool of the Vitis software. The DDR data stored in the PS side of the board is read out using the AXI4 protocol and sent to the PL side of the board for convolution calculation with the data in the IP core.

Images