Winograd algorithm convolution accelerator based on FPGA
By designing the Winograd algorithm convolutional neural network accelerator on FPGA, using small block parallel computing and global buffer modules to optimize data processing, the problem of large amount of convolutional neural network calculation is solved, and more efficient convolutional computing and energy consumption reduction is achieved .
Patent Information
- Application Number
- CN202411436144.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-15
- Publication Date
- 2025-01-17
AI Technical Summary
Convolutional neural networks are computationally expensive and time-consuming when processing large-scale data and deep networks, and it is difficult for the existing technology to effectively improve the computing speed and efficiency.
Design a Winograd algorithm convolutional neural network accelerator based on FPGA. By segmenting the input feature map into small blocks of equal size and performing Winograd convolution in parallel, and using the global buffer module to cache weight and bias data, optimize data storage and calculation process , make full use of the parallelism of FPGA.
It realizes more efficient convolutional calculation, significantly reduces the number of multiplication operations, improves calculation efficiency and reduces energy consumption, and is suitable for large-scale data processing and complex network structures.
Smart Images
Figure CN119323241A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of CNN accelerated computing of FPGA, and in particular relates to a Winograd algorithm convolutional neural network accelerator based on FPGA. Background Art
[0002] Nowadays, convolutional neural networks (CNNs) are widely used in various fields such as signal processing, image processing, and deep learning. In signal processing, it simulates the interaction between signals and systems and is used for filtering, feature extraction, etc. In image processing, convolution is used for edge detection, blurring, sharpening, etc. to effectively extract image features. In deep learning, the convolution layer of CNN is the core. By automatically learning spatial hierarchical features, it has greatly promoted the development of image recognition, natural language processing and other fields. However, with the increase in data scale and network depth, convolution operations face challenges such as large amount of calculation and long time consumption, and efficient algorithms are needed to improve the speed and efficiency of operations, such as the Winograd algorithm.
[0003] The Winograd algorithm is an acceleration technology for convolution operations, especially in the field of CNN accelerator design. The algorithm transforms conventional convolution operations into a more compact small matrix multiplication form through matrix decomposition and transformation, thereby significantly reducing the number of multiplication operations and improving computational efficiency. The core idea of the Winograd algorithm is to use non-time-consuming operations such as addition to replace some multiplication operations in order to reduce the time complexity of the algorithm. Although this method will increase the storage requirements for addition and intermediate variables, its advantage in reducing multiplication operations is particularly obvious in large-scale processing and computationally intensive applications. In addition, the Winograd algorithm is easy to parallelize and can achieve higher computational efficiency and lower computational energy consumption on multi-core processors and FPGAs. This feature makes the Winograd algorithm very popular in the design of CNN accelerators based on FPGAs, especially when dealing with large-scale data sets and complex network structures. Summary of the invention
[0004] Based on this, in order to achieve higher computing efficiency of the Winograd algorithm on the CNN accelerator, the present invention designs a Winograd algorithm CNN accelerator based on FPGA.
[0005] The objective of the present invention is achieved through the following technical solutions:
[0006] S100. Given an input / output sample pair, training the convolutional neural network based on the input / output sample pair, and ending the training process when reaching a specified error range or a maximum number of times;
[0007] S200. Obtaining trained convolutional neural network model parameters and structure, wherein the model parameters include convolution kernel parameters and bias parameters in each convolution layer;
[0008] S300. Set quantization parameters, load the trained convolutional neural network model parameters, execute the quantization process and end the quantization process when the specified error range is reached.
[0009] Further, the S300 includes the following steps:
[0010] S310. Given an input / output sample pair;
[0011] S320. Obtain the trained convolutional neural network model parameters and structure;
[0012] S330. Set quantization parameters to quantize the convolution kernel parameters in each convolution layer in the convolutional neural network model into 8-bit integer type data;
[0013] S340. Use the input sample as the input vector to make the quantized convolutional neural network perform forward propagation, and determine whether the accuracy loss meets the preset conditions. If so, end the quantization process, if not, execute S330.
[0014] S400. The quantized convolutional neural network parameters are stored in an external DRAM memory;
[0015] S500. Open a global buffer module on the convolution accelerator, which is used to cache the weight data and bias data required for calculation from the external DRAM. At the same time, the global buffer module also includes an input feature cache for caching a given input pair, and an output feature cache for caching output features;
[0016] S600. Transform the data of the input feature map buffer and the weight buffer to make them conform to the Winograd algorithm convolution input.
[0017] Furthermore, the step S600 includes the following steps:
[0018] S610. Cut the input feature map into blocks of equal size, where the size of each block is determined by the convolution kernel;
[0019] S620. Allocate a data buffer and store the overlapping data between adjacent blocks in the data buffer to facilitate further parallel computing;
[0020] S630. Execute Winograd algorithm convolution in parallel on each small block of input feature map and data overlap area to achieve high-performance parallel computing;
[0021] S640. Repeat S610 to S630 until all input channels are traversed.
[0022] S700. Execute the addition module to add the convolution results of the Winograd algorithm on the corresponding input channels to obtain an output result;
[0023] S800. Repeat S500 to S700 until the entire convolutional neural network calculation is completed.
[0024] The beneficial effects of the present invention are as follows: the present invention proposes a Winograd algorithm convolutional neural network accelerator based on FPGA, that is, the input feature map with a larger original size is divided into small blocks of equal size, and Winograd convolution is performed in parallel on each small block. In addition, the present invention divides the data overlapping parts between adjacent small blocks into small blocks of the same size as the input feature map segmentation blocks, so that Winograd convolution can be performed in parallel on each small block. The convolution accelerator designed by the present invention proposes a reasonable acceleration architecture for deploying the Winograd algorithm, fully utilizes the parallelism of FPGA, and realizes more efficient convolution calculation. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 The present invention provides a flow chart for obtaining calculation parameters.
[0026] Figure 2 A quantitative schematic diagram provided by the present invention.
[0027] Figure 3 A data flow diagram of the accelerator provided by the present invention.
[0028] Figure 4 The Winograd accelerator structure provided by the present invention is intended.
[0029] Figure 5 Schematic diagram of the convolution accelerator segmentation method provided by the present invention. DETAILED DESCRIPTION
[0030] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0031] See also Figure 1-Figure 5 , this embodiment provides a Winograd algorithm convolutional neural network accelerator based on FPGA:
[0032] In order to save storage space on FPGA, the present invention quantizes the network model. Figure 1As shown, the calculation parameter acquisition process includes steps S100 to S300, which are described in detail below.
[0033] Step S100. Given an input / output sample pair, the convolutional neural network is trained based on the input / output sample pair, and the training process is terminated when a specified error range or a maximum number of times is reached.
[0034] Specifically, the network model is trained using the PyTorch network model framework, and the model parameters are exported for quantization when the specified error range or maximum number of times is reached.
[0035] S200. Obtain the trained convolutional neural network model parameters and structure, wherein the model parameters include convolution kernel parameters and bias parameters in each convolution layer.
[0036] Specifically, the convolutional neural network model parameters include convolution kernel parameters, bias parameters and fully connected layer parameters in each convolution layer. The purpose of obtaining the network model structure is that the quantization factors of the convolution kernel parameters and bias parameters of each convolution layer are different during quantization, and different convolution layers need to be quantized.
[0037] S300. Set quantization parameters, load the trained convolutional neural network model parameters, execute the quantization process and end the quantization process when the specified error range is reached.
[0038] In the process of quantizing the entire convolutional neural network model, a certain loss of accuracy will be caused. Therefore, in this embodiment, an input / output sample pair is given, and then the post-convolutional neural network model is quantized based on the input / output sample pair, and the quantization process is ended when the specified error range is reached.
[0039] In some embodiments, Figure 2 As shown, the S300 includes the following steps:
[0040] S310. Given an input / output sample pair.
[0041] S320. Obtain the trained convolutional neural network model parameters and structure.
[0042] S330. Set quantization parameters to quantize the convolution kernel parameters in each convolution layer in the convolutional neural network model into 8-bit integer type data.
[0043] S340. Use the input sample as the input vector to make the quantized convolutional neural network perform forward propagation, and determine whether the accuracy loss meets the preset conditions. If so, end the quantization process, if not, execute S330.
[0044] S400. The quantized convolutional neural network parameters are stored in an external DRAM memory.
[0045] External memory DRAM is mainly used to store the quantized weights, bias data, and input feature data of convolutional neural networks. In convolutional neural networks, weights, bias data, and input feature data are often relatively large, and the storage space they occupy is often relatively large. Therefore, on-chip memory is difficult to meet storage needs, and off-chip memory is required.
[0046] S500. A global buffer module is opened on the convolution accelerator, which is used to cache the weight data and bias data required for calculation from the external DRAM. At the same time, the global buffer module also includes an input feature cache for caching a given input pair, and an output feature cache for caching output features.
[0047] Since the on-chip memory resources of FPGA are limited, it is not enough to cache all the calculation parameters of the entire convolutional neural network model at one time. Therefore, the convolution accelerator first caches some convolution calculation parameters, and then outputs the results to the external memory after completing the convolution calculation. The convolution accelerator then repeats this process until the entire convolutional neural network calculation is completed.
[0048] In some embodiments, the computational data flow of the convolution accelerator is as follows: Figure 3 As shown in the figure, the weight data, bias data and input feature map stored in the external DRAM are read to the FPGA through the AXI bus, and then the convolution is performed using the Winograd accelerator deployed on the FPGA platform. The calculation results are first stored in the output feature cache, and then stored in the external memory through the AXI bus for the next calculation.
[0049] S600. Transform the data of the input feature map buffer and the weight buffer to make them conform to the Winograd algorithm convolution input.
[0050] The Winograd convolution algorithm is a technology used to accelerate convolution operations. Its core idea is to convert standard convolution into a more efficient matrix multiplication operation by transforming the convolution kernel and the input matrix, thereby reducing the amount of calculation. The Winograd algorithm usually transforms the input feature map by multiplying it by a predetermined transformation matrix, but such an operation requires additional memory resources to store the transformation matrix when executed on an FPGA. To avoid this problem, this embodiment manually builds the calculation process to complete the Winograd algorithm part. The accelerator structure is as follows: Figure 4 As shown, the S600 includes the following steps:
[0051] S610. Cut the input feature map into blocks of equal size, and the size of each block is determined by the convolution kernel.
[0052] Specifically, the Winograd algorithm convolutional neural network accelerator cuts the input feature map into small blocks of equal size according to formula (1), and performs Winograd algorithm convolution on each small block in parallel. The more parallel Winograd algorithm convolutions there are, the higher the overall computing performance of the accelerator. The size out_size of each small block is:
[0053] out_size = stride + k (1)
[0054] Where stride is the convolution step size, and k represents the size of the convolution kernel. Cutting the input feature image in this way can make the size of the output feature image 2×2.
[0055] S620. Allocate a data buffer and store the overlapping data between adjacent blocks in the data buffer to facilitate further parallel computing.
[0056] Specifically, the specific segmentation operation is as follows: Figure 5 As shown in the figure, the size of the input feature image is 8×8, and it is segmented into 4×4 blocks, which can get four small blocks numbered 1, 2, 3, and 4. Considering the subsequent parallel calculation, the overlapping data between the four blocks are stored in an additional data buffer, and the 5th small block is the overlapping data between blocks 1 and 2, the 6th small block is the overlapping data between blocks 1 and 3, the 8th small block is the overlapping data between blocks 2 and 4, the 9th small block is the overlapping data between blocks 3 and 4, and the last 7th small block is the overlapping data between blocks 1, 2, 3, and 4. After the segmentation, there is no data overlap between the blocks, and Winograd convolution can be performed in parallel.
[0057] Note that the block-based convolution does not add additional computational complexity, which is different from model compression techniques that aim to reduce the amount of computation or network parameters.
[0058] S630. Execute Winograd algorithm convolution in parallel on each small block of input feature map and data overlapping area to achieve high-performance parallel computing.
[0059] Specifically, the ratio of the number of multiplications reduced by the Winograd algorithm to the number of original convolution multiplications is calculated by formula (1) to evaluate the number of multipliers reduced, and the Winograd convolution operation is performed in parallel using these spare multipliers. The specific calculation formula for reducing the number of multiplications by the Winograd convolution algorithm is:
[0060] count = k × k × r out ×c out -r in ×c in (2)
[0061] Where k represents the convolution kernel size, r out ,c out Represents the rows and columns of the output, r in ,c in Represents the rows and columns of the input. The minuend of the formula is the number of multiplications of the traditional convolution, and the subtrahend is the number of convolution multiplications of the Winograd algorithm. The ratio of the number of reduced multiplications to the number of traditional convolution multiplications is:
[0062]
[0063] According to the input feature map size formula of claim (1), further, formula (3) can be transformed into:
[0064]
[0065] This is the ratio of the number of multiplications reduced by the Winograd algorithm convolutional neural network accelerator to the number of multiplications of the original convolution. Taking the traditional convolution VGG-16 as an example, when the convolution kernel size is 3×3 and the convolution step is 1, the ratio is calculated according to formula (4) to be 0.56, which means that if the Winograd algorithm is used to complete the VGG-16 network model, the multiplication calculation complexity will be reduced by about 0.56, which means that the use of multipliers on the FPGA is reduced by about half, and the multiplier resources reduced by the Winograd algorithm can be used to execute more Winograd convolutions to achieve higher computing efficiency.
[0066] S640. Repeat S610 to S630 until all input channels are traversed.
[0067] S700. Execute the addition module to add the convolution results of the Winograd algorithm on the corresponding input channels to obtain the output result.
[0068] Specifically, by indication Figure 4 As shown, the input feature map on each channel is convolved and input to the adder to add the Winograd convolution result of the previous input channel. The final result is then added to the output feature map of the bias buffer and cached in the output feature buffer. In this process, the activation function part is omitted in this embodiment. This part is placed between the adder and the feature buffer. When the Winograd convolution result is added to the bias, it passes through the activation function and is finally output to the output feature buffer.
[0069] S800. Repeat S500 to S700 until the entire convolutional neural network calculation is completed.
[0070] Specifically, by repeatedly calling the designed convolution accelerator, loading different input data and weight data, the convolution calculation of the entire network is finally completed.
[0071] The above description is only a preferred embodiment of the present invention. It should be appreciated that the present invention is not limited to the form disclosed herein, nor is it considered to exclude other embodiments, but can be used in various other combinations, variations and environments, and can be changed within the scope of the concept described herein by the above teachings or the technology or knowledge of the relevant field. However, as long as it does not deviate from the spirit and scope of the present invention, changes and modifications made by those skilled in the art will fall within the scope of protection of the claims attached to the present invention.
Claims
1. A Winograd algorithm convolution accelerator based on FPGA, characterized in that: The following steps are involved: S100. Given an input / output sample pair, training the convolutional neural network based on the input / output sample pair, and ending the training process when reaching a specified error range or a maximum number of times; S200. Obtaining trained convolutional neural network model parameters and structure, wherein the model parameters include convolution kernel parameters and bias parameters in each convolution layer; S300. Set quantization parameters, load the trained convolutional neural network model parameters, execute the quantization process and end the quantization process when the specified error range is reached; Further, the S300 includes the following steps: S310. Given an input / output sample pair; S320. Obtain the trained convolutional neural network model parameters and structure; S330. Set quantization parameters to quantize the convolution kernel parameters in each convolution layer in the convolutional neural network model into 8-bit integer type data; S340. Use the input sample as the input vector to forward propagate the quantized convolutional neural network, and determine whether the accuracy loss meets the preset conditions. If so, end the quantization process; if not, execute S330; S400. The quantized convolutional neural network parameters are stored in an external DRAM memory; S500. A global buffer module is set on the convolution accelerator, including an input cache, a weight cache and an output cache, wherein the input cache and the weight cache are used to store input data and weight data required for calculation read from an external DRAM, and the output cache is used to store feature map data after convolution, activation and pooling operations are completed on the input feature map and the input weight data; S600. Transform the data of the input feature map buffer and the weight buffer to make them conform to the convolution input of the Winograd algorithm; Furthermore, the step S600 includes the following steps: S610. Cut the input feature map into blocks of equal size, where the size of each block is determined by the convolution kernel; Specifically, the Winograd algorithm convolutional neural network accelerator cuts the input feature map into small blocks of equal size according to formula (1), and performs Winograd algorithm convolution on each small block in parallel. The more parallel Winograd algorithm convolutions there are, the higher the overall computing performance of the accelerator. The size out_size of each small block is: out_size = stride + k (1) Where stride is the convolution step size, and k represents the size of the convolution kernel. This allows the size of the output feature image after each small convolution to be 2×2, ensuring as many parallel Winograd algorithm convolutions as possible. S620. Allocate a data buffer and store the overlapping data between adjacent blocks in the data buffer to facilitate further parallel calculation; S630. Execute Winograd algorithm convolution in parallel on each small block of input feature map and data overlap area to achieve high-performance parallel computing; S640. Repeat S610 to S630 until all input channels are traversed; S700. Execute the addition module to add the convolution results of the Winograd algorithm on the corresponding input channels to obtain an output result; S800. Repeat S500 to S700 until the entire convolutional neural network calculation is completed.
2. The Winograd algorithm convolutional neural network accelerator according to claim 1, characterized in that: The Winograd algorithm convolution is performed in parallel on each small block after segmentation as much as possible. Specifically, the number of multipliers reduced is evaluated by calculating the ratio of the number of multiplications reduced by the Winograd algorithm to the number of original convolution multiplications using formula (1), and the Winograd algorithm convolution operation is performed in parallel using these free multipliers: The specific calculation formula for reducing the number of multiplications of Winograd algorithm convolution is: count=k×k×r out ×c out -r in ×c in (2) Among them, k represents the convolution kernel size, r out Represents the output line, c out Output columns, r in Represents the input line, c in Represents the input column, k×k×r of the formula out ×c out is the number of multiplications of the original convolution, r in ×c in is the number of convolution multiplications of the Winograd algorithm, and the ratio of the number of reduced multiplications to the number of original convolution multiplications is: According to the input feature map size formula of claim 1, further, formula (3) can be transformed into: This is the number of multiplications reduced by using the Winograd algorithm convolutional neural network accelerator and the number of multiplications of the original convolution. The ratio of times.