A hardware acceleration method, system and application of a convolutional neural network convolutional layer

Abstract

A hardware acceleration method, system and application of a convolutional neural network convolutional layer, the method comprising the following steps: converting a floating point number into a fixed point number and inputting it into an FPGA for calculation; using a small size convolution method based on a shift register and a depth separable convolution structure to reduce the calculation complexity; using a row fixed flow array structure for data flow, and hiding other layers before and after the convolutional layer; embedding a batch normalization layer, an activation function layer and a padding layer into the convolutional layer for hiding, and completing hardware acceleration. The application also proposes a hardware acceleration system of a convolutional neural network convolutional layer and an application of the hardware acceleration method of the convolutional neural network convolutional layer in a MobileNet model and a YOLOv2 model FPGA convolution hardware accelerator. The hardware acceleration method of the convolutional neural network convolutional layer has advantages in resource occupation, power consumption, unit frame rate and energy consumption ratio.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of convolutional neural network, and particularly relates to a hardware acceleration method, system and application of a convolutional neural network convolutional layer. BACKGROUND

[0002] Deep learning is usually aimed at unstructured data, i.e. image, audio, video and text, and other feature separation difficult data sources. It is very difficult for computers to process such data, and deep learning has brought a great breakthrough in the research field of target detection and recognition. Deep learning is usually composed of an input layer, a plurality of intermediate layers and an output layer. The input layer and the output layer are also called visible layers, and the data of the intermediate layers is difficult to be observed externally, so they are also called hidden layers. The hidden layer obtains data through the input layer, and then performs feature extraction on the data, including shallow features such as color and contour of pictures and abstract features such as shape and category. Deep learning divides a complex mapping task into a simple feature extraction process one by one through a layer of hidden layers. The more the number of hidden layers, the deeper the deep neural network, and the more abstract the extracted features.

[0003] Convolutional neural network is a typical supervised learning algorithm in deep neural network, which is a feedforward network with a plurality of intermediate hidden layers. The basic network structure is similar to that of neural network, which is composed of an input layer, a convolutional layer, a pooling layer, a fully connected layer and an output layer. Compared with other neural network models, convolutional neural network has the characteristics of local perception and parameter sharing. Among many classical networks of convolutional neural network, the lightweight network MobileNet greatly reduces the network model while not affecting the performance because of the deep separable convolution method proposed by it, and is suitable for image processing tasks on mobile terminals. The one stage algorithm is proposed in the YOLO series network, which realizes the positioning and recognition of the target in one step, and the processing speed of the image exceeds other algorithms, which is amazing. The v2 improves the accuracy on the basis of v1, and plays an important role in the field of real-time target monitoring.

[0004] Currently, there are several ways to accelerate convolutional neural networks (CNNs): GPUs, ASICs, and FPGAs. GPUs are the most widely used hardware accelerators, characterized by high data throughput. However, their power consumption is enormous, primarily serving cloud services for enterprises or individuals, and they cannot be used in energy-constrained devices. ASICs and FPGAs have limited storage resources and input / output bandwidth, but they can maintain performance without degradation while consuming very low power. ASIC designs are more specialized, but FPGAs have lower development costs and greater flexibility. FPGAs offer advantages such as flexible development, low power consumption, and high computation speed, while also possessing abundant storage resources. To achieve higher accuracy, CNNs are evolving towards greater complexity and depth, and parameter storage, data transfer, and convolution computation are gradually posing significant challenges to FPGAs. Therefore, a thorough study of the characteristics of convolution operations in CNNs and exploration of high-speed, low-power applications of CNNs in mobile devices is particularly important. Summary of the Invention

[0005] The purpose of this invention is to address the problems in the prior art by providing a hardware acceleration method, system, and application for convolutional neural network convolutional layers, which improves resource consumption, power consumption, unit frame rate, and energy efficiency.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] Firstly, a hardware acceleration method for convolutional layers of convolutional neural networks is proposed, including the following steps:

[0008] The floating-point number is converted into a fixed-point number and then input into the FPGA for calculation;

[0009] We employ a small-size convolution method based on shift registers and a depthwise separable convolution structure to reduce computational complexity;

[0010] The data stream uses a fixed-row array structure to hide other layers before and after the convolutional layer;

[0011] The batch normalization layer, activation function layer, and padding layer are all hidden within the convolutional layer to achieve hardware acceleration.

[0012] As a preferred embodiment of the acceleration method of the present invention, the step of converting floating-point numbers to fixed-point numbers involves converting the floating-point numbers to eight-bit fixed-point numbers, including one sign bit, m integer bits, and 7-m decimal bits. The loss incurred in converting floating-point numbers to fixed-point numbers satisfies the following:

[0013] As a preferred scheme of the acceleration method of the application, the small-size convolution method based on the shift register comprises: for an input image with a size of FxF and a convolution kernel with a size of CxC, filling serially input feature data into shift registers, and the total number of the shift registers is Fx(C-1)+F;

[0014] The CxC weight values are loaded at the same time as the feature data is loaded;

[0015] When all the shift registers are filled with the input feature map data, the data in the CxC shift registers are feature data for the first convolution calculation, at which time multiplication calculation is performed corresponding to the weight values, and the calculation result is sent to the addition tree for accumulation.

[0016] Further, as a preferred scheme of the acceleration method of the application, when the convolution kernel slides on the feature image, there is an overlapping part of the data used in adjacent two times of calculation, after the calculation is started and the convolution operation of the previous channel is ended, Fx(C-1)+C input feature data and the convolution kernel weight data corresponding to the current channel are loaded into two shift registers respectively, after the data is loaded, except for the invalid period of the convolution window sliding line change, one convolution multiplication operation is completed every clock, the feature data of each channel is input once, and the convolution operation of one channel is completed.

[0017] As a preferred scheme of the acceleration method of the application, the deep separable convolution structure, the calculation in one deep convolution layer is completed by one multi-channel convolution kernel, the convolution kernel and the input feature map are consistent in the depth direction, M channel feature maps are converted into N channel feature maps through N convolution kernels and through multiple two-dimensional convolution, and the output result of the deep convolution is a feature map with consistent channel number and input.

[0018] As a preferred scheme of the acceleration method of the application, in the row-fixed stream array structure, the feature map data, the convolution kernel weight and the partial sum are multiplexed by the row-fixed data stream, the convolution kernel data of each row is transmitted to multiple PE units in the horizontal direction to repeatedly participate in the calculation, the feature map data of each row is transmitted to multiple PE units in the diagonal direction to repeatedly participate in the calculation, and the partial sum of the output row data is accumulated in the vertical direction.

[0019] As a preferred scheme of the acceleration method of the application, the steps of embedding the batch normalization layer, the activation function layer and the padding layer into the hidden convolution layer comprise: the batch normalization layer limits the data transmitted between layers within a threshold range; the activation function layer directly activates the nonlinear output result of the convolution layer; and the padding layer performs padding operation on the current input feature map before the convolution calculation of each layer starts, and after the size of the feature map is supplemented by zero around, the convolution result of the current layer becomes the input feature size of the next layer convolution.

[0020] In a second aspect, a hardware acceleration system for a convolutional layer of a convolutional neural network is provided, comprising:

[0021] a data conversion module configured to convert a floating-point number into a fixed-point number and input the fixed-point number into the FPGA for calculation;

[0022] a calculation complexity simplification module configured to reduce calculation complexity by using a small-size convolution method based on a shift register and a depth separable convolution structure;

[0023] a data flow processing module configured to use a row fixed flow array structure for data flow and hide other layers before and after the convolutional layer;

[0024] a convolutional layer embedding and hiding module configured to embed a batch normalization layer, an activation function layer and a padding layer into the convolutional layer to complete hardware acceleration.

[0025] In a third aspect, the hardware acceleration method for the convolutional layer of the convolutional neural network is applied to a MobileNet model and a YOLOv2 model FPGA convolution hardware accelerator.

[0026] Compared with the prior art, the present application has at least the following beneficial effects:

[0027] The weight values obtained by training the convolutional neural network are mostly floating-point numbers. Since the storage resources of the FPGA are limited, floating-point operations are not suitable. In order to reduce calculation resources and facilitate the acceleration processing of the FPGA, the floating-point numbers are converted into fixed-point numbers and then input into the FPGA for calculation. The small-size convolution method based on the shift register and the depth separable convolution structure are used to reduce the calculation complexity, which can reduce the hardware resources required by the convolutional layer. The data flow uses a row fixed flow array structure to hide other layers before and after the convolutional layer. The batch normalization layer, the activation function layer and the padding layer are embedded into the convolutional layer, which reduces the data movement between layers, improves the circuit calculation rate and saves the cache resources. The hardware acceleration method for the convolutional layer of the convolutional neural network has advantages in resource occupation, power consumption, unit frame rate and energy consumption ratio. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 FIG. 1 is a schematic diagram of a standard convolutional layer structure;

[0029] Figure 2 FIG. 4 is a convolution calculation flowchart based on a shift register according to an embodiment of the present application;

[0030] Figure 3 FIG. 6 is a schematic diagram of a depth separable convolution calculation process according to an embodiment of the present application;

[0031] Figure 4A standard convolution operation parallel structure diagram for MobileNet;

[0032] Figure 5 A standard convolution module overall structure diagram;

[0033] Figure 6 A deep convolution operation module structure diagram;

[0034] Figure 7 A deep convolution module overall structure diagram;

[0035] Figure 8 A point convolution module overall structure diagram. DETAILED DESCRIPTION

[0036] In order to enable personnel in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor should fall within the scope of protection of the present application.

[0037] It should be noted that the terms "first", "second", and the like are used to distinguish similar objects, and do not necessarily have to be used to describe a particular order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] The hardware acceleration method for the convolutional layer of the convolutional neural network proposed in the embodiments of the present application includes the following steps:

[0039] S1, converting a floating-point number into a fixed-point number and inputting it into an FPGA for calculation;

[0040] S2, using a small-size convolution method based on a shift register and a depth separable convolution structure to reduce the calculation complexity;

[0041] S3, using a row fixed flow array structure for data flow, and hiding other layers before and after the convolution layer;

[0042] S4, embedding the batch normalization layer, the activation function layer and the padding layer into the convolution layer for hiding, and completing the hardware acceleration.

[0043] In step S1, since a large number of DSPs and memory resources are integrated in the FPGA, it is suitable to implement large convolution operations and intermediate calculation result storage in the convolutional neural network. Therefore, it is theoretically feasible to use the FPGA as a hardware accelerator of the convolutional neural network. The weight values obtained by training the convolutional neural network are mostly floating-point numbers. Since the FPGA storage resource is limited and not suitable for floating-point operation, in order to reduce the calculation resource and facilitate the acceleration processing of the FPGA, the floating-point numbers are converted into fixed-point numbers and then input into the FPGA for calculation. Considering that the parameters of the convolutional layer are mainly involved in multiplication and addition operations, and the multiplication operation is mainly implemented by calling the DSP resource in the FPGA, the bit number of the floating-point number parameter fixed-point conversion needs to be less than the bit width of the multiplier 18. Therefore, the parameter can be fixed-pointed to 8-bit or 16-bit decimal. Further, the embodiment comprehensively considers the FPGA storage resource problem and the precision loss problem, and fixes the data to eight-bit fixed-point decimal, including one sign bit, m integer bits and 7-m decimal bits. The loss generated by converting the floating-point number into the fixed-point number satisfies: loss≤1 / 2^(8-m).

[0044] A complete standard convolutional layer structure is as follows Figure 1 , including padding, convolution, batch normalization and ReLU activation four operations. After the feature map data is input into the convolutional layer, the zero operation is performed to expand the feature map size, then the convolution operation is performed with multiple convolution kernels, the batch normalization is performed on the obtained result, and finally the activation function is sent.

[0045] The parameters of the convolutional layer are mainly involved in multiplication and addition operations, and the multiplication operation is mainly implemented by calling the DSP resource in the FPGA. Therefore, the parameter can be fixed-pointed to 8-bit decimal. However, after the parameter fixed-point processing, a certain precision loss will be generated. The parameter is converted into eight-bit fixed-point decimal, including one sign bit, m integer bits and 7-m decimal bits. The loss generated by the parameter fixed-point conversion is:

[0046]

[0047] According to the loss expression generated by the above parameter fixed-point conversion, if the integer bit is set to 1 bit, the loss caused by the parameter fixed-point conversion is only about 0.0078, which does not cause too much influence on the precision of the neural network model.

[0048] For the convolution processing unit with a small convolution kernel size, the shift register is used to realize acceleration, as follows Figure 2As shown, the size of the input image in the figure is FxF, and the size of the convolution kernel is CxC. The serially input feature data is gradually filled into the shift register. The total number of shift registers is Fx(C-1)+F, and the CxC weight values are loaded at the same time as the feature data is loaded. When all the shift registers are filled with the input feature map data, the data in the CxC shift registers in the dashed box is the feature data for the first convolution calculation, at which time the multiplication calculation is performed corresponding to the weight values, and the calculation result is sent to the addition tree for accumulation. Taking a 3x3 size convolution kernel as an example, in the first clock cycle, the first three columns of data in the first three rows are selected, in the second clock cycle, the second to fourth columns of data in the first three rows are selected, and so on.

[0049] When the convolution kernel slides on the feature image, there is an overlapping part of the data used by adjacent two times of calculation, and most of the input feature data will be repeatedly multiplied with the convolution kernel data. The application uses a shift register, and this structure can fully realize the reuse of data. After the calculation just starts and the convolution operation of the previous channel ends, Fx(C-1)+C input feature data and the convolution kernel weight data corresponding to the current channel are loaded into the two shift registers, respectively. After the data loading is completed, except for the invalid period of the convolution window sliding line change, the multiplication operation of convolution can be completed once per clock, and the feature data of each channel only needs to be input once to complete the convolution operation of one channel.

[0050] The depth separable convolution can be regarded as two stages: 3x3 depth convolution and 1x1 point convolution. The depth convolution can be regarded as a filtering operation, and the calculation in a depth convolution layer only needs to use one multi-channel convolution kernel, and the convolution kernel and the input feature map remain consistent in the depth direction. That is, one channel of the feature map corresponds to one channel of the convolution kernel, and each channel of the feature is extracted by multiple two-dimensional convolutions, and after all channel convolutions are completed, no accumulation in the channel direction is needed, and no fusion in the channel direction is involved. The output result of the depth convolution is a feature map with the same number of channels as the input. Since the number of convolution kernels is only 1, the amount of calculation is significantly reduced. The point convolution can be regarded as a combination operation, and a 1x1 convolution kernel is used to expand and compress the input feature map in the depth direction, so as to keep the feature map size unchanged and change the number of feature map channels. Through N convolution kernels, M channel feature maps are converted into N channel feature maps.

[0051] The weight fixed data flow stores the convolution kernel data fixedly in the register of the PE unit, and maximizes the multiplexing of the convolution kernel. The output fixed data flow stores the partial sum generated in the convolution operation process in the register of the PE unit, and minimizes the energy consumption of reading and writing the partial sum. The non-local multiplexing data flow does not fix any data in the PE array, and the required data needs to be read from the global cache area every time, which increases the memory access time and is low in efficiency, and the advantage is that the area of the array is saved. The row fixed data flow multiplexes the feature map data, the convolution kernel weight and the partial sum, reduces the flow and movement of various data. The high-dimensional convolution operation using the row fixed data flow will be split into multiple one-dimensional convolutions, that is, one row of convolution kernel data acts on one row of feature map data for multiplication and accumulation calculation. The two-dimensional convolution is based on multiple one-dimensional convolutions, and the convolution kernel data is transmitted horizontally in the PE array, and the feature map data is arranged in the PE array and moves in the diagonal direction. Each column vertically accumulates the partial sum of all PE units and caches. From the above analysis, each row of convolution kernel data is transmitted horizontally to multiple PE units to participate in the calculation repeatedly, each row of feature map data is transmitted to multiple PE units in the diagonal direction to participate in the calculation repeatedly, and the partial sum of the output row data is accumulated vertically. The maximum data multiplexing rate is achieved, the number of PE array accessing the global buffer is reduced, and the time is saved and the energy consumption is reduced. The convolutional neural network with the row fixed flow structure has a performance 1.4-2.5 times higher than other structures, so the invention selects this data flow structure to design the convolutional neural network accelerator.

[0052] When the derivative of back propagation is propagated, the data after the activation function or the weight is easy to deviate to both ends of the effective value interval, and then gradient explosion or gradient disappearance occurs, causing the training speed to decrease. The batch normalization limits the data transmitted between layers to a certain range, so that the data is more stable in the process, thereby avoiding gradient explosion and disappearance. After the weight preprocessing, the batch normalization operation is fused into the convolution layer, and the hidden processing of the normalization operation is realized, thereby saving the storage resources and the operation time.

[0053] In order to reduce the movement of data between layers and save the cache resources, the hidden processing of the activation function can be used, and the activation function circuit is embedded in the pipeline of convolution calculation, that is, the result of the convolution layer accumulation output is directly nonlinearly activated.

[0054] Before the convolution calculation of each layer starts, padding operation is also needed for the current input feature map, and after the feature map size is supplemented by zero around the four sides, the convolution result of the current layer is the input feature size required by the next layer convolution. If the padding layer and the convolution layer are designed separately, that is, the output result of the previous layer convolution layer is stored, the padding layer reads and supplements zero again and then stores it as the input of the current layer, data moving between modules and modules is needed. In order to save storage resources, the padding operation and the convolution layer are fused and hidden, and the invalid output period when the line is changed is used to pad the convolution result with zero.

[0055] The invalid output period when the line is changed is used to pad the convolution result with zero. The rightmost side of the first row and the leftmost side of the second row of the feature map are padded.

[0056] Another embodiment of the application provides a hardware acceleration system for a convolutional neural network convolutional layer, comprising:

[0057] A data conversion module is configured to convert floating-point numbers into fixed-point numbers and input the fixed-point numbers into the FPGA for calculation.

[0058] A calculation complexity simplification module is configured to reduce the calculation complexity by using a small-size convolution method based on a shift register and a depth separable convolution structure.

[0059] A data flow processing module is configured to use a row fixed flow array structure for data flow, and to hide other layers before and after the convolutional layer.

[0060] A convolutional layer embedding and hiding module is configured to embed a batch normalization layer, an activation function layer and a padding layer into the convolutional layer to complete hardware acceleration.

[0061] Another embodiment of the application also provides an application of the hardware acceleration method for the convolutional neural network convolutional layer in a MobileNet model and a YOLOv2 model FPGA convolution hardware accelerator.

[0062] By using a small-size convolution unit, a depth separable convolution method and an operation hiding processing method, a convolutional layer accelerator for the MobileNet model is designed. The MobileNet model convolutional accelerator mainly includes a standard convolution module of a small-size convolution unit and a lightweight convolution module, an input control module of a depth convolution module and a point convolution module, a calculation module and an output control module. The standard convolution module and the depth convolution module in the MobileNet network model adopt similar convolution methods, that is, a small-size convolution unit based on 3*3. After simulation of the data input control module, the convolution operation module, the data output control module and the point convolution module, the feasibility of the small-size convolution method, the depth separable convolution module and the operation hiding processing method is verified.

[0063] The standard convolution layer is the first convolution layer of the MobileNet network, and the input of the layer is an image with three channels. The layer performs a standard convolution operation on the image, and the result is used as the input value of the subsequent layer. The standard convolution module first controls the reading of image data, convolution kernel data, and normalization weight through the data input control module, then performs feature data through the standard convolution operation module, and finally eliminates invalid data and controls the size reduction due to the convolution step through the data output control module. The first layer of MobileNet uses a small size convolution kernel of 3x3 to perform a standard convolution operation, so the RGB image needs to be zero-padded for one round. In hardware implementation, after the module enable signal is pulled high, in the first step, the image data reading signal remains low for F+2 clock periods to complete the zero padding of the top row; in the second step, the signal remains low for one clock period and then is pulled high, remains low for F clock periods and then is pulled high again, and remains low for one clock period, and this step is repeated F times to complete the left and right zero padding of F rows; finally, the first step is repeated to complete the zero padding of the bottom row. In this way, zero padding is performed while inputting a frame of image data. After zero padding, the size of the RGB image becomes 226x226. The layer requires 226x2+3 shift registers. When all the shift registers have obtained image data of the same channel, valid image data is started to be output.

[0064] There are 13 deep convolution layers in MobileNet, so in order to save FPGA resources, the deep convolution module is designed as a parameter configurable circuit, which can meet the calculation needs of 13 convolution layers at the same time, increasing the reusability of the circuit. The hardware flow of the deep convolution module is very similar to that of the standard convolution module, and both include a data input control module, a convolution operation unit, and a data output control module. The difference is that the data input control module of the deep convolution module needs to add an input path selector, and the input data of the first layer deep convolution and the second to thirteenth layer deep convolution come from the standard convolution module output buffer and the point convolution module output buffer respectively; another difference is that all deep convolution modules only need a single 3x3 size convolution kernel, and the convolution kernel is the same as the input in the depth direction, so the deep convolution module only has a 3x3 size convolution calculation unit, and there is no need to add all convolution results with the same coordinate position in the depth direction.

[0065] Point convolution is paired with deep convolution, and the output feature map of deep convolution is fused in the channel direction. Consistent with deep convolution, point convolution has 13 layers, so it also needs configurable design, and the module structure is similar to standard convolution and deep convolution, which is composed of data input control module, point convolution operation module and data output control module. The difference is that: (1) the weight parameter of point convolution is much larger than that of standard convolution and deep convolution, so the weight parameter of point convolution is saved in DDR and needs to be read through bus; (2) point convolution does not need to perform padding zero padding operation.

[0066] The structure of the YOLOv2 model convolution accelerator borrows the row fixed data flow idea proposed by Eyeriss architecture, and designs a size configurable array based on PE unit. By controlling the input format of data, three-dimensional convolution is decomposed into one-dimensional convolution, realizing the convolution calculation circuit of multiple feature maps, multiple convolution kernels and multiple channels in parallel.

[0067] The convolution layer accelerator of YOLOv2 model is based on PE array structure and adopts row fixed data flow design. The PE array is simulated and verified, which verifies the feasibility of the PE array convolution structure adopting row fixed flow idea.

[0068] Two-dimensional convolution is composed of multiple one-dimensional convolution operations, and the summation result of each column of PE array is the data of the corresponding column in the output feature map. In the above calculation process, the row data of convolution kernel weight is input in a cycle, which is repeatedly used horizontally between PEs, and the row data of feature map is repeatedly used between the diagonal lines of PE after rearrangement. Convolution kernel weight data and feature map data from different rows are sent to PE unit in a time interleaved manner. All PE units that use the same value repeatedly will receive data at the same time. Moreover, there will be an accumulator after each column of PE unit to complete all accumulation operations. That is, all PE units only undertake the multiplication part in the calculation process, and the multiplication result is sent to the accumulator immediately.

[0069] The convolution layer main part circuit of the two models is accelerated based on the xcvu9p-flga2104-2L-e model of the Virtex Ultrascale+ series development board, and hardware synthesis is implemented and analyzed by using the Vivado development environment. The energy efficiency ratio of the accelerator is 0.145 fps / W and 0.309 fps / W, that is, the ratio of unit frame rate to power consumption. Table 1 is a performance comparison table of the present application and other existing accelerators, as shown in Table 1, the MobileNet and YOLOv2 FPGA convolution hardware accelerator in the present application is superior to the accelerator of other acceleration platforms, and is also superior to the MobileNet and YOLOv2 accelerator based on other FPGAs. The accelerator designed by the method of the present application has advantages in resource occupation, power consumption, unit frame rate and energy consumption ratio.

[0070] Table 1

[0071]

[0072] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A hardware acceleration method of a convolutional neural network convolutional layer, characterized in that, The method comprises the following steps: Convert the floating point number into a fixed point number and input into the FPGA for calculation; The calculation complexity is reduced by using a small-size convolution method based on a shift register and a deep separable convolution structure; The data stream adopts a row fixed flow array structure; The other layers before and after the convolution layer are hidden, the batch normalization layer, the activation function layer and the padding layer are embedded in the convolution layer for hiding, and hardware acceleration is completed; The small-size convolution method based on the shift register comprises: For an input image with a size of FxF and a convolution kernel with a size of CxC, the serially input feature data is filled into the shift register, and the total number of the shift registers is Fx(C-1)+F; The CxC weight values are loaded at the same time as the feature data is loaded; When all the shift registers are filled with the input feature map data, the data in the CxC shift registers are the feature data for the first convolution calculation, multiplication calculation is performed on the data corresponding to the weight values at this time, and the calculation results are sent to an addition tree for accumulation; When the convolution kernel slides on the feature image, the data used for adjacent two times of calculation have an overlapping part, after the convolution operation of the previous channel is completed, Fx(C-1)+C input feature data and the convolution kernel weight data corresponding to the current channel are loaded into two shift registers, after the data loading is completed, one convolution multiplication operation is completed every clock, the feature data of each channel is input once, and the convolution operation of one channel is completed; In the step of embedding the batch normalization layer, the activation function layer and the padding layer in the convolution layer for hiding, the batch normalization layer limits the data transmitted between layers within a threshold range; the activation function layer directly activates the nonlinear output result of the convolution layer; and the padding layer performs padding operation on the current input feature map before the convolution calculation of each layer starts, and the convolution result of the current layer becomes the input feature size of the next layer convolution after the feature map size is supplemented with zeros around.

2. The method of claim 1, wherein, The step of converting the floating-point number into the fixed-point number is converting the floating-point number into an eight-bit fixed-point decimal number, which includes one sign bit, m integer bits and 7-m decimal bits. The loss generated by converting the floating-point number into the fixed-point number satisfies: .

3. The method of claim 1, wherein the method further comprises: The deep separable convolution structure, the calculation in one deep convolution layer is completed by one multi-channel convolution kernel, the convolution kernel and the input feature map are consistent in the depth direction, M-channel feature maps are converted into N-channel feature maps through N convolution kernels and multiple two-dimensional convolution, and the output result of the deep convolution is a feature map with consistent channel number and input.

4. The method of claim 1, wherein, The row fixed data stream in the row fixed flow array structure multiplexes the feature map data, the convolution kernel weight and the partial sum, the convolution kernel data of each row is transmitted to multiple PE units in the horizontal direction to repeatedly participate in the calculation, the feature map data of each row is transmitted to multiple PE units in the diagonal direction to repeatedly participate in the calculation, and the partial sum of the output row data in the vertical direction is accumulated.

5. A hardware acceleration system for a convolutional neural network convolutional layer, the system comprising: The method comprises the following steps: The data conversion module is used for converting the floating point number into a fixed point number and inputting into the FPGA for calculation; The calculation complexity simplification module is used for reducing the calculation complexity by using a small-size convolution method based on a shift register and a deep separable convolution structure; The data stream processing module is used for adopting a row fixed flow array structure for the data stream; The convolution layer embedding hidden module is used for embedding other layers before and after the convolution layer to hide batch normalization layers, activation function layers and padding layers, embedding the batch normalization layers, the activation function layers and the padding layers in the convolution layer to hide, and completing hardware acceleration. The small-size convolution method based on the shift register comprises the following steps: For an input image with a size of FxF and a convolution kernel with a size of CxC, serially input feature data is filled into the shift register, and the total number of the shift register is FxC-1+F; CxC weight values are loaded at the same time as the feature data is loaded; When all the shift registers are filled with the input feature map data, the data in the CxC shift registers is the feature data for the first convolution calculation, multiplication calculation is performed corresponding to the weight values at this time, and the calculation result is sent to the addition tree for accumulation; When the convolution kernel slides on the feature image, there is an overlapping part for the data used in adjacent two times of calculation, after the calculation just starts and the convolution operation of the previous channel ends, FxC-1+C input feature data and the convolution kernel weight data corresponding to the current channel are loaded into two shift registers respectively, after the data loading is completed, except for the invalid period of the convolution window sliding line change, multiplication operation of convolution is completed once per clock, the feature data of each channel is input once, and the convolution operation of one channel is completed. In the step of embedding the batch normalization layer, the activation function layer and the padding layer in the convolution layer to hide, the batch normalization layer limits the data transmitted between layers within a threshold range; the activation function layer directly activates the nonlinear of the convolution layer accumulation output result; and the padding layer performs padding operation on the current input feature map before convolution calculation of each layer starts, and after the size of the feature map is supplemented by zero around, the convolution result of the current layer becomes the input feature size of the next layer convolution.

6. Application of the hardware acceleration method of the convolution layer of the convolutional neural network according to any one of claims 1-4 in a MobileNet model, a YOLOv2 model FPGA convolution hardware accelerator.

Images