A target detection neural network compression method and system based on FPGA

Abstract

The application provides a compression method and system of a target detection neural network based on FPGA, and comprises the following steps: using an improved SqueezeNet network for target detection; the improved SqueezeNet network comprises 6 layers of Fire modules; and the improved SqueezeNet network is deployed in the FPGA memory in a folding calculation mode based on a multiplexer and a multi-selector. The application uses 6 layers of Fire layers as a main framework, avoids the problem of poor parallelism caused by the original network with deeper network depth for reducing training parameters, and reduces the calculation time by reducing the calculation amount of the whole network.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of neural network acceleration and compression, and particularly relates to a target detection neural network compression method and system based on FPGA. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] With the rapid development of visual acquisition technology, image frame frequency and resolution are getting higher and higher, and target recognition is applied to the fields of unmanned driving and robot vision, and has a solid foundation. The introduction of convolutional neural networks enables target detection technology to meet the accuracy requirements of most application scenarios, but at the same time, it also brings problems such as high computational complexity and high spatial complexity. Traditional general-purpose processors cannot meet the high real-time application scenarios, especially in embedded platforms with limited computing power, storage space and running power consumption. The current mainstream solution is embedded GPU, but due to its computing architecture, the system has low performance per watt (PPW). FPGA, as a kind of semi-custom integrated circuit, has natural advantages in power consumption, flexibility and parallelism, so the target detection acceleration technology based on FPGA has become one of the important research topics.

[0004] Field Programmable Gate Array (FPGA) is a kind of hardware reconfigurable device based on lookup table, which is often used for small batch of alternative application specific integrated circuit (ASIC). Compared with traditional CPU and GPU based on von Neumann architecture, FPGA has great operation bandwidth and data transmission rate due to its unique architecture of not relying on instruction set, non-shared memory and parallel operation mode. At the same time, because of the hardware reconfigurability of FPGA, users can freely define the hardware structure and data interface, so as to realize the purpose of deploying the required algorithm on FPGA. Therefore, FPGA as an accelerator has become the only choice to improve the real-time performance and energy consumption ratio of target detection system.

[0005] SqueezeNet is born in this environment, it can achieve the effect of AlexNet on the ImageNet dataset, but the parameters are less than 50 times of AlexNet. SqueezeNet designs Fire module to reduce the training parameters, combined with the convolution layer in the convolution network, the down-sampling layer, the full connection and other layer composition network. A Fire module is composed of Squeeze and Expand parts. The Squeeze part is a group of consecutive 1X1 convolution, and the Expand part is composed of a group of consecutive 1X1 convolution and a group of consecutive 3X3 convolution cancatnate.

[0006] However, SqueezeNet also has disadvantages, first, the application direction of the network is embedded environment, and the main problem of the embedded environment is real-time. SqueezeNet replaces the number of parameters by deeper depth, although it can reduce the number of network parameters, but it loses the parallel ability of the network, and the test time will be longer, which is contrary to the main challenge. In addition, the down-sampling moves to the classification layer, although it can improve the accuracy of the network, but it has a very serious disadvantage: it increases the computational complexity of the network. SUMMARY

[0007] In order to overcome the above-mentioned deficiencies of the prior art, the present application provides a target detection neural network compression method and system based on FPGA, which modifies the original network of eight layers of Fire to six layers, avoids the problem of poor parallel ability caused by the deeper network depth of the original network to reduce the training parameters, and reduces the calculation time by reducing the computational complexity of the entire network.

[0008] In order to achieve the above-mentioned purpose, one or more embodiments of the present application provide the following technical solutions: a target detection neural network compression method based on FPGA, comprising the following steps:

[0009] The improved SqueezeNet network is used for target detection, and the improved SqueezeNet network comprises six layers of Fire modules.

[0010] The improved SqueezeNet network is deployed in the FPGA memory based on the folding calculation through the multiplexer and the multiplexer.

[0011] Further, the improved SqueezeNet network adopts front sampling, and the down-sampling is performed after the first Fire module and the second layer Fire module.

[0012] Further, the Fire module includes a squeeze part and an expand part, and the number ratio of 1x1 convolution kernels of the squeeze part, 1x1 convolution kernels of the expand part and 3x3 convolution kernels is set to 1:1:2.

[0013] Further, in one layer of the Fire module, the number of parameters num prams is calculated by the following formula:

[0014] num params =∑(ConvS*ConvS*ConvCH+1)×ConvCount

[0015] wherein, ConvS is the size of the convolution kernel, ConvCH is the number of channels of the convolution kernel, and ConvCount is the number of convolution kernels.

[0016] Further, three multiplexers and three multi-output selectors are used, and the multiplexers and the multi-output selectors are configured in a folding calculation manner to realize the calculation of different stages of the whole neural network containing the improved SqueezeNet.

[0017] The second aspect of the application provides a compression system of a target detection neural network based on FPGA, which comprises: a network structure module: an improved SqueezeNet network is used for target detection; the improved SqueezeNet network comprises six layers of Fire modules;

[0018] a storage module: the improved SqueezeNet network is deployed in the FPGA memory in a folding calculation manner based on multiplexers and multi-output selectors.

[0019] Further, in the network structure module, the Fire module includes a squeeze part and an expand part, and the number ratio of 1x1 convolution kernels of the squeeze part, 1x1 convolution kernels of the expand part and 3x3 convolution kernels is set to 1:1:2.

[0020] Further, in the storage module, three multiplexers and three multi-output selectors are used, and the multiplexers and the multi-output selectors are configured in a folding calculation manner to realize the calculation of different stages of the whole neural network containing the improved SqueezeNet.

[0021] The third aspect of the application provides a computer readable storage medium for storing computer instructions, wherein the computer instructions are executed by a processor to complete the steps of the above method.

[0022] The fourth aspect of the present application provides an electronic device comprising a memory and a processor and computer instructions stored on the memory and running on the processor, when the computer instructions are run by the processor, the steps of the above method are completed.

[0023] The above one or more technical solutions have the following beneficial effects:

[0024] In the present application, the SqueezeNet network is modified, the eight layers of Fire in the original network are modified to 6 layers, the problem of poor parallelism caused by the deeper network depth of the original network to reduce the training parameters is avoided, the calculation time is reduced by reducing the calculation amount of the whole network.

[0025] In the present application, pre-downsampling is adopted, although larger feature maps contain more information, pre-downsampling can improve network accuracy, but in the embedded environment, the biggest challenge is to achieve high computing performance under the condition of limited bandwidth and limited hardware resources, larger feature maps will increase the training parameters, thereby increasing the calculation amount of the whole network, pre-downsampling reduces the network calculation amount at the expense of a small accuracy.

[0026] In the present application, the SqueezeNet network reduces the number of 3x3 convolution input channels through the Squeeze part in the Fire module, reduces the model training parameters, and can ensure the accuracy of the network while reducing the training parameters.

[0027] The advantages of the additional aspects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be known by the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0028] The drawings accompanying the specification of the present application serve to provide further understanding of the present application, the illustrative embodiments of the present application and the description thereof serve to explain the present application, and do not constitute an improper limitation on the present application.

[0029] Figure 1 is a schematic diagram of an existing Squeezenet network structure;

[0030] Figure 2 is a schematic diagram of the improved Squeezenet network structure in the first embodiment of the present application;

[0031] Figure 3 is a schematic diagram of the HLS design in the first embodiment of the present application;

[0032] Figure 4 is a schematic diagram of the network structure in the first embodiment of the present application. DETAILED DESCRIPTION

[0033] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0034] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the application.

[0035] In the case without conflict, the embodiments in the application and the features in the embodiments can be combined with each other.

[0036] Embodiment one

[0037] As shown in the Figures 2-3 The embodiment discloses a compression method of a target detection neural network based on FPGA, and has the characteristics that the method comprises the following steps:

[0038] An improved SqueezeNet network is used for target detection; the improved SqueezeNet network comprises six Fire modules;

[0039] The improved SqueezeNet network is deployed in the FPGA memory based on the folding calculation through the multiplexer and the multi-selector.

[0040] In the existing SqueezeNet network, three strategies are used for model compression:

[0041] 1. Replace 3x3 convolution with 1x1 convolution. Through this step, the number of parameters of one convolution operation is reduced by 9 times.

[0042] 2. Reduce the channel number of 3x3 convolution: the calculation amount of one 3x3 convolution is 3x3xMxN (M and N are the channel numbers of the input Feature Map and the output Feature Map respectively), so the calculation amount is too large, and therefore it is hoped that M and N are reduced to reduce the number of parameters.

[0043] 3. Postpone downsampling: a larger Feature Map contains more information, so the downsampling is moved to the classification layer. In order to ensure the accuracy of the network, the method of delaying downsampling is adopted, that is, the downsampling is moved to the classification layer, which will increase the calculation amount of the network; in addition, SqueezeNet replaces fewer parameters through a deeper network depth, loses the parallelism of the network, and the calculation time is longer, which does not meet the real-time requirement.

[0044] In the embodiment, the SqueezeNet network is modified under the condition of limited FPGA bandwidth and limited hardware resources, the network depth is reduced, eight layers of Fire in the original network are modified to six layers, and the network calculation amount is reduced with a small precision cost through the pre-degradation sampling mode.

[0045] As shown in Figure 1 , the existing SqueezeNet structure contains eight fire modules, and there are three pooling layers respectively marked as pooling layers A, B and C. There are three fire modules between the pooling layers A and B, four fire modules between the pooling layers B and C, and one fire module connected after the pooling layer C.

[0046] In the embodiment, six layers of Fire layers are used as the main framework, one fire module is arranged between the pooling layers A and B, one fire module is arranged between the pooling layers B and C, and four fire modules are connected after the pooling layer C.

[0047] Specifically, according to Figure 2 , the entire calculation process can be divided into six stages.

[0048] The first stage is the first five blocks, that is, from Resize to the first pooling Poola, the input size is 224x224x3, and the output size is 28x28x96. The channel number here can be understood as the third dimension of a three-dimensional matrix;

[0049] The second stage is the 6th block and the 7th block, one Fire and one poolb, the input size is 28x28x96, and the output is 14x14x96;

[0050] The third stage is the eighth block, one fire, and the input and output are both 14x14x96;

[0051] The fourth stage is the ninth block, one fire, and the input and output are both 14x14x96;

[0052] The fifth stage is the tenth block, one fire, and the input and output are both 14x14x96;

[0053] The sixth stage is the remaining part, the input is 14x14x96, and the output has two, and we need the 14x14x4 frame information.

[0054] In the embodiment, the SqueezeNet network reduces the 3x3 convolution input channel number through the Squeeze part in the Fire module, reduces the model training parameters, and the single-step speed is faster. At the same time, because the parameters are reduced, the model file is also reduced, which is more conducive to the saving and transmission of the model.

[0055] Specifically, the fire module includes a Squeeze part and an Expand part, the number ratio of the 1x1 convolution kernel of the squeeze part, the 1x1 convolution kernel and the 3x3 convolution kernel of the Expand part is set to 1:1:2. Not only can the number of parameters of one-step convolution be reduced by 20%, but also, because the Expand part retains more 3x3 convolution, because it is composed of 1x1 and 3x3 convolution, the ratio is 1:2, instead of 1:1, the more 3x3 convolution kernels, the greater the accuracy of the network, so here, more 3x3 convolution is retained compared with the 1:1 ratio. And the accuracy of the network will not decrease too much. The purpose of designing the Fire module is to reduce the training parameters, the number of parameters num params The calculation method is as follows:

[0056] num params =∑(ConvS*ConvS*ConvCH+1)×ConvCount (1)

[0057] Wherein, ConvS is the size of the convolution kernel, ConvCH is the number of channels of the convolution kernel, and ConvCount is the number of convolution kernels.

[0058] In the embodiment, the Fire module is used to replace the simple 3x3 convolution, the 1x1 convolution kernel of the Squeeze part is used to reduce the dimension of the input image data, and then the 1x1 convolution and 3x3 convolution combination of the Expand part is used to increase the dimension. In the Expand, there are 1x1 and 3x3, to output 96-dimensional data, 96 convolution kernels are needed, here, some 3x3 convolutions are replaced by 1x1, so that the number of 3x3 convolution kernels is reduced, that is, the ConvCount of the formula is reduced, so that the parameters of one Fire are reduced.

[0059] The Fire module proposed in the embodiment reduces the network by 2% compared with the network using 3x3 convolution in the Expand. However, due to the reduction of the training parameters, the difference in training time between the two networks is very obvious. The training time of one round is reduced by 50% compared with the original network, the calculation speed is greatly accelerated, the network calculation performance is effectively improved, the data processing throughput of the convolution operation basic operation unit is matched with the bandwidth provided by the FPGA platform.

[0060] The existing SqueezeNet network structure uses the method of delayed downsampling to obtain more information in the Feature Map, and performs downsampling after the third and seventh Fire, respectively. This brings the problem of greatly increasing the network calculation amount; in addition, SqueezeNet uses network depth to replace smaller network parameters, so that the network loses parallelism and seriously reduces real-time performance.

[0061] In this embodiment, in order to enable the network to be better deployed in an embedded environment, a downsampling layer is arranged in advance, and downsampling is performed after the first and second Fire, so that the image is reduced when the subsequent Fire operation is performed, the number of parameters to be calculated is reduced, and the purpose of compressing the model is achieved.

[0062] In the target recognition task, a standard for measuring the accuracy of detecting corresponding objects is IoU (Intersection over Union), that is, the intersection ratio. The IoU calculates the ratio of the intersection and the union of the "predicted bounding box" and the "real bounding box". The formula is:

[0063]

[0064] Among them, Intersection is the size of the intersection of the "predicted bounding box" and the "real bounding box", and Union is the size of the union of the "predicted bounding box" and the "real bounding box".

[0065] It is verified that after training on the GPU platform Tesla T4, the IoU of the modified network decreases by 1% compared with the IoU of the 8-layer Fire network, but the training time of the target detection network and the inference time on the FPGA platform are greatly shortened due to the reduction of network depth and training parameters, and the network model is greatly reduced due to the reduction of network parameters. Therefore, the new network architecture designed in the present application reduces the network depth at a very low precision cost, compresses the network model, speeds up the calculation, improves the calculation performance, enhances the real-time performance, and makes the network more suitable for deployment on the FPGA platform with limited bandwidth and limited hardware resources.

[0066] Limited by the computing resources and on-chip storage resources of the FPGA chip, the embodiment adopts a folded computing mode to deploy the network on the FPGA platform with limited resources.

[0067] As shown in Figures 3-4 Folded Computing is used to configure multiplexers and multi-output selectors to implement the calculation of different stages of the neural network. The embodiment uses three multiplexers and three multi-output selectors to divide the entire neural network into six times for calculation by selecting data streams. Specifically:

[0068] 1. DemuxStream2_0 divides Input into two outputs, output 1 Conv2D and output 2 MuxStream2_0;

[0069] 2. MuxStream2_0, selects two inputs, which are output 2 of DemuxStream2_0 and the result of Pooling, respectively;

[0070] 3. DemuxStream3, divides the result of Fire into three outputs, output 1 Pool2D1, output 2 Pool2D2, and output 3 directly to MuxStream3;

[0071] 4. MuxStream3, selects three inputs, which are the result of Pooling 1, the result of Pooling 2, and output 3 of DemuxStream3, respectively;

[0072] 5. DemuxStream2, takes the output of MuxStream3 as input, and divides it into two outputs, output 1 directly to MuxStream2, and output 2 to the classification and prediction part;

[0073] 6. MuxStream2, selects two inputs, which are the result after classification and prediction, and output 1 of DemuxStream2, respectively.

[0074] The operation of each stage is as follows:

[0075] (1) input the weights of Fire1, input the picture to be detected, the network input image size is 224x224x3, pass through the Conv2D layer with a convolution kernel size of 3x3x32, and pool. Then pass through the Fire1 module, and then pool. The control data flow is Input->Conv2D->Pool->Fire->Pool->Output, and the intermediate result 1 is sent back to the CPU;

[0076] (2) input the weights of Fire2, input the intermediate result 1, pass through the Fire2 module, and then pool. The control data flow is Input->Fire->Pool->Output, and the intermediate result 2 is sent back to the CPU;

[0077] (3) input the weights of Fire3, input the intermediate result 2, pass through the Fire3 module. The control data flow is Input->Fire->Output, and the intermediate result 3 is sent back to the CPU;

[0078] (4) input the weights of Fire4, input the intermediate result 3, pass through the Fire4 module. The control data flow is Input->Fire->Output, and the intermediate result 4 is sent back to the CPU;

[0079] (5) Input the weights of Fire5, input the intermediate result 4, and pass through the Fire5 module. The control data flow direction is Input->Fire->Output, and the intermediate result 5 is sent back to the CPU;

[0080] (6) Input the weights of Fire6, input the intermediate result 5, and then pass through the Fire6 module, and then pass through the 1x1 convolution layer ConvObj. After merging with the original input, it is respectively input into the convolution layer ConvObj and ConvBox. Finally, the maximum value selection is performed in the ConvObj layer activation, and on this basis, the maximum value selection is performed on the ConvBox layer activation. The control data flow direction is Input->Fire->ConvClass->ConvBox, ConvObj->SelectMax->Output, and the detection result is sent back to the CPU.

[0081] Through the multiplexer and the multi-output selector, the designed network architecture can be deployed on the FPGA platform in a folded computing manner. The Fire operation is only implemented once in one calculation, and the computing resources and storage resources of other Fire modules are used for parallel calculation, thereby fully utilizing the high parallelism characteristics of the FPGA and better adapting to the FPGA platform.

[0082] Embodiment Two

[0083] The purpose of this embodiment is to provide a compression system of a target detection neural network based on FPGA, which comprises:

[0084] The network structure module: an improved SqueezeNet network is used for target detection; the improved SqueezeNet network comprises 6 layers of Fire modules;

[0085] The storage module: the improved SqueezeNet network is deployed in the FPGA memory based on the folded computing manner through the multiplexer and the multi-selector.

[0086] In the network structure module, the Fire module comprises a squeeze part and an expand part, and the number ratio of the 1x1 convolution kernel of the squeeze part, the 1x1 convolution kernel and the 3x3 convolution kernel of the expand part is set to 1:1:2.

[0087] In the storage module, three multiplexers and three multi-output selectors are used, and the multiplexers and the multi-selectors are configured in a folded computing manner to realize the calculation of different stages of the entire neural network containing the improved SqueezeNet.

[0088] Embodiment Three

[0089] The object of this embodiment is to provide a computer readable storage medium.

[0090] Embodiment four

[0091] The object of this embodiment is to provide a computer readable storage medium.

[0092] A computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the above method.

[0093] The steps and methods involved in the above embodiments two, three and four correspond to embodiment one, and the detailed description can be found in the relevant description part of embodiment one. The term "computer readable storage medium" should be understood as including a single medium or multiple media of one or more instruction sets; it should also be understood as including any medium which can store, encode or carry instruction sets for execution by a processor and make the processor perform any of the methods in the present application.

[0094] Those skilled in the art should understand that the above modules or steps of the present application can be realized by a general computer device, alternatively, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device for execution by a computing device, or they can be respectively made into individual integrated circuit modules, or a plurality of modules or steps among them can be made into a single integrated circuit module. The present application is not limited to any specific combination of hardware and software.

[0095] The above describes the specific embodiments of the present application in combination with the accompanying drawings, but is not a limitation on the protection scope of the present application, and those skilled in the art should understand that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. A compression method of an FPGA-based target detection neural network, characterized in that, The method comprises the following steps: The improved SqueezeNet network is used for target detection; the improved SqueezeNet network comprises six layers of Fire modules, the Fire module comprises a squeeze part and an expand part, and the number ratio of the 1*1 convolution kernel of the squeeze part, the 1*1 convolution kernel and the 3*3 convolution kernel of the expand part is set to 1:1:2; The improved SqueezeNet network is deployed in the FPGA memory based on the folding calculation through the multiplexer and the multi-selector, three multiplexers and three multi-output selectors are used, the multiplexer and the multi-selector are configured in the folding calculation mode to realize the calculation of different stages of the whole neural network of the improved SqueezeNet, the whole neural network is divided into six times for calculation through the selection of the data stream, and the specific calculation process is as follows: (1) DemuxStream2_0, the Input is divided into two outputs, output 1 Conv2D and output 2 MuxStream2_0; (2) MuxStream2_0, two inputs are selected, which are the output 2 of DemuxStream2_0 and the result of pooling; (3) DemuxStream3, the result after the Fire is divided into three outputs, output 1 Pool2D1, output 2 Pool2D2 and output 3 directly sent to MuxStream3; (4) MuxStream3, three inputs are selected, which are the result of pooling 1, the result of pooling 2 and the output 3 of DemuxStream3; (5) DemuxStream2, the output of MuxStream3 is taken as the input and divided into two outputs, output 1 directly sent to MuxStream2 and output 2 sent to the classification and prediction part; (6) MuxStream2, two inputs are selected, which are the result after the classification and prediction and the output 1 of DemuxStream2.

2. The compression method of a target detection neural network based on FPGA according to claim 1, wherein, The improved SqueezeNet network adopts front sampling, and the down-sampling is performed after the first Fire module and the second layer Fire module.

3. The compression method of a target detection neural network based on FPGA according to claim 1, wherein, Number of parameters in a one-layer Fire module The calculation method is: Wherein, ConvS is the size of the convolution kernel, ConvCH is the number of channels of the convolution kernel, and ConvCount is the number of convolution kernels.

4. A compression system for an FPGA-based object detection neural network, the system comprising: The method comprises the following steps: The network structure module: the improved SqueezeNet network is used for target detection; the improved SqueezeNet network comprises six layers of Fire modules, the Fire module comprises a squeeze part and an expand part, and the number ratio of the 1*1 convolution kernel of the squeeze part, the 1*1 convolution kernel and the 3*3 convolution kernel of the expand part is set to 1:1:2; The storage module: the improved SqueezeNet network is deployed in the FPGA memory based on the folding calculation through the multiplexer and the multiplexer, adopts three multiplexers and three multiplexers, adopts the folding calculation to configure the multiplexer and the multiplexer, realizes the calculation of different stages of the whole neural network containing the improved SqueezeNet, divides the whole neural network into 6 times for calculation through the selection of data flow, and the specific is: (1) DemuxStream2_0, divides Input into two outputs, output 1 Conv2D, and output 2 MuxStream2_0; (2) MuxStream2_0, selects two inputs, which are output 2 of DemuxStream2_0 and the result of pooling; (3) DemuxStream3, divides the result after Fire into three outputs, output 1 Pool2D1, output 2 Pool2D2, and output 3 directly sent to MuxStream3; (4) MuxStream3, selects three inputs, which are the results of pooling 1, pooling 2 and output 3 of DemuxStream3; (5) DemuxStream2, divides the output of MuxStream3 as input into two outputs, output 1 directly sent to MuxStream2, and output 2 sent to the classification and prediction part; (6) MuxStream2, selects two inputs, which are the results after classification and prediction and output 1 of DemuxStream2.

5. The compression system of a target detection neural network based on FPGA according to claim 4, wherein, The improved SqueezeNet network adopts front sampling, and performs down sampling after the first Fire module and the second layer Fire module.

6. The compression system of a target detection neural network based on FPGA according to claim 4, wherein, Number of parameters in a one-layer Fire module The calculation method is: Wherein, ConvS is the size of the convolution kernel, ConvCH is the number of channels of the convolution kernel, and ConvCount is the number of convolution kernels.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that The program is executed by the processor to realize the steps in the compression method of the FPGA-based target detection neural network in any one of claims 1-3.

8. A processing device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the steps in the compression method of the FPGA-based target detection neural network in any one of claims 1-3.

Images