FPGA-based u-net network accelerator

Abstract

The application discloses a U-Net network accelerator based on FPGA, and belongs to the fields of embedded systems and signal processing; a DDR interface is connected with an input cache module and a weight FIFO module through an input module; the input cache module and the weight FIFO module are connected with a convolution pooling module, and are used for storing feature map data required by current calculation; the convolution pooling module is connected with an output module through an output cache module; the output module is connected with the DDR interface, and is used for storing results obtained by convolution or pooling calculation; and the weight FIFO module is used for storing weight parameters required by current calculation; the application improves the utilization rate of internal hardware resources of FPGA, increases the parallel degree of the convolution operation hardware accelerator, and improves the overall operation performance of the hardware system.

Description

TECHNICAL FIELD

[0001] The application relates to the field of embedded systems and signal processing, and in particular to a convolutional neural network accelerator based on FPGA. BACKGROUND

[0002] With the rapid development of deep learning, neural network algorithms are widely used in the field of image processing. Image semantic segmentation plays an important part in image processing and machine vision technology. Compared with traditional convolutional neural networks, semantic segmentation networks contain both up-convolution and down-convolution for image reconstruction. Due to the limitation of hardware computing power, neural network hardware accelerators based on FPGA have been proposed.

[0003] As a kind of semantic segmentation network, the operation bottleneck of U-Net network is also convolution operation. By using hardware circuit design methods such as loop unrolling and loop pipelining, the parallelism of the convolution operation hardware accelerator is increased by improving the utilization of FPGA internal hardware resources, and the overall operation performance of the hardware system is improved. SUMMARY

[0004] The application aims to provide a U-Net network acceleration method based on FPGA, which solves the problems of low utilization of FPGA hardware resources and poor overall operation performance.

[0005] The application provides a U-Net network accelerator based on FPGA, which comprises:

[0006] a convolution pooling module, an input module, an output module, an input cache module, an output cache module and a weight FIFO module;

[0007] The DDR interface is connected to the input cache module and the weight FIFO module through the input module, and the input cache module and the weight FIFO module are connected to the convolution pooling module, and are used for storing feature map data required for current calculation;

[0008] The product pooling module is connected to the output module through the output cache module, and the output module is connected to the DDR interface, and is used for storing the results obtained by convolution or pooling calculation;

[0009] The weight FIFO module is used for storing the weight parameters required for current calculation.

[0010] Further, the convolution pooling module comprises a convolution calculation module and a pooling calculation module, and the convolution calculation module comprises a convolution kernel and a fused convolution layer.

[0011] Further, the fused convolution layer is formed by fusing a convolution layer and a normalization layer, which improves the running efficiency of the hardware IP and reduces the overall running time.

[0012] The convolution calculation module adopts a systolic array calculation acceleration method.

[0013] Further, the network accelerator includes an on-chip memory BRAM as a cache, and after starting, data is transmitted to the on-chip memory BRAM, reducing data exchange between the FPGA and an external storage medium.

[0014] Further, FIFO is used as an external RAM weight data cache interface, and a dynamic depth configuration FIFO is designed.

[0015] Compared with the prior art, the beneficial effects of the present application are:

[0016] Compared with the method using model compression processing, the method can improve operation efficiency, reduce resource consumption, shorten development period, save development cost, and effectively accelerate the U-Net network. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to facilitate the description, the present application is described in detail by the following specific embodiments and drawings.

[0018] Figure 1 The figure is a schematic diagram of the overall structure of the present application.

[0019] Figure 2 The figure is a structure diagram of the input module of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical scheme and advantages of the present application clearer and more intelligible, the present application will be described below through specific embodiments shown in the drawings. However, it should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present application. The structures, proportions, sizes, etc. shown in the drawings of the present specification are only used to cooperate with the content disclosed in the specification, to be understood and read by those skilled in the art, and are not intended to limit the defined conditions under which the present application can be implemented, so they do not have technical significance. Any modification of structure, change of proportion relationship or adjustment of size, without affecting the effect and purpose that the present application can produce, should still fall within the scope of the technical content disclosed by the present application. In addition, in the following description, the description of known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present application.

[0021] Here, it should also be noted that, in order to avoid obscuring the present application due to unnecessary details, only structures and / or processing steps closely related to the scheme according to the present application are shown in the drawings, and other details not closely related to the present application are omitted.

[0022] As shown in Figure 1 The FPGA-based U-Net network accelerator includes a convolution pooling module, an input module, an output module, an input cache module, an output cache module, a weight FIFO module, and an off-chip storage module.

[0023] The DDR interface is connected to the input cache module and the weight FIFO module through the input module, and the input cache module and the weight FIFO module are connected to the convolution pooling module, for storing feature map data required for current calculation;

[0024] The convolution pooling module is connected to the output module through the output cache module, and the output module is connected to the DDR interface, for storing results obtained by convolution or pooling calculation;

[0025] The weight FIFO module is used for storing weight parameters required for current calculation.

[0026] The off-chip storage module is used for data storage.

[0027] The convolution pooling module includes a convolution calculation module and a pooling calculation module, the convolution calculation module includes a convolution kernel and a fused convolution layer, the fused convolution layer is formed by fusing a binary convolution layer and a normalization layer together, the network after fusion can be regarded as a new convolution layer, the layer fusion design reduces off-chip storage access times, improves memory access efficiency of the model, thereby accelerating network running speed, and reducing operation times and improving efficiency.

[0028] The convolution calculation module adopts a systolic array to replace a traditional convolution parallel and pipeline acceleration method, in the convolution calculation process, PE units in the systolic array go through input, calculation, and accumulation output three stages, pipeline optimization instructions are used to reduce running time of the convolution calculation module, and a ping-pong pipeline operation mode is used to optimize execution sequence of each module of the accelerator, so that each module can execute tasks in each time period, and the use efficiency of the module is improved.

[0029] In this embodiment, the FPGA has rich hardware resources, the utilization rate of internal hardware resources of the FPGA is improved, and higher hardware operation performance is obtained.

[0030] A linear quantization method is used to convert 32-bit floating point operation into a low-bit-width fixed point calculation unit, that is, a linear function is used for mapping to convert 32-bit floating point operation into 16-bit fixed point number, and hardware calculation consumption is reduced.

[0031] After the optimization of the computing structure by the above method, the memory access needs to be optimized to reduce the influence of memory access on the parallel computing efficiency and reduce the additional delay. A FIFO is designed as an external RAM weight data buffer interface to realize the reading and writing of the data stream in the chip. In order to adapt to the weight parameters of different convolution layers, a dynamically depth configurable FIFO is designed, and a compare depth is introduced in the FIFO controller as a zero point. When the reading and writing pointers fifo_rd and fifo_wr are equal to the compare depth, the pointers are reset to zero, the depth of the FIFO is changed, and the new weight parameters are read from the external storage.

[0032] It will be obvious to a person skilled in the art that the application is not limited to the details of the above-described exemplary embodiments, and the application can be implemented in other specific forms without departing from the spirit or essential characteristics of the application. Therefore, the embodiments should be considered as exemplary and non-limiting, and the scope of the application is defined by the appended claims rather than the above description, and all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the application.

[0033] Furthermore, it should be understood that although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the present specification is described in this way only for the sake of clarity, and a person skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be properly combined to form other embodiments that can be understood by a person skilled in the art.

Claims

1. A U-Net network accelerator based on FPGA, characterized in that, include: The system includes a convolutional pooling module, an input module, an output module, an input buffer module, an output buffer module, and a weight FIFO module. The DDR interface connects to the input cache module and the weight FIFO module via the input module. The input cache module and the weight FIFO module are connected to the convolutional pooling module to store the feature map data required for the current computation. The convolutional pooling module is connected to the output module via the output cache module. The output module is connected to the DDR interface to store the results obtained from convolution or pooling computations. The weight FIFO module stores the weight parameters required for the current computation. The FIFO is used as an external RAM weight data cache interface, and dynamic depth configuration of the FIFO is designed. The convolutional pooling module includes a convolutional computation module and a pooling computation module. The convolutional computation module includes a convolutional kernel and a fused convolutional layer. The fused convolutional layer is formed by fusing a convolutional layer and a normalization layer. The convolutional computation module uses a systolic array acceleration method.

2. The FPGA-based U-Net network accelerator according to claim 1, characterized in that: The network accelerator includes an on-chip memory (BRAM) as a cache. After startup, data is transferred to the on-chip memory (BRAM).

Images