Convolutional neural network acceleration architecture based on FPGA

Abstract

The invention discloses a convolutional neural network acceleration architecture based on an FPGA in the technical field of deep learning hardware acceleration, and the architecture comprises a processor system PS which is used for reading data from an external storage device, storing the data in a memory, and carrying out the data interaction with a programmable logic PL through an interface module; the programmable logic PL comprises a convolution module, a pooling module and an up-sampling module, and the programmable logic PL receives the data from the processor system PS through the interface module and executes parallel computing; and the interface module is used for connecting the processor system PS and the programmable logic PL and binding logic ports of different modules in the programmable logic PL to independent buses respectively through a time delay optimization mechanism. According to the FPGA-based convolutional neural network acceleration architecture provided by the invention, the problems of low data transmission efficiency, high calculation complexity and insufficient cooperative work among modules can be effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning hardware acceleration, and in particular to a convolutional neural network acceleration architecture based on FPGA. Background Art

[0002] With the development of deep learning technology, convolutional neural networks (CNNs) have been widely used in image recognition, object detection, and semantic segmentation. However, the CNN calculation process usually involves a large number of convolution, pooling, and upsampling operations, which places high demands on computing resources and operating efficiency. Traditional processor architectures are inefficient in processing these tasks, especially in real-time applications that require low latency and high throughput, and it is difficult to meet performance requirements. As a programmable hardware, FPGA (field programmable gate array) has become an important choice for accelerating CNN due to its parallel processing capabilities and efficient resource utilization. However, existing FPGA acceleration architectures often have shortcomings in data transmission efficiency, computational complexity optimization, and inter-module collaboration, resulting in the failure to fully utilize system performance. Summary of the invention

[0003] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a convolutional neural network acceleration architecture based on FPGA, which fully utilizes the hardware acceleration capability of FPGA while ensuring the high precision of the convolutional neural network, thereby significantly improving the operating efficiency of the convolutional neural network and the utilization of system resources.

[0004] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0005] The present invention provides a convolutional neural network acceleration architecture based on FPGA, comprising:

[0006] The processor system PS is used to read data from an external storage device and store the data in a memory, and to interact with the programmable logic PL through an interface module. The processor system PS controls and coordinates the calculation process, and transmits feature maps, weight data, and calculation results to and from the PL through the interface module.

[0007] The programmable logic PL includes a convolution module, a pooling module and an upsampling module. The programmable logic PL receives data from the processor system PS through the interface module and performs parallel calculations, wherein:

[0008] The convolution module is used to extract image features, and reduce computational complexity by fusing the convolution layer and the batch normalization layer into a single convolution calculation unit and optimizing using the Winograd convolution algorithm; the pooling module is used to perform pooling operations; the upsampling module is used to perform upsampling operations;

[0009] The interface module is used to connect the processor system PS and the programmable logic PL, supports data transmission and control, and enables the control instructions on the processor system PS side and the parallel computing on the programmable logic PL side to work together. Through the delay optimization mechanism, the logic ports of different modules in the programmable logic PL are bound to independent buses to support parallel interaction and independent operation between modules in the programmable logic PL.

[0010] Furthermore, the interface module includes an AXI-Lite interface and an AXI-Full interface, wherein the AXI-Lite interface is used to transmit control signals between the processor system PS and the programmable logic PL, and the AXI-Full interface is used to transmit feature maps, weight data and calculation results between the processor system PS and the programmable logic PL, thereby realizing the separation of control signals and data streams.

[0011] Furthermore, the interface module allows initiation of a bus request before executing a read or write operation by setting an expected AXI interface delay.

[0012] Furthermore, the interface module binds the logic ports of each module in the programmable logic PL to an independent AXI bus respectively through port binding to support parallel data transmission of each module.

[0013] Furthermore, the interface module binds multiple logic ports of each module in the programmable logic PL to the same physical AXI interface through bus binding.

[0014] Furthermore, the interface module supports continuous access to data at adjacent addresses, and when performing a port read operation, adopts a burst transmission mode to transmit feature map data and weight data, and sets a maximum burst length to optimize memory access efficiency.

[0015] Furthermore, the convolutional layer and the batch normalization layer are fused into a single convolutional computation unit, including:

[0016] Obtain parameters of the convolutional layer, wherein the parameters include a weight W and an offset b of the convolutional layer;

[0017] Get BN layer parameters, including scaling parameters , offset parameters , mean and variance ;

[0018] The scaling parameter of BN and variance Applied to the convolution weight W, the fused convolution weight is obtained by the following formula :

[0019] ;

[0020] By BN offset parameter , mean And the convolution offset b, the new offset is obtained by the following formula :

[0021] ;

[0022] In the formula, is a constant used to prevent the denominator from being zero;

[0023] Update the parameters of the convolutional layer to and .

[0024] Furthermore, the convolution module is based on the Winograd convolution algorithm, which reduces the number of multiplication operations required for the convolution operation by converting the convolution calculation into a matrix multiplication form.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] 1. Through the design of the interface module, the present invention realizes efficient data interaction between the processor system PS and the programmable logic PL. The interface module uses a latency optimization mechanism and binds the logic ports of different modules in the programmable logic PL to independent buses, supporting parallel interaction and independent operation between modules in the programmable logic PL. In addition, by separating the control signal from the data flow, the interface module can simultaneously complete high-speed data transmission and operation scheduling, thereby improving bus resource utilization.

[0027] The convolution module, pooling module and upsampling module in the programmable logic PL run independently through the interface module. The data transmission and calculation of each module are independent of each other, avoiding data access conflicts between modules and realizing parallel calculation, thereby improving the execution efficiency and response speed of the entire system. The processor system PS is responsible for data management and task allocation, while the programmable logic PL is responsible for executing calculation tasks. The two work together through the interface module to ensure the synchronization of control instructions and calculation tasks, thereby improving the execution efficiency and response speed of the entire system.

[0028] 2. The convolution module reduces the intermediate calculation steps and data access times by fusing the convolution layer and batch normalization layer into a single convolution calculation unit. At the same time, the Winograd convolution algorithm is used to convert the convolution operation into a matrix multiplication form, which significantly reduces the number of multiplication operations and thus speeds up the convolution operation.

[0029] The present invention adopts an extensible modular design, which can flexibly adjust FPGA resource allocation and module functions according to specific application requirements. It is suitable for edge computing of small embedded systems and can also support the acceleration requirements of large-scale computing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a schematic diagram of the FPGA-based convolutional neural network acceleration architecture provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0031] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0032] Embodiment 1:

[0033] This embodiment provides a convolutional neural network acceleration architecture based on FPGA, such as Figure 1 As shown in the figure, it includes a processor system PS, a programmable logic PL, and an interface module. The processor system PS is responsible for reading data from an external storage device and storing it in memory, and interacting with the programmable logic PL through the interface module. The processor system PS not only manages data transmission, but also coordinates the execution order and task allocation of each module through control instructions, thereby ensuring that the entire system maintains efficient collaborative work during operation.

[0034] Convolution modules, pooling modules, and upsampling modules are designed in the programmable logic PL to perform key computing tasks in convolutional neural networks. The convolution module implements feature extraction through hardware optimization. Its hardware structure integrates the convolution layer and the batch normalization layer, thereby reducing the storage and transmission of intermediate calculation results. In addition, the module uses an optimized algorithm to reduce the computational complexity and improve the throughput of the convolution operation. The pooling module is used to perform pooling operations; the upsampling module is used to perform upsampling operations.

[0035] The interface module is the core part of data transmission between the processor system PS and the programmable logic PL in this architecture, and can support batch transmission of data blocks. This module optimizes the latency design to make data transmission more efficient, thereby avoiding idle resources between the processor and the programmable logic. The logical ports of different modules in the programmable logic PL are bound to independent buses to support parallel interaction and independent operation between modules in the programmable logic PL. In addition, the design of the interface module makes the data interaction between the processor system PS and the programmable logic PL smoother, realizing collaborative computing at the hardware level.

[0036] Through the above design, the hardware acceleration architecture in this embodiment can significantly improve data interaction efficiency and system integration while meeting the high computing requirements of convolutional neural networks, and is suitable for efficient acceleration tasks of deep learning models.

[0037] Embodiment 2:

[0038] This embodiment further illustrates the FPGA-based convolutional neural network acceleration architecture based on the first embodiment, focusing on the optimized design of the interface module and the efficient calculation method of the convolution module, and elaborating its functions and advantages in combination with specific scenarios.

[0039] like Figure 1 As shown in the figure, on the PS side of the processor system, the PS is responsible for reading data from the SD card and storing it in the DDR memory. The data stored in the DDR is then transmitted to the PL side through the AXI interface to efficiently perform convolution and pooling operations for deep learning.

[0040] In terms of interface design, the interface module of this embodiment includes an AXI-Lite interface and an AXI-Full interface. The AXI-Lite interface is mainly used to transmit control signals to ensure that the instruction interaction between the processor system PS and the programmable logic PL is simple and reliable. The separation design of the control signal avoids the signal congestion problem caused by the intensive transmission of data streams. The AXI-Full interface is used for the efficient transmission of large-scale data, including the exchange of core data such as feature maps, weight data and calculation results. Through this interface, the system can realize the transmission of high-bandwidth data streams, make full use of hardware resources, and improve the overall computing performance.

[0041] In the design process of HLS (High-Level Synthesis), the #pragma HLSINTERFACE directive is used to define the interface type and optimization properties of the input and output ports of the module. It can clarify how the module interacts with the external system or other modules, and affects the performance and resource usage of the interface. Especially when performing a large number of read and write operations on DDR memory, the optimization of the interface is particularly critical. In view of the inherent delay characteristics of DDR access, if each operation in the HLS design needs to wait for the DDR access to complete, it will inevitably cause a performance bottleneck. In order to optimize resource utilization, this embodiment realizes task scheduling optimization by setting the expected AXI interface delay. Before performing data read and write operations, the interface module allows bus requests to be initiated in advance and executes other instructions during request processing, thereby reducing processor idle time. Through the delay optimization strategy, other tasks can be performed during the DDR response waiting period, thereby effectively hiding the DDR access delay, helping to build a more efficient pipeline, and thus improving the throughput of the overall system. This design not only improves the utilization of bus resources, but also reduces the waiting time of computing tasks.

[0042] In terms of supporting parallel interaction and independent operation between modules in the programmable logic PL, the interface module adopts an optimization strategy of port binding. By independently binding the logical ports of each functional module in the PL to a dedicated AXI bus, each module can independently perform data transmission tasks to avoid competition between modules. Taking the convolution module and pooling module in the PL as an example, each module needs to read different feature maps and weight data from the DDR. The logical ports of the convolution module and the pooling module are bound to two independent AXI buses respectively, so that the two can read data in parallel at the same time without interfering with each other, so that when the convolution module processes data, the pooling module can pre-fetch the next batch of data in advance, thereby improving the overall pipeline efficiency. Through such a design, when the convolution module, the pooling module and the upsampling module execute tasks at the same time, the interface module can coordinate the allocation of data streams, making it possible for each module to perform parallel calculations, significantly improving the efficiency of data interaction. At the same time, in the case of limited resources, the interface module also provides a bus binding mode to bind multiple logical ports to a single physical AXI interface to achieve dynamic allocation of interface resources. For example, different computing units of the convolution module, such as Winograd accelerator and ordinary convolution kernel, access DDR through different logical ports. Binding these logical ports to the same AXI interface reduces the occupation of hardware interface resources by sharing the physical interface. This flexible design adapts to the performance requirements of different application scenarios and improves the utilization of interface resources.

[0043] In the continuous address access scenario, the interface module of this embodiment supports burst transfer mode. In the process of transmitting feature map data and weight data, the frequency of memory access is reduced by setting the maximum burst length, which reduces the communication overhead of the bus. For example, when processing high-resolution images, the burst transfer mode enables each transmission operation to efficiently complete the transmission of the entire block of data, avoiding the delay and power consumption problems caused by byte-by-byte transmission in the traditional mode.

[0044] The above optimization measures for the interface module can significantly improve the system performance when frequently accessing DDR, reduce latency, increase throughput, and more effectively utilize available bandwidth. This has particularly important practical application value for data-intensive applications.

[0045] For the convolution module, this embodiment optimizes computational efficiency by fusing the convolution layer and the batch normalization layer into a single convolution calculation unit. Specifically, the system combines the scaling parameters and offset parameters of batch normalization based on the convolution weight W and the offset parameter b, and integrates the calculation logic of the two into one operation. The update of weights and offsets is completed through the following steps, thereby eliminating the extra step of performing a separate batch normalization operation.

[0046] Obtain parameters of the convolutional layer, wherein the parameters include a weight W and an offset b of the convolutional layer;

[0047] Get BN layer parameters, including scaling parameters , offset parameters , mean and variance ;

[0048] The scaling parameter of BN and variance Applied to the convolution weight W, the fused convolution weight is obtained by the following formula :

[0049] ;

[0050] By BN offset parameter , mean And the convolution offset b, the new offset is obtained by the following formula :

[0051] ;

[0052] In the formula, is a constant used to prevent the denominator from being zero. This optimization strategy not only reduces the computational complexity, but also saves intermediate data storage resources, making the convolution operation more efficient.

[0053] In addition, the convolution module uses the Winograd convolution algorithm, which reduces the number of multiplication operations in convolution calculations by converting traditional convolution operations into matrix multiplications. For example, the traditional 3×3 convolution requires 9 multiplications, but the Winograd algorithm can reduce it to 4. This improvement is particularly significant when processing high-resolution images or multi-layer deep networks, effectively reducing computing energy consumption and shortening processing time.

[0054] Through the optimized design of this embodiment, the interface module and the convolution module form an efficient collaboration, so that the FPGA-based convolutional neural network acceleration system can perform well in computationally intensive tasks. In addition, the system also provides flexible resource allocation capabilities to adapt to a variety of practical scenarios and provide comprehensive support for the acceleration needs of convolutional neural network training.

[0055] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A convolutional neural network acceleration architecture based on FPGA, characterized in that: include: The processor system PS is used to read data from an external storage device and store the data in a memory, and to interact with the programmable logic PL through an interface module. The processor system PS controls and coordinates the calculation process, and transmits feature maps, weight data, and calculation results to and from the PL through the interface module. The programmable logic PL includes a convolution module, a pooling module and an upsampling module. The programmable logic PL receives data from the processor system PS through the interface module and performs parallel calculations, wherein: The convolution module is used to extract image features and reduce computational complexity by fusing the convolution layer and the batch normalization layer into a single convolution calculation unit and optimizing the Winograd convolution algorithm; The pooling module is used to perform a pooling operation; The up-sampling module is used to perform an up-sampling operation; The interface module is used to connect the processor system PS and the programmable logic PL, supports data transmission and control, and enables the control instructions on the processor system PS side and the parallel computing on the programmable logic PL side to work together. Through the delay optimization mechanism, the logic ports of different modules in the programmable logic PL are bound to independent buses to support parallel interaction and independent operation between modules in the programmable logic PL.

2. The FPGA-based convolutional neural network acceleration architecture according to claim 1, characterized in that: The interface module includes an AXI-Lite interface and an AXI-Full interface, wherein the AXI-Lite interface is used to transmit control signals between the processor system PS and the programmable logic PL, and the AXI-Full interface is used to transmit feature maps, weight data and calculation results between the processor system PS and the programmable logic PL, thereby realizing the separation of control signals and data streams.

3. The FPGA-based convolutional neural network acceleration architecture according to claim 2, characterized in that: The interface module allows bus requests to be initiated before performing read and write operations by setting the expected AXI interface delay.

4. The FPGA-based convolutional neural network acceleration architecture according to claim 2, characterized in that: The interface module binds the logic ports of each module in the programmable logic PL to an independent AXI bus through port binding to support parallel data transmission of each module.

5. The FPGA-based convolutional neural network acceleration architecture according to claim 4, characterized in that: The interface module binds multiple logic ports of each module in the programmable logic PL to the same physical AXI interface through bus binding.

6. The FPGA-based convolutional neural network acceleration architecture according to claim 1, characterized in that: The interface module supports continuous access to data at adjacent addresses, and when performing a port read operation, adopts a burst transmission mode to transmit feature map data and weight data, and sets a maximum burst length.

7. The FPGA-based convolutional neural network acceleration architecture according to claim 1, characterized in that: Fusion of convolutional layers and batch normalization layers into a single convolutional computation unit, including: Obtain parameters of the convolutional layer, wherein the parameters include a weight W and an offset b of the convolutional layer; Get BN layer parameters, including scaling parameters , offset parameters , mean and variance ; The scaling parameter of BN and variance Applied to the convolution weight W, the fused convolution weight is obtained by the following formula : ； By BN offset parameter , mean And the convolution offset b, the new offset is obtained by the following formula : ； In the formula, is a constant used to prevent the denominator from being zero; Update the parameters of the convolutional layer to and .

8. The FPGA-based convolutional neural network acceleration architecture according to claim 1, characterized in that: The convolution module is based on the Winograd convolution algorithm, which reduces the number of multiplication operations required for the convolution operation by converting the convolution calculation into the form of matrix multiplication.

Images