Target detection network hardware acceleration system based on FPGA

Abstract

The invention discloses a target detection network hardware acceleration system based on an FPGA (Field Programmable Gate Array) in the field of hardware acceleration of a deep learning algorithm, and the system comprises a cache unit which is connected with an external memory and is used for storing a feature map, weight data and intermediate calculation data; the method comprises the following steps: parallelly loading data to a cache unit from a plurality of ports of an external memory, storing a feature map and weight data in blocks, and dynamically loading data blocks required by current calculation as required; the convolution module is connected with the cache unit, and is used for performing convolution calculation on the feature map, converting the floating-point number into a fixed-point number based on a quantization strategy, and performing optimization by adjusting bit widths of integer and decimal parts in the fixed-point number; and the pooling module is connected with the convolution module and is used for carrying out downsampling processing on the feature map, and a multi-channel parallel pooling structure is adopted to realize simultaneous pooling calculation of a plurality of channels. According to the invention, the calculation speed of the target detection network on FPGA hardware is obviously improved, and the storage and calculation resource consumption of the system is reduced.

Description

Technical Field

[0001] The present invention relates to the field of hardware acceleration of deep learning algorithms, and in particular to an FPGA-based target detection network hardware acceleration system. Background Art

[0002] In the industrial and engineering fields, high-precision and high-efficiency defect detection technology is the core element for realizing intelligent manufacturing and product quality control. Whether it is the processing of metal raw materials or other types of manufacturing processes, surface damage (such as scratches, stains, etc.) is common, which seriously affects the quality and service life of downstream products. Traditional machine vision inspection methods are limited by light sources, hardware equipment and complex environmental factors, and it is difficult to cope with multiple challenges in industrial scenarios, such as subtle defects, scarce samples, high-speed operation of assembly lines, and complex lighting and background.

[0003] In response to these challenges, deep learning-based defect detection algorithms have attracted much attention due to their high accuracy and strong adaptability. However, the high demand for computing resources by deep learning algorithms makes it difficult for traditional central processing units (CPUs) to cope with the requirements. Although GPUs perform well in processing speed, their high energy consumption is also a problem. In contrast, FPGAs perform better in low-power application scenarios. In order to cope with a large number of parameters and calculations, accelerator optimization and model quantization are huge challenges in deep learning today. The detection network has a more complex structure than convolutional computing, so it is more meaningful to study how to achieve efficient network acceleration. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a target detection network hardware acceleration system based on FPGA, which significantly improves the computing speed of the target detection network and reduces the storage and computing resource consumption of the system. It is suitable for target detection network hardware acceleration scenarios with strict requirements on latency and resource utilization.

[0005] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0006] The present invention provides a target detection network hardware acceleration system based on FPGA, comprising:

[0007] A cache unit is connected to an external memory and is used to store feature maps, weight data, and intermediate calculation data. The feature maps and weight data are stored in blocks by loading data from multiple ports of the external memory to the cache unit in parallel, and the data blocks required for the current calculation are dynamically loaded on demand. A dual buffer dynamic switching mechanism is used to store the results of the convolution calculation and the data required for the current convolution calculation.

[0008] The convolution module is connected to the cache unit and is used to perform convolution calculations on the feature map. It converts floating-point numbers into fixed-point numbers based on the quantization strategy and optimizes them by adjusting the bit width of the integer and decimal parts of the fixed-point numbers. It reduces the number of data transfers by sharing intermediate calculation data based on the data reuse strategy. It uses channel expansion technology to simultaneously calculate the convolution results of multiple output channels.

[0009] The pooling module is connected to the convolution module and is used to downsample the feature map. It loads the feature map data in blocks and combines it with pipeline calculation to calculate the maximum value in the pooling window in a sliding window manner. It adopts a multi-channel parallel pooling structure to realize pooling calculations of multiple channels at the same time.

[0010] Furthermore, by loading data from multiple ports of the external memory to the cache unit in parallel, the input feature map and weight data are stored in blocks, and the data blocks required for the current calculation are dynamically loaded on demand, including:

[0011] A parallel data transmission channel is established between the external memory and the cache unit through multiple ports, supporting the simultaneous transmission of multiple data blocks;

[0012] A cyclic block strategy is adopted to divide the feature map and weight data into several small blocks. Each time, only the small block data required for the current calculation is read from the external memory and loaded into the cache unit;

[0013] After the calculation is completed, the calculation results are written back to the external memory through the multiple ports, and the next batch of data blocks are loaded at the same time.

[0014] Furthermore, the convolution module includes multiple convolution calculation units, which are responsible for processing the convolution operation of performing convolution kernel calculation on the input feature map and weight data.

[0015] Furthermore, a dual buffer dynamic switching mechanism is used to store the result of the convolution calculation and the data required for the current convolution calculation, including:

[0016] The cache unit is equipped with two groups of buffers, which are used for calculating the current block and storing the last calculation result respectively;

[0017] In each convolution calculation cycle, the buffer to be used is determined according to the pingpong flag;

[0018] In response to the pingpong flag being true, performing convolution calculation in the first buffer and writing the calculation result in the second buffer into the external memory;

[0019] In response to the pingpong flag being false, a convolution calculation is performed in the second buffer, and at the same time, the calculation result in the first buffer is written to the external memory.

[0020] Furthermore, the convolution calculation unit replaces the multiplication operation in the convolution calculation with shift and addition operations, and splits the complex conditional judgment statements, converts them into independent Boolean expressions and calculates part of the results in advance.

[0021] Furthermore, based on the data reuse strategy, the number of data transfers is reduced by sharing intermediate calculation results, including:

[0022] In convolution calculation, the input feature map is cached in the on-chip memory or register in the cache unit, so that multiple convolution kernels in the convolution calculation unit can process the same feature map in parallel;

[0023] Some intermediate calculation data are temporarily stored in the on-chip registers in the cache unit for subsequent calculations.

[0024] Furthermore, the channel expansion technique is used to simultaneously calculate the convolution results of multiple output channels, including:

[0025] Load feature map data from external memory and distribute the same feature map data to multiple convolution computing units;

[0026] Multiple convolutional computing units calculate the results of multiple channels in parallel. After each convolutional computing unit completes the calculation, it transmits its respective convolution results to the next layer of the convolutional neural network.

[0027] Furthermore, the floating-point number is converted into a fixed-point number based on the quantization strategy, and the bit width of the integer and decimal parts in the fixed-point number is adjusted for optimization, including:

[0028] Determine the total bit width, integer bit width and decimal bit width of the fixed-point number to limit the representation range of the fixed-point number;

[0029] Adjust the allocation ratio of integer and decimal bit width of fixed-point numbers, giving priority to increasing the bit width of the decimal part to improve the accuracy of numerical representation;

[0030] Select the rounding quantization method to convert the floating point number to a fixed point number;

[0031] Determine the scaling factor, map the floating-point number to the fixed-point number range, and calculate the quantized fixed-point number using the following formula:

[0032] ;

[0033] Where Q is the quantized fixed-point number; round is the rounding function; x is the original floating-point number; Z is the zero point for asymmetric quantization; for symmetric quantization, Z=0; and s is the scaling factor.

[0034] Furthermore, by loading feature map data in blocks and combining pipeline calculation, the maximum value in the pooling window is calculated in a sliding window manner; a multi-channel parallel pooling structure is adopted to realize the pooling calculation of multiple channels at the same time, including:

[0035] By using the block loading method, the feature map data is divided into several small blocks according to the channel, height and width, and loaded into the cache unit in parallel;

[0036] In each pooling operation, the data in the cache unit is expanded into multiple independently accessible arrays; pipeline instructions are applied in each loop iteration to execute each stage of the pooling operation concurrently, so that each clock cycle processes a data element in the pooling window;

[0037] Compare and calculate the feature values ​​in the sliding window, determine the maximum value, and store the pooling result as a downsampled feature map;

[0038] Through the multi-channel parallel pooling structure, the inner channel loop is expanded so that the pooling calculations of multiple channels can be performed simultaneously.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1. The present invention realizes efficient hardware acceleration of target detection network on FPGA through multiple optimizations in data transmission, storage and calculation. Through multi-port connection between cache unit and external memory, combined with data block storage and on-demand dynamic loading mechanism, the pressure of external memory bandwidth is effectively reduced, and efficient data support is provided for convolution and pooling modules. The introduction of dual buffer dynamic switching mechanism realizes seamless connection between data loading and calculation, avoids waiting between loading and storage, and greatly improves the resource utilization and throughput of the system.

[0041] The convolution module achieves an effective balance between numerical representation accuracy and computing resource consumption by quantizing floating-point numbers to fixed-point numbers and dynamically adjusting the allocation ratio of integer and decimal bit widths. Combined with data reuse strategy and channel expansion technology, the number of data transfers is significantly reduced and parallel calculation of multiple output channels is achieved, thereby accelerating the execution of convolution operations based on full utilization of FPGA hardware parallelism.

[0042] The pooling module uses block loading and sliding window methods to process feature map data. Through pipeline operations and multi-channel parallel pooling structure, the pooling calculation can complete the downsampling tasks of multiple channels with low latency, thereby further accelerating the overall process of feature extraction.

[0043] 2. By replacing the multiplication operation in the convolution calculation with shift and addition operations, the present invention effectively reduces the computational complexity of the convolution calculation unit. Since the resources and delays required for shift and addition operations are much lower than those for multiplication operations in the hardware circuit, this optimization not only reduces the hardware resource consumption of the calculation unit, but also significantly shortens the critical path of the convolution operation, improving the timing performance and circuit operation frequency.

[0044] At the same time, by splitting complex conditional judgment statements, converting them into independent Boolean expressions and calculating some results in advance, the depth of logical operations on the critical path is effectively reduced, thereby further optimizing timing performance and reducing delay bottlenecks in hardware design. This optimization method can ensure that the convolution calculation unit remains stable and efficient at high frequency operation.

[0045] Combining the above optimization measures, the present invention not only significantly improves the computing speed of the target detection network, but also reduces the storage and computing resource consumption of the system. It has high efficiency and practicality, and is suitable for application scenarios with limited resources and high real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a schematic diagram of double ping-pong operation provided by the second embodiment of the present invention;

[0047] Figure 2 This is the pooling processing intention provided by the second embodiment of the present invention;

[0048] Figure 3 It is a schematic diagram of the recognition accuracy of the target detection network when the integer bit width in the fixed-point number provided by the second embodiment of the present invention is 6 bits;

[0049] Figure 4 This is a schematic diagram of the recognition accuracy of the target detection network when the integer bit width in the fixed-point number provided in the second embodiment of the present invention is 7 bits. DETAILED DESCRIPTION

[0050] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0051] The terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of this disclosure / application, unless otherwise specified, "plurality" means two or more.

[0052] Embodiment 1:

[0053] This embodiment provides an FPGA-based target detection network hardware acceleration system, which aims to achieve efficient acceleration of target detection tasks by optimizing data transmission, storage and calculation links. The system mainly includes a cache unit, a convolution module and a pooling module. The design and function of each part are as follows:

[0054] In this embodiment, the cache unit is connected to the external memory through multiple data ports, and can efficiently load feature maps and weight data in parallel. In order to solve the problem of limited on-chip storage resources, the system adopts a block loading strategy to divide the feature map and weight data into several small blocks, and only load the data blocks required for the current calculation into the cache unit each time. This method makes it possible to break through the on-chip resource limitations and process larger data. In order to further improve the efficiency of data transmission and calculation, the cache unit introduces a dual buffer dynamic switching mechanism. When the convolution module is calculating the current data block, the next batch of required data has been pre-loaded in the cache unit, and the calculation results are written directly back to the external memory. The whole process is efficient and smooth without waiting.

[0055] The convolution module is the core computing unit of the system, which is used to perform convolution operations on feature maps. In order to adapt to the limitations of hardware resources and the characteristics of different computing requirements of the network layer, this embodiment converts floating-point numbers into fixed-point numbers based on a quantization strategy, and balances the numerical representation range and calculation accuracy by dynamically adjusting the bit width allocation ratio of the integer and decimal parts of the fixed-point numbers. This design ensures that the system achieves a good balance between calculation accuracy and hardware resource occupancy. In addition, in order to reduce the overhead caused by repeated data handling, the convolution module shares intermediate calculation results through a data reuse strategy, significantly reducing the access requirements to the cache unit and external memory. At the same time, the convolution module also uses channel expansion technology, which allows convolution calculations of multiple channels to be performed simultaneously, greatly improving the calculation speed. This technology makes full use of the hardware parallel characteristics of FPGA, bringing the efficiency of convolution calculations to a new level.

[0056] After completing the convolution calculation, the feature map is passed to the pooling module for downsampling. The pooling module loads the feature map data in blocks, and combines pipeline computing technology to traverse each data block of the feature map row by row and column by column in a sliding window manner. In each pooling window, the system compares the data and extracts the maximum value, and then stores the result in the cache unit. In order to further accelerate the pooling process, this embodiment introduces a multi-channel parallel pooling structure, and the pooling units of each channel work simultaneously in the same clock cycle, thereby realizing multi-channel parallel processing. In addition, the pooling module also uses pipeline operation to preload the data of the next window when calculating the current window, making the entire downsampling process more efficient.

[0057] Through the collaborative work of the above modules, this embodiment successfully solves multiple technical difficulties in running the target detection network on FPGA, significantly improves the computing speed, reduces resource consumption, and meets the needs of target detection application scenarios with high real-time requirements and limited hardware resources.

[0058] Embodiment 2:

[0059] In Example 1, the present invention provides a hardware acceleration system for a target detection network based on FPGA, which optimizes the data transmission, storage and calculation processes from the perspective of system architecture design to improve the overall performance. Based on Example 1, this embodiment will further explain in detail the specific implementation scheme of each module, including the block loading mechanism of the cache unit, the quantization and channel expansion design of the convolution module, and the pipeline and multi-channel parallel processing structure of the pooling module, so as to more comprehensively demonstrate the hardware optimization strategy and actual working process of the system.

[0060] Since the storage resources of the cache unit in the FPGA are limited, it is impractical to cache the entire feature map and all weights. Therefore, the present invention stores the feature map and weight data in blocks by loading data from multiple ports of the external memory to the cache unit in parallel, and dynamically loads the data blocks required for the current calculation on demand.

[0061] Specifically, a cyclic block strategy is adopted to divide the input feature map, weights, and output feature map into several small blocks. Each time, only the small block data required for the current calculation is loaded into the on-chip memory. After the calculation is completed, the result is written back to the external memory, and then the next batch of data blocks is loaded. Tr, Tc, and Tn represent the size of the data block in the three dimensions of row, column, and channel respectively. This block movement strategy enables the algorithm to gradually process the entire feature map, and only needs to process a small block at a time, thereby effectively utilizing the on-chip storage space. At the same time, this block strategy can also adapt well to different hardware configurations.

[0062] By adjusting the block size (such as Tc and Tr), the system performance can be optimized to achieve the best computing efficiency and balance under given on-chip storage resources. By reasonably dividing and managing computing tasks, it not only alleviates the limitations of storage resources, but also improves the efficiency of data processing, making real-time processing of large-scale neural networks possible.

[0063] In order to improve the architectural optimization strategy of computing and data access parallelism, the present invention proposes a solution of dynamically switching double buffers to realize computing and data transmission parallelism, realizes the overlap of convolution computing and external memory access, and thus improves the efficiency of the overall system.

[0064] Two groups of buffers are provided in the cache unit, which are used for calculating the current block and storing the last calculation result respectively; in each convolution calculation cycle, the buffer to be used is determined according to the pingpong flag; in response to the pingpong flag being true, the convolution calculation is performed in the first buffer, and the calculation result in the second buffer is written to the external memory; in response to the pingpong flag being false, the convolution calculation is performed in the second buffer, and the calculation result in the first buffer is written to the external memory.

[0065] For example, when pingpong is true, the system performs convolution calculations in buffer fm_out2 and stores the output data in buffer fm_out1 in DDR; on the contrary, when pingpong is false, the system performs calculations in fm_out1 and writes the data of fm_out2 to DDR. This operation allows data calculation and storage operations to be performed simultaneously, avoiding waiting when loading or storing data.

[0066] The convolution module includes multiple convolution calculation units, which are responsible for processing the convolution operation of the feature map and weight data. From a local point of view, similar parallel processing is also used inside the convolution calculation unit. In each convolution cycle, the loading and calculation operations of the convolution kernel weights and feature map data are performed simultaneously. By preloading the data required for the next block of calculation during the calculation process, the system further reduces the overhead of data loading and achieves complete overlap of calculation and data transmission.

[0067] The advantage of this embodiment is that it makes full use of hardware resources and minimizes data access waiting time. Compared with the traditional single buffer solution, the dual ping-pong operation solution enables the system to seamlessly switch between data loading, calculation and result storage through highly parallel calculation and storage operations. This mechanism effectively improves the throughput of convolution calculations, especially when processing large-scale feature maps, and can significantly reduce the performance bottleneck caused by frequent access to external memory.

[0068] Furthermore, the present invention uses the HLS (High-Level Synthesis) tool to analyze the reasons for limiting the operating frequency of the system circuit. The analysis report generated by the HLS tool can provide a comprehensive understanding of the design performance, including key information such as resource usage, timing path, and critical path analysis. We focus on WNS (Worst Negative Slack) and critical path analysis in the report: the WNS indicator is used to evaluate whether the design meets the timing constraints, while the critical path analysis reveals the main factors that limit the system frequency, that is, the slowest operation that affects the operating frequency of the circuit.

[0069] The analysis results show that the critical path is located in the convolution calculation unit in the image processing process. Further analysis found that the multiplier link delay of this unit is too high, which becomes the main bottleneck limiting the operating frequency of the circuit and prevents the system from reaching a higher clock frequency.

[0070] Therefore, the present invention replaces the multiplication operation in the convolution calculation unit with shift and addition operations, and splits the complex conditional judgment statements, converts them into independent Boolean expressions and calculates some results in advance. Through the optimization of these two aspects, the design achieves a high-frequency operation of 200MHz under limited FPGA resources, which provides a guarantee for the real-time and high efficiency of the entire system.

[0071] In hardware, multiplication operations are usually much more complex than addition and shift operations, and consume more time and resources. For optimization, the present invention decomposes multiplication operations into shift and addition operations, which is very efficient in hardware circuits and can significantly shorten the delay of the critical path.

[0072] Conditional judgments (such as if-else statements) contain complex mathematical operations (such as multiplication and comparison) that need to be completed within one clock cycle, which will lengthen the critical path and limit the maximum operating frequency of the circuit. The present invention splits these complex logics into simpler Boolean expressions and calculates some results in advance, reducing the burden of each clock cycle, thereby increasing the operating frequency of the circuit.

[0073] Data reuse is also a key strategy to reduce power consumption and improve computing efficiency in FPGA accelerators. Data handling is the main source of FPGA power consumption, so reducing the number of data handling times and improving data usage efficiency is of great significance. Therefore, the present invention is based on a data reuse strategy to reduce the number of data handling times by sharing intermediate calculation results, including:

[0074] In convolution calculations, the feature map is cached in the on-chip memory or register in the cache unit, so that multiple convolution kernels in the convolution calculation unit can process the same feature map in parallel; some intermediate calculation data is temporarily stored in the on-chip register in the cache unit for subsequent calculations.

[0075] In convolution calculations, some intermediate calculation results can be temporarily stored in registers for subsequent calculations, avoiding frequent writing of intermediate results back to external memory and re-reading. In the absence of dependencies between data, the computing unit can complete all related operations on-chip as much as possible, reducing the number of times intermediate results are moved. At the same time, by caching the input feature map in on-chip memory or registers, multiple convolution kernels can process the same feature map in parallel. In this way, the feature map only needs to be read from the external memory once and can be reused on multiple convolution calculation units, reducing the overhead of repeatedly loading the input feature map. This reuse method greatly improves data utilization efficiency and reduces the number of external memory accesses by sharing the same feature map data in multiple convolution calculation units.

[0076] On the basis of optimizing the computational efficiency of the convolutional neural network through data segmentation and data reuse strategies, in order to further improve the parallel computing capability of the system, the channel expansion technology is used in this embodiment to simultaneously calculate the convolution results of multiple output channels. Specifically, after loading the feature map data from the external memory, the same feature map data is distributed to multiple convolution calculation units, and the parallel characteristics of these calculation units are used to calculate the convolution kernels of multiple output channels synchronously. After the calculation is completed, the convolution results of each convolution unit are efficiently transmitted to the next layer of the neural network for subsequent processing.

[0077] In this way, channel expansion technology can fully utilize the potential of hardware computing resources, and work together with data segmentation and data reuse strategies to not only further reduce the data handling overhead of external memory, but also significantly improve the parallelism and overall throughput of calculations. This three-way joint optimization method greatly shortens the reasoning time of convolutional neural networks, providing a strong guarantee for achieving efficient and low-latency deep learning calculations.

[0078] In FPGA design, fixed-point quantization is a method of converting floating-point numbers into fixed-point numbers. It reduces the required storage space and computing resources by reducing the data bit width, which is especially important for embedded devices with limited resources. In the process of deep learning inference, the use of fixed-point numbers can significantly improve the processing speed and efficiency of FPGAs while maintaining sufficient numerical accuracy. Linear quantization is the most commonly used quantization scheme. First, the total bit width, integer bit width, and decimal bit width of the fixed-point number are determined to limit the representation range of the fixed-point number;

[0079] Adjust the allocation ratio of integer and decimal bit width of fixed-point numbers, giving priority to increasing the bit width of the decimal part to improve the accuracy of numerical representation;

[0080] Select the rounding quantization method to convert the floating point number to a fixed point number;

[0081] Determine the scaling factor, map the floating-point number to the fixed-point number range, and calculate the quantized fixed-point number using the following formula:

[0082] ;

[0083] Where Q is the quantized fixed-point number; round is the rounding function; x is the original floating-point number; Z is the zero point for asymmetric quantization; for symmetric quantization, Z=0; and s is the scaling factor.

[0084] In order to explore the impact of different bit width allocation ratios on image recognition accuracy, this embodiment defines two fixed-point data types data_t, which have different bit widths and representation formats:

[0085] (1) Use <ap_fixed<16,6,AP_RND_CONV,AP_SAT> , represents a 16-bit fixed-point number, of which 6 bits are used to represent the integer part, using the round-to-nearest conversion mode (AP_RND_CONV), and saturation processing (AP_SAT) on overflow;

[0086] (2) Use <ap_fixed<16,7,AP_RND_CONV,AP_SAT> , which is also a 16-bit fixed-point number, but 7 bits are allocated to represent the integer part, and the rest of the characteristics are the same.

[0087] In the experiment, we selected the YOLOv4-tiny network model and replaced all floating-point operations with corresponding fixed-point operations, using the above two quantization strategies respectively.

[0088] By comparing the recognition accuracy under the two quantization strategies, we can observe that the quantization strategy using 6-bit integers is better than 7-bit integers in recognition accuracy, which shows that the model can maintain higher accuracy when the decimal width increases.

[0089] This is because more bit width can provide more refined numerical representation, thereby reducing quantization error. This experiment shows that the quantization strategy has a direct impact on the accuracy of the image recognition model. By appropriately increasing the decimal bit width, the accuracy loss caused by quantization can be reduced while maintaining low hardware resource consumption.

[0090] Optionally, the optimal bit width allocation is different in different network layers or models, and the computational properties and data range differences of each layer will affect the choice of quantization scheme. For example, the bit width allocation requirements of convolutional layers and fully connected layers are usually different. Convolutional layers usually process two-dimensional data such as images, which may have a small range of values ​​but require high precision. Especially in image processing, the changes in pixel values ​​are relatively small, so convolutional layers usually use more decimal places to ensure the accuracy of calculations, and the number of integer bits is relatively small. For example, if an 8-bit fixed-point representation is used, 5 bits may be allocated to the integer part and 3 bits to the decimal part, which can ensure that the accuracy is fully preserved during the convolution calculation and avoid excessive quantization errors.

[0091] In contrast, the fully connected layer usually relies on the product calculation of a larger weight matrix and the input data, so it has a higher demand for the integer part, especially when the network size becomes larger, the numerical range of the weight value may increase significantly. In order to ensure that a larger numerical range can be processed, the fully connected layer usually chooses more integer bits and appropriately reduces the decimal places. For example, in an 8-bit fixed-point representation, you may choose to allocate 4 bits to the integer part and 4 bits to the decimal part, which can ensure that the integer part of the larger value can be represented while retaining sufficient decimal precision.

[0092] In addition, there are differences in the bit width allocation of activation values ​​and weights. Activation values ​​are usually limited in range after being processed by an activation function (such as ReLU). For example, the values ​​after ReLU activation are usually positive and have a small range, so more decimal places and fewer integer places can be selected. For example, in 8-bit fixed-point representation, 6 decimal places and 2 integer places can be used to process activation values, which can effectively represent the decimal part and avoid the loss of precision caused by quantization error. For weights, since their values ​​may be large, especially in the early stages of the network, the weight values ​​may span a large range, so more integer bits are required. For example, an 8-bit fixed-point representation can be used, with 4 bits for the integer part and 4 bits for the decimal part to accommodate larger weight values ​​and a wider dynamic range.

[0093] For lightweight models (such as MobileNet), due to the need for efficient calculations under limited hardware resources, a smaller bit width allocation is usually chosen. In these models, smaller integer and decimal bits help reduce the storage and computational overhead of the model. For example, convolutional layers and fully connected layers may quantize weights and activations into 8-bit fixed-point numbers, usually with a 4-bit integer and 4-bit decimal configuration, which can find a good balance between performance and accuracy. For some deeper networks (such as ResNet), due to the more complex calculations and larger numerical range, a larger bit width may be used, such as 16-bit fixed-point representation, where 8 bits are used for the integer part and 8 bits are used for the decimal part to ensure calculation accuracy and a larger dynamic range.

[0094] In short, the computational characteristics and data distribution of different network layers have different requirements for bit width allocation. In the quantization process of convolutional layers and activation values, more decimal places help maintain fine numerical representation, while fully connected layers and deep networks usually require more integer bits to ensure the representation range of numerical values. Through reasonable bit width allocation, the computational efficiency can be effectively improved and the necessary accuracy can be maintained under the condition of limited hardware resources.

[0095] In FPGA accelerators, pooling is a common operation that helps reduce the amount of computation, reduce the risk of overfitting, and improve the generalization ability of the model. This embodiment divides the feature map data into several small blocks by channel, height and width through block loading, and loads them into the cache unit in parallel; in each pooling operation, the current data block is traversed with a 2x2 sliding window, and the data in the cache unit is expanded into multiple independent accessible arrays; in each loop iteration, pipeline instructions are applied to execute each stage of the pooling operation concurrently to shorten the start time interval of the loop, significantly improve the timing performance, and process a data element in a pooling window in each clock cycle; compare and calculate the feature values ​​in the sliding window, determine the maximum value, and store the pooling result as a downsampled feature map; through a multi-channel parallel pooling structure, the inner channel loop is expanded so that the pooling calculations of multiple channels are performed simultaneously.

[0096] Specifically, in this embodiment, the external memory is provided with 4 ports, and the cache unit reads data from four different ports in parallel, calculates the maximum value in the pooling window, and after such comparison, writes the obtained maximum value into the output cache. At the same time, the row and column loops of the feature map are expanded into a pipeline using pipeline instructions to achieve the goal of processing one data element per clock cycle, thereby reducing the overall processing time. The speed of the pooling operation is increased by processing multiple pooling units in parallel. The outer loop in the code uses two nested loops to traverse the rows and columns of the feature map, and the inner loop is a channel loop that is expanded through the UNROLL instruction, which means that the operations in the loop will be executed simultaneously, thereby achieving parallel processing.

[0097] Take a feature map of size 16×32×32 as an example, that is, the feature map has 16 channels, and the spatial size of each channel is 32 rows and 32 columns. The feature map data is processed efficiently by block division and parallel computing.

[0098] First, the feature map is divided into small "data blocks" for processing. The size of each data block is 4×8×8, that is, 4 channels are processed at a time, and each channel processes 8 rows and 8 columns of space. The feature map has a total of 16 channels, so it needs to be divided into 4 groups, each with 4 channels.

[0099] In each group, the 32×32 space of the feature map is divided into 8×8 small blocks. A total of 8 blocks are required (each time processing 8 rows of height, 32 rows are divided 4 times in total); 8 blocks (each time processing 8 columns of width, 32 columns are divided 4 times in total). Therefore, a total of 8×8=64 data block loads are required to complete the processing of the entire feature map. Through parallel transmission, 8×8 data blocks of 4 channels are transmitted to the cache unit at one time. This method utilizes the multi-port characteristics of the hardware and loads data from multiple channels at the same time, improving transmission efficiency. After each transmission of 4×8×8 data blocks, pipeline calculation is used to complete the pooling operation: 8×8 data is calculated each time; because a data block has 4 channels, 4 calculations can complete the pooling of a data block. The characteristic of pipeline calculation is multi-stage parallel processing. When calculating the first data block, the second data block can start loading; when calculating the pooling window of the previous row, the next row can prepare data.

[0100] In this embodiment, an accelerator based on the Zynq series FPGA is designed, which is mainly used to accelerate the convolution operation, pooling operation and upsampling calculation in the deep learning model. The accelerator consists of four sub-modules: standard convolution (stdconv), point-by-point convolution (pwconv), maximum pooling (maxpool) and upsampling (usample). Each sub-module consumes different resources on the hardware platform, including lookup table (LUT), block RAM (BRAM_18K), digital signal processing unit (DSP48E) and trigger (FF). Through the precise allocation and optimization of these resources, the computing efficiency and resource utilization are improved.

[0101] As shown in Table 1, the HLS tool estimates resource utilization under the application of the above optimization strategies. For example, the standard convolution accelerator (stdconv) occupies 94% of LUT resources, the point-by-point convolution accelerator (pwconv) consumes 11% of BRAM_18K resources, the maximum pooling accelerator (maxpool) occupies 3% of FF resources, and the DSP48E resource utilization of the entire accelerator is as high as 97%. These resource consumption data provide valuable optimization references for the design and show that the accelerator can effectively utilize FPGA hardware resources in each module to achieve efficient computing.

[0102] Table 1

[0103]

[0104] The hardware platform selected in this embodiment is the Zynq series FPGA, in which the main chip is XC7Z020, equipped with a dual-core ARM Cortex-A9 CPU, with 220 DSP48E units, 53200 LUT resources and 140 BRAM36K storage resources. The onboard DDR3 memory supports large bandwidth and large memory computing, which can fully meet the needs of complex computing tasks.

[0105] As shown in Table 2, in terms of specific resource utilization, the entire accelerator design consumed 39,328 LUTs, accounting for 73.92% of the total; 48,077 FFs, accounting for 45.19% of the total; 215 DSPs, accounting for 97.73% of the total; and 96 BRAMs, accounting for 68.57% of the total. In particular, the high utilization of LUTs and BRAMs shows that the design team has achieved effective optimization of FPGA resources through reasonable resource allocation. Through this resource management, the accelerator can fully utilize the computing power of the FPGA and maximize computing parallelism while maintaining high computing accuracy.

[0106] In addition, the utilization rate of DSP resources in the accelerator has reached 97%. This shows that the accelerator can efficiently perform parallel calculations when performing mathematical operations such as multiplication and addition, making full use of the parallel processing capabilities of the FPGA platform. This high utilization design enables the accelerator to achieve superior computing performance when performing complex convolution, pooling, and upsampling operations.

[0107] Table 2

[0108]

[0109] In summary, this embodiment improves the computing efficiency of the accelerator on the FPGA through reasonable hardware resource allocation and optimization, especially in the utilization of LUT, BRAM, DSP and FF. Through these optimizations, it is possible to effectively improve computing parallelism, reduce data transmission delays, and ensure that the accelerator can efficiently perform deep learning tasks.

[0110] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An FPGA-based target detection network hardware acceleration system, characterized in that: include: A cache unit, connected to an external memory, for storing feature maps, weight data, and intermediate calculation data; By loading data from multiple ports of the external memory into the cache unit in parallel, the feature map and weight data are stored in blocks, and the data blocks required for the current calculation are dynamically loaded on demand; A dual buffer dynamic switching mechanism is used to store the results of convolution calculation and the data required for the current convolution calculation; The convolution module is connected to the cache unit and is used to perform convolution calculations on the feature map, convert floating-point numbers into fixed-point numbers based on the quantization strategy, and optimize the fixed-point numbers by adjusting the bit width of the integer and decimal parts; Based on the data reuse strategy, the number of data transfers is reduced by sharing intermediate calculation data; channel expansion technology is used to simultaneously calculate the convolution results of multiple output channels; The pooling module is connected to the convolution module and is used to downsample the feature map. It loads the feature map data in blocks and combines it with pipeline calculation to calculate the maximum value in the pooling window in a sliding window manner. It adopts a multi-channel parallel pooling structure to realize pooling calculations of multiple channels at the same time.

2. The FPGA-based target detection network hardware acceleration system according to claim 1, characterized in that: By loading data from multiple ports of external memory to the cache unit in parallel, the feature map and weight data are stored in blocks, and the data blocks required for the current calculation are dynamically loaded on demand, including: A parallel data transmission channel is established between the external memory and the cache unit through multiple ports, supporting the simultaneous transmission of multiple data blocks; A cyclic block strategy is adopted to divide the feature map and weight data into several small blocks. Each time, only the small block data required for the current calculation is read from the external memory and loaded into the cache unit; After the calculation is completed, the calculation results are written back to the external memory through the multiple ports, and the next batch of data blocks are loaded at the same time.

3. The FPGA-based target detection network hardware acceleration system according to claim 1, characterized in that: The convolution module includes multiple convolution calculation units, which are responsible for processing the convolution operation of performing convolution kernel calculation on the feature map and weight data.

4. The FPGA-based target detection network hardware acceleration system according to claim 3, characterized in that: A dual buffer dynamic switching mechanism is used to store the results of the convolution calculation and the data required for the current convolution calculation, including: The cache unit is equipped with two groups of buffers, which are used for calculating the current block and storing the last calculation result respectively; In each convolution calculation cycle, the buffer to be used is determined according to the pingpong flag; In response to the pingpong flag being true, performing convolution calculation in the first buffer and writing the calculation result in the second buffer into the external memory; In response to the pingpong flag being false, a convolution calculation is performed in the second buffer, and at the same time, the calculation result in the first buffer is written to the external memory.

5. The FPGA-based target detection network hardware acceleration system according to claim 3, characterized in that: The convolution calculation unit replaces the multiplication operation in the convolution calculation with shift and addition operations, and at the same time splits the complex conditional judgment statements, converts them into independent Boolean expressions and calculates part of the results in advance.

6. The FPGA-based target detection network hardware acceleration system according to claim 3, characterized in that: Based on the data reuse strategy, the number of data transfers is reduced by sharing intermediate calculation results, including: In convolution calculation, the feature map is cached in the on-chip memory or register in the cache unit, so that multiple convolution kernels in the convolution calculation unit can process the same feature map in parallel; Some intermediate calculation data are temporarily stored in the on-chip registers in the cache unit for subsequent calculations.

7. The FPGA-based target detection network hardware acceleration system according to claim 1, characterized in that: Channel expansion technology is used to simultaneously calculate the convolution results of multiple output channels, including: Load feature map data from external memory and distribute the same feature map data to multiple convolution computing units; Multiple convolutional computing units calculate the results of multiple channels in parallel. After each convolutional computing unit completes the calculation, it transmits its respective convolution results to the next layer of the convolutional neural network.

8. The FPGA-based target detection network hardware acceleration system according to claim 1, characterized in that: Convert floating-point numbers to fixed-point numbers based on the quantization strategy, and optimize by adjusting the bit width of the integer and fractional parts of the fixed-point numbers, including: Determine the total bit width, integer bit width and decimal bit width of the fixed-point number to limit the representation range of the fixed-point number; Adjust the allocation ratio of integer and decimal bit width of fixed-point numbers, giving priority to increasing the bit width of the decimal part to improve the accuracy of numerical representation; Select the rounding quantization method to convert the floating point number to a fixed point number; Determine the scaling factor, map the floating-point number to the fixed-point number range, and calculate the quantized fixed-point number using the following formula: ； Where Q is the quantized fixed-point number; round is the rounding function; x is the original floating-point number; Z is the zero point for asymmetric quantization; for symmetric quantization, Z=0; and s is the scaling factor.

9. The FPGA-based target detection network hardware acceleration system according to claim 1, characterized in that: By loading feature map data in blocks and combining pipeline calculations, the maximum value in the pooling window is calculated in a sliding window manner; A multi-channel parallel pooling structure is used to implement pooling calculations of multiple channels simultaneously, including: By using the block loading method, the feature map data is divided into several small blocks according to the channel, height and width, and loaded into the cache unit in parallel; In each pooling operation, the data in the cache unit is expanded into multiple independently accessible arrays; pipeline instructions are applied in each loop iteration to execute each stage of the pooling operation concurrently, so that each clock cycle processes a data element in the pooling window; Compare and calculate the feature values ​​in the sliding window, determine the maximum value, and store the pooling result as a downsampled feature map; Through the multi-channel parallel pooling structure, the inner channel loop is expanded so that the pooling calculations of multiple channels can be performed simultaneously.

Images