Neural network based image acceleration device

Abstract

The present application relates to the technical field of FPGA hardware acceleration and neural network calculation, in particular to an image acceleration system based on neural network. The present application constructs an acceleration architecture composed of multiple DMA controllers and modular hardware operators. The operators adopt a unified interface and a ping-pong buffer design, and are scheduled by a register unit according to instructions. By fusing dequantization, convolution, addition and re-quantization into a single integer calculation, and using a pre-computation lookup table method to realize efficient quantization reasoning, the calculation delay and resource consumption are significantly reduced. Through quantization fusion and parallel computing architecture, the throughput is greatly improved. The hardware resource consumption is significantly reduced, which is suitable for low-power edge scenarios. The modular design supports flexible configuration of network structure, which meets the high real-time detection demand while ensuring the accuracy.

Description

Technical Field

[0001] This invention relates to the field of FPGA hardware acceleration and neural network computing technology, specifically to an image acceleration device based on neural networks. Background Technology

[0002] Against the backdrop of the rapid development of artificial intelligence and computer vision technologies, image processing has shifted from traditional basic operations such as filtering and noise reduction to intelligent tasks relying on deep neural networks, such as object detection and image segmentation. These tasks, with their powerful feature extraction and fitting capabilities, have shown great potential in fields such as industrial quality inspection, autonomous driving, and security monitoring.

[0003] However, high-precision neural network models, such as the YOLO series, place stringent demands on hardware computing power due to their massive convolution and pooling operations. This sharply contradicts the core requirements of edge deployment scenarios for low power consumption, low cost, and high real-time performance. Therefore, exploring dedicated acceleration solutions that can achieve high-performance neural network inference with limited resources has become crucial for driving technology adoption. Currently, mainstream techniques for implementing neural network image processing typically follow this process: First, training and inference of floating-point models are performed on general-purpose processors or graphics processing units using frameworks such as TensorFlow and PyTorch. To reduce deployment costs, model pruning and quantization techniques are commonly used to compress the model. Subsequently, attempts are made to deploy the optimized model to hardware platforms such as FPGAs. This process often relies on high-level synthesis tools or manually writing hardware description languages ​​to map various computational operations in the network to specific hardware circuits. The common approach of existing technologies is to improve efficiency on general-purpose hardware through software optimization or by designing dedicated acceleration units for specific algorithms.

[0004] The aforementioned existing technologies have inherent technical shortcomings. First, GPU-based solutions are too power-consuming and expensive in edge computing scenarios, hindering widespread adoption. Second, simple model quantization introduces accuracy loss, and traditional quantization deployment processes incur significant hardware overhead, failing to fully leverage the parallel computing advantages of FPGAs. More importantly, existing hardware acceleration solutions typically lack flexibility, with tight coupling between computing units and inefficient coordination between data flow and control flow, causing computing units to frequently wait for data transfer, thus limiting the overall system throughput. Furthermore, the lack of efficient hardware adaptation design for complex operators such as Focus and SPPF in YOLOv5 often becomes a performance bottleneck. These shortcomings collectively restrict the efficient and high-precision deployment of neural networks in resource-constrained environments. Summary of the Invention

[0005] This application provides an image acceleration device based on neural networks, which solves the problems of high power consumption, low hardware resource utilization, inflexible data flow scheduling, and lack of efficient hardware support for special operators in networks such as YOLO faced by existing neural network deployment schemes based on GPUs or traditional FPGAs in edge scenarios.

[0006] To achieve the above objectives, the embodiments of this application disclose the following technical solutions:

[0007] This solution discloses an image acceleration device based on neural networks, including:

[0008] The computing module is used to perform forward inference calculations for a low-bit quantized neural network model. The computing module includes multiple hardware operator units and a register control unit. The register control unit receives and parses control instructions from the host computer and dynamically schedules the execution order of the multiple hardware operator units, distributing start signals, address information, and calculation parameters to the multiple hardware operator units. Each hardware operator unit is used to perform a specific type of computational operation in the neural network model.

[0009] The data link module includes multiple DMA controllers for data interaction with external DDR memory via a bus interface protocol. The DMA controllers include a first DMA for communication with the host computer, a second DMA for instruction interaction, a third DMA for writing data to external DDR memory, and a fourth DMA for reading data from external DDR memory.

[0010] Furthermore, the hardware operator unit is configured to: transform the integer operation process for quantized feature map data into a single integer calculation process that integrates dequantization, arithmetic operations, and requantization.

[0011] Furthermore, the calculation module performs a single integer calculation process based on the following fusion quantization formula:

[0012] ;

[0013] in, , , These represent the input feature map data, quantization weights, and output quantized feature map data, respectively. K is a constant derived from the quantization factor, and BiasTerm is a constant term pre-calculated with the bias and zero-point correction terms.

[0014] Furthermore, each hardware operator unit adopts a three-layer pipeline structure, including an input buffer, a computing core, and an output buffer connected in sequence; the input buffer and the output buffer adopt a ping-pong buffering mechanism.

[0015] Furthermore, the hardware operator unit includes a convolution operator unit, which is configured to perform parallel computation, that is, to process data from multiple input channels simultaneously in one computation process and generate computation results from multiple output channels in parallel.

[0016] Furthermore, the hardware operator unit includes a Focus operator unit, which is configured to: reorganize the multi-channel quantized feature map data at different spatial locations in the input feature map data according to a preset data arrangement rule, map at least some spatial dimension information to the channel dimension, generate a target feature map data stream with increased channel number and reduced spatial size, and output the target feature map data stream to the subsequent hardware operator unit for processing.

[0017] Furthermore, the hardware operator unit includes an SPPF operator unit, which is configured to: receive quantized feature map data from multiple parallel input feature map data streams, and perform element-wise maximum value comparison operations on the quantized feature map data on multiple parallel channels. The SPPF operator unit includes at least a two-stage pipelined maximum value comparison structure.

[0018] The first-level maximum comparison structure is used to compare the maximum values ​​of the input feature map data stream and output multiple local maximum values.

[0019] The second-level maximum value comparison structure compares multiple local maximum values ​​output by the first-level maximum value comparison structure to generate the global maximum value for the corresponding channel.

[0020] Furthermore, the hardware operator unit includes a Cat operator unit. The Cat operator receives quantized feature map data from at least two different computation branches and performs quantization parameter alignment processing on the quantized feature map data before the quantized feature map data enters the fusion operation to generate a feature map data stream output with unified quantization parameters.

[0021] Furthermore, the hardware operator unit includes an Add operator unit, which is configured to: first perform quantization parameter alignment and dequantization on two input feature map data streams with independent quantization parameters, then perform element-wise addition, and finally quantize the result back into a feature map data stream with unified quantization parameters for output.

[0022] Furthermore, the hardware operator unit includes an Upsample operator unit, which includes an input controller and an output controller. The Upsample operator unit is configured to: read source feature map data line by line from external DDR memory through the input controller, cache the line data, and perform repeated write-back operations on the same line of cached data by the output controller, thereby enlarging the spatial size of the source feature map data by a specified factor.

[0023] This invention provides a high-efficiency neural network image acceleration device. Through innovative hardware architecture and quantization fusion technology, it effectively solves the core challenges of high power consumption, high latency, and insufficient flexibility in edge deployment. The device employs a parallel pipeline composed of multi-DMA controllers and modular operators, significantly improving data throughput and resource utilization. The core technology lies in fusing dequantization, convolution, and requantization into a single integer computation, combined with a pre-computation lookup table method, which greatly reduces computational latency and logical overhead. The convolution operator unit supports multi-channel parallelism; the focus operator unit achieves spatial transformation through hardware rearrangement; the SPPF operator unit adopts a two-stage pipelined comparison tree, all of which have undergone efficient hardware adaptation for typical network operators. The register unit supports both autonomous and slave modes, enhancing the device's flexibility and debuggability. Attached Figure Description

[0024] Figure 1 This is an overall device block diagram according to an embodiment of the present invention;

[0025] Figure 2 This is a data link flow diagram according to an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of the general three-layer structure of the operator in an embodiment of the present invention;

[0027] Figure 4 This is a schematic diagram illustrating the parallelism of convolution computation in an embodiment of the present invention;

[0028] Figure 5 This is a schematic diagram of the data structure for implementing double multiplication using DSP bit splicing in an embodiment of the present invention;

[0029] Figure 6 This is a schematic diagram of the convolution channel summation pipeline according to an embodiment of the present invention;

[0030] Figure 7 This is a diagram illustrating the computational effect of the Focus operator unit in an embodiment of the present invention.

[0031] Figure 8 This is a schematic diagram of the SPPF operator unit operation in an embodiment of the present invention. Detailed Implementation

[0032] Specific embodiments of the invention will now be described in detail. Although the invention is described in conjunction with these specific embodiments, it should be understood that the invention is not intended to be limited to these specific embodiments. Rather, these embodiments are intended to cover alternative, modified, or equivalent embodiments that may be included within the spirit and scope of the invention as defined by the claims. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the invention. The invention may be practiced without some or all of these specific details. In other instances, well-known processes have not been described in detail so as not to unnecessarily obscure the invention.

[0033] When used in conjunction with the terms "comprising," "method comprising," or similar language in this specification and appended claims, the singular forms "a," "some," and "the" include plural references unless the context clearly indicates otherwise. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0034] This solution discloses an image acceleration device based on neural networks, such as... Figure 1 ,include:

[0035] The computation module is used to perform forward inference computation of a low-bit quantized neural network model. The computation module includes multiple hardware operator units and a register control unit (Reg). The register control unit receives and parses control instructions from a host computer (such as a CPU) and dynamically schedules the execution order of the multiple hardware operator units, distributing start signals and computation parameters to the multiple hardware operator units. Each hardware operator unit is used to perform a specific type of computational operation in the neural network model, and based on the received computation parameters, the integer operation process for the quantized feature map data is equivalent to a single computational flow that integrates dequantization, arithmetic operations, and requantization. Forward inference computation refers to the computational process of sequentially executing computational operations of each layer from the input end according to the network structure of the neural network model, given that the parameters of the neural network model are already determined, to obtain the output result.

[0036] Data link module, such as Figure 2 It includes multiple DMA controllers for data interaction with external DDR memory via the AXI4 protocol, including:

[0037] The first DMA is used for bidirectional data communication with the host computer via the AXI4 protocol, receiving instructions and data from the host computer and writing them to the external DDR memory, and reading the processing results from the external DDR memory and returning them to the host computer.

[0038] The second DMA is used to read instructions from external DDR memory via the AXI4 protocol and provide them to the register control unit;

[0039] The third DMA is a dedicated 128-bit wide write channel used to write data generated by the computing module to external DDR memory via the AXI4 protocol.

[0040] The fourth DMA is a dedicated 128-bit wide read channel used to read the data required for computation from external DDR memory via the AXI4 protocol and provide it to the computing module;

[0041] The data link module uses multiple DMA controllers for data reading and writing, thereby improving the efficiency of data reading and writing and avoiding the problem of bus congestion when a single DMA controller is reading and writing.

[0042] The device architecture of this invention is divided into two main parts: a data link module and a computing module. The core of the data link module consists of multiple DMA controllers, which serve as high-speed data channels between the computing module and external DDR memory. These DMA controllers are all implemented using standard IP cores conforming to the AXI4 bus protocol to ensure high-speed and reliable data transmission. The computing module includes multiple hardware operator units and a unified register control unit (Reg). In this scheme, the hardware operator units include: a convolution operator unit, a focus operator unit (Focus), a stitching operator unit (Cat), a spatial pyramid pooling fast operator unit (SPPF), an addition operator unit (Add), and an upsampling operator unit (Upsample).

[0043] Each hardware operator unit adopts a pipelined structure, including sequentially connected input buffers, computational cores, and output buffers; the input buffers and output buffers employ a ping-pong buffering mechanism.

[0044] The register control unit is the unified scheduling core of the entire computing module. It integrates a ROM to store pre-written microinstruction sequences that describe the structure of specific neural networks (such as YOLOv5n). This allows each hardware operator unit to run independently and be coordinated and scheduled in strict accordance with the order and logic defined by the instructions in the ROM. In actual use, the execution order of operators and the network structure can be flexibly changed simply by updating the instruction information stored in the ROM, thereby achieving hardware adaptation for different neural network models.

[0045] Each hardware operator unit is designed as follows: Figure 3The three-layer pipeline structure shown includes an input buffer (Inbuf), a computing core (Calc), and an output buffer (Outbuf). Both the input and output buffers employ a ping-pong buffering mechanism, namely an input ping-pong buffer and an output ping-pong buffer. This mechanism allows the two buffer units to work alternately. When one buffer unit (such as the Ping unit) is providing data to the computing core, the other buffer unit (such as the Pong unit) can simultaneously prefetch the next batch of data from the external DDR memory through the DMA controller of the data link module. The two switch roles in turn according to control signals, thereby hiding the time-consuming external data access delay within the computing process, realizing a continuous and stable supply of data flow, and eliminating the waiting time between the data input and output buffers and the computing core.

[0046] All hardware operator units are connected to the data link module via a standard streaming interface based on valid / ready handshake signals, thus forming a high-efficiency data stream processing pipeline. This device is used to execute neural network models after low-bit quantization. Its core innovation lies in integrating the traditionally complex process of dequantizing to floating-point numbers, performing calculations, and then requantizing into a one-step integer calculation process, thereby greatly improving computational efficiency.

[0047] This solution further proposes that the operating modes of the register control unit include:

[0048] Autonomous operation mode: Automatically executes instructions one by one according to the internally stored instruction sequence;

[0049] Slave mode: Responds to instructions sent by the host computer to perform single-step or specific operations.

[0050] In autonomous operation mode, after receiving a start signal from the host computer, the register control unit automatically reads the pre-stored microinstruction sequence sequentially from its internal instruction ROM. For each instruction read, the corresponding start signal, address information, and calculation parameters are distributed to the hardware operator unit and data link module. This process repeats until the entire instruction sequence in the ROM has been executed, thus achieving fully automated, pipelined scheduling of the entire neural network's forward inference computation.

[0051] In slave mode, the register control unit acts as a slave device to the host computer, and its execution flow is entirely driven by the host computer. The host computer sends operation commands to the register control unit in real time and sequentially via the device bus. Upon receiving each command, the register control unit immediately decodes and executes it, and then reports the execution status back to the host computer. This mode enables single-step debugging of the hardware calculation process, non-sequential operations, and dynamic parameter configuration capabilities, providing great flexibility for device development and verification.

[0052] This solution further proposes that the hardware operator unit includes a convolution operator unit, which is configured to perform parallel computation, specifically, to process data from multiple input channels simultaneously and generate computation results from multiple output channels in parallel.

[0053] In this embodiment, the convolution operator unit includes a 1×1 convolution operator unit (Conv11) and a 3×3 convolution operator unit (Conv33), employing a high-parallelism architecture to achieve high-speed computation. Specifically, as... Figure 4 The convolution operator unit is a parallel processing array capable of simultaneously processing data from 16 input channels and convolution kernels from 8 output channels. For a 3x3 convolution operator unit (Conv33), this array can perform 16 (input channels) x 8 (output channels) x 9 (3x3 kernels) = 1152 multiply-accumulate operations per clock cycle. Figure 5 To optimize the use of valuable DSP resources in FPGA, bit splicing technology is adopted for DSP units in FPGA. Common DSP units support 27×18-bit unsigned operations. In this embodiment, two 12-bit input data are spliced ​​into 24 bits, and two 4-bit weights are spliced ​​into 8 bits. Each multiplier unit performs one 24×8 multiplication, which can be split into two 16-bit products. Thus, a single DSP can complete two multiplication operations in one cycle. Finally, only 576 DSP units are needed to complete 1152 operations, doubling the resource utilization rate.

[0054] like Figure 6 After the multiplication is completed, the 16 channels of data are accumulated using a four-stage pipeline addition method. This embodiment requires a maximum of 8 adders, which greatly saves logic resources while ensuring throughput.

[0055] In this embodiment, the 1×1 convolution operator unit (Conv11) is similar to the 3x3 convolution operator unit in that it also uses 16 input channels and 8 output channels; the difference is that the size of the convolution kernel is 1×1.

[0056] This solution further proposes that the hardware operator unit includes a Focus operator unit, which is configured to: reorganize the multi-channel quantized feature map data at different spatial locations in the input feature map data according to a preset data arrangement rule, map at least some spatial dimension information to the channel dimension, generate a target feature map data stream with increased channel number and reduced spatial size, and output the target feature map data stream to the subsequent hardware operator unit for processing.

[0057] In one specific implementation, since the input feature map data is stored in a pixel-ordered, channel-data-continuously-arranged manner, and the data link supports multi-channel parallel transmission, the Focus operator unit can reorganize the input feature map data stream to map spatial dimension information to channel dimension, thereby completing the Focus operation.

[0058] like Figure 7 The input feature map is in the form of a two-dimensional image, which can be represented as... The input feature map data is multi-channel feature map data with coordinates, where each spatial location corresponds to a set of input channel data. The input feature map data is arranged in memory or data link as follows: arranged sequentially according to spatial coordinates. In the data of each spatial location, the data of multiple input channels are stored continuously in the data stream. That is, the data of all input channels corresponding to the first spatial location are arranged first, then the data of all input channels corresponding to the second spatial location are arranged, and so on.

[0059] In this embodiment, the input feature map data is transmitted using a wide data bus, with a single transmission width of 1128 bits, and supports parallel transmission of 16 channels. Based on the above data organization method and parallel transmission characteristics, the Focus operator unit selects and reorganizes the input data stream according to a preset spatial location and channel correspondence, reorganizing multi-channel data from different spatial locations into a channel dimension, thereby generating a target feature map data stream with an increased number of channels and a correspondingly reduced spatial size.

[0060] The Focus operator unit, without introducing arithmetic operations, reorganizes the spatial information in the original two-dimensional feature map by selecting and arranging the corresponding channel data in the input feature map data stream. The changes in data form before and after processing are shown in the figure. The target feature map data stream processed by the Focus operator unit is output to subsequent hardware operator units scheduled by the register control unit for performing subsequent convolution calculations or other hardware operator unit operations.

[0061] This solution further proposes that the hardware operator unit includes an SPPF operator unit, which is configured to: receive quantized feature map data from multiple parallel input feature map data streams, and perform element-wise maximum value comparison operations on the quantized feature map data on multiple parallel channels. The SPPF operator unit includes at least a two-stage pipelined maximum value comparison structure.

[0062] The first-level maximum comparison structure is used to compare the maximum values ​​of the input feature map data stream and output multiple local maximum values.

[0063] The second-level maximum value comparison structure compares multiple local maximum values ​​output by the first-level maximum value comparison structure to generate the global maximum value for the corresponding channel.

[0064] In one specific embodiment, such as Figure 8 As shown, the SPPF operator unit in the device is used to perform multi-scale maximum feature extraction and fusion processing on the quantized feature map data of the input feature map data stream. This SPPF operator unit is deployed in the neural network feature extraction stage, and its input is the quantized feature map data output from the upstream operator unit. It generates the fused target feature map data through a parallel maximum value comparison mechanism.

[0065] In this embodiment, the SPPF operator unit receives multiple parallel input feature map data streams. Each input feature map data stream is transmitted using a fixed-width data format and is processed simultaneously on multiple parallel channels. Specifically, the SPPF operator unit performs an element-wise maximum value comparison operation on each input feature map data stream, thereby achieving the fusion of feature information at different scales while maintaining the feature space resolution.

[0066] To support the processing of feature map data with spatial or temporal continuity, the SPPF operator unit constructs a sliding window structure in each input feature map data stream. The sliding window is implemented through multi-level register cascading, used to cache feature map data from adjacent positions or adjacent time steps within consecutive clock cycles. This allows the SPPF operator unit to simultaneously access feature map data sets from multiple input feature map data streams and multiple adjacent positions at any given processing time.

[0067] During feature computation, the SPPF operator unit splits the feature map data in the sliding window into multiple parallel processing channels, each corresponding to an independent set of feature map components. For each parallel channel, the SPPF operator unit constructs a hierarchical pipeline maximum value comparison structure to quickly select the maximum value from multiple input feature map data.

[0068] Specifically, the maximum value comparison structure includes at least two pipeline processing stages: in the first pipeline, for each input feature map data stream, multiple sets of feature map data within its sliding window are compared and calculated to obtain the intermediate maximum value of the corresponding input feature map data stream; in the second pipeline, the intermediate maximum values ​​generated by each input feature map data stream are further compared to obtain the final maximum value result of the corresponding parallel channel.

[0069] After completing the maximum value calculation for each parallel channel, the SPPF operator unit reassembles the calculation results of each parallel channel to form a unified output feature map data stream, and sends the output feature map data stream to the subsequent hardware operator unit for feature fusion or convolution processing to continue the subsequent calculation of the neural network.

[0070] In addition, to ensure the consistency between the feature map data stream and the control signal during pipeline processing, the SPPF operator unit performs synchronous delay compensation on the valid flag signal and boundary flag signal corresponding to the input feature map data, so that the timing of the control signal is consistent with the processing delay of the maximum value comparison pipeline, thereby ensuring the integrity and timing accuracy of the output feature map data in continuous feature map data stream processing scenarios.

[0071] Through the above structural design, the SPPF operator unit can achieve high parallelism and high throughput multi-scale feature map data maximum value fusion processing in a hardware environment, effectively expanding the feature perception range while reducing computational latency and hardware resource consumption.

[0072] This solution further proposes that the calculation module performs a single integer calculation process based on the following fusion quantization formula:

[0073] ;

[0074] in, , , These represent the input feature map data, quantization weights, and output quantized feature map data, respectively. K is a constant derived from the quantization factor, and BiasTerm is a constant term pre-calculated with the bias and zero-point correction terms.

[0075] The single integer calculation process is specifically implemented by the fusion quantization formula derived below. First, the basic quantization formula is: ;

[0076] in, It refers to the quantized integer. This refers to the floating-point number before quantization. To quantize the scaling factor, For quantization zero point;

[0077] Using this formula, floating-point values ​​are linearly mapped to the integer range, and the dequantization formula is: ;

[0078] This means that the original floating-point number r can be obtained by zero-point correction of the quantized integer q and then multiplying it by the scaling factor s;

[0079] Quantization formula for convolution calculation: +bias;

[0080] in, and These are the floating-point values ​​of the input features and weights, respectively, and bias is the bias of the convolution or BN layer;

[0081] After model training, the network weights remain fixed, so they can be treated as constants in the inference phase. Substituting the dequantization formula into the above convolution calculation formula, the formula becomes: ;

[0082] Substituting the dequantization formula, we get: ;

[0083] The weights are symmetrically quantized, so =0, the formula can be derived as follows:

[0084] = ;

[0085] = ;

[0086] ;

[0087] = * ;

[0088] The formula implemented in the FPGA is the one described above, where s1, s2, s3, z3, z1, and q2 have definite values ​​after training. Therefore, in this design, we can... * As a whole, it is calculated in advance that the FPGA only needs to perform table lookups. In addition, the batch layer has a bias, so the formula should become:

[0089] ;

[0090] = * + ;

[0091] make ,make Then we obtain the final formula for hardware calculation:

[0092] ;

[0093] Where K is a pre-calculated constant, and BiasTerm is the weighted average of the two variables. Related pre-calculated items.

[0094] This solution further proposes, in the hardware implementation of feature map data on FPGA, a quantization scaling factor. , , Zero-point parameter , and weight All constants are determined as fixed values ​​during the model deployment phase. Therefore, this solution pre-calculates the constants in the above formulas and performs only integer multiplication and addition operations in the hardware. Since the bias term is related to the weights, this solution obtains the BiasTerm term corresponding to the weight values ​​using a lookup table method. Specifically, it pre-calculates the bias term based on the network parameters and weights. The BiasTerm values ​​corresponding to all possible values ​​are stored in the FPGA's block memory to form a lookup table, which is used during calculation. Use the address index to read the corresponding value from the table.

[0095] The BiasTerm item is obtained through a table lookup method. During the model deployment phase, the host computer software pre-calculates the trained network parameters (s1, s2, s3, z1, z3, bias) and weights. All possible values ​​for the INT8 weights, ranging from -128 to 127, totaling 256 values, are calculated for each. The corresponding BiasTerm value. After calculation, these BiasTerm values ​​are quantized into 32-bit fixed-point numbers and converted into unsigned addresses from zero to 255 according to the weight q2 value, initialized as a lookup table (LUT) with a depth of 256 and a bit width of 32. In the FPGA implementation, this lookup table is implemented using a block of BlockRAM (BRAM) resource. When the computation unit is working, the current weight is... As a read address input to BRAM, the corresponding BiasTerm value can be read after one clock cycle, and used for subsequent multiplication and accumulation operations.

[0096] This solution further proposes that the hardware operator unit includes a Cat operator. The Cat operator receives quantized feature map data from at least two different computation branches and performs quantization parameter alignment processing on the quantized feature map data before the quantized feature map data enters the fusion operation to generate a feature map data stream output with unified quantization parameters, so as to facilitate subsequent splicing, addition or other fusion operations on the feature map data.

[0097] During the inference process of a neural network, feature map data generated by different computational branches may be represented using different quantization parameters. To ensure the correctness of subsequent fusion operations, the Cat operator unit performs a unified quantization domain conversion on the quantized feature map data from different sources at the hardware level.

[0098] Specifically, in this embodiment, the Cat operator unit sequentially receives input data streams from at least two different data sources through the same physical input port. The input data streams are time-multiplexed, and data from different sources are identified by valid input signals. In this embodiment, data from different sources correspond to different sets of quantization parameters, including their respective scaling factors and zero-point parameters.

[0099] When receiving the input data stream, the Cat operator unit dynamically selects the corresponding set of quantization parameters based on the valid input signal, and synchronously transmits the selected quantization parameters along the data processing pipeline to ensure that the input data matches the correct quantization parameters in each processing stage.

[0100] In the specific calculation process, the Cat operator unit first performs equivalent inverse quantization processing on the input quantized feature map data. The inverse quantization processing includes: offset correction of the input quantized integers based on the zeros in the selected quantization parameters, and fixed-point transformation of the corrected data in combination with the selected scaling factor to obtain data represented with intermediate precision.

[0101] After completing the dequantization process, the Cat operator unit further performs requantization on the intermediate precision data to uniformly map data from different sources to the target quantization domain. The requantization process includes: performing an equivalent scaling operation on the intermediate precision data according to preset target quantization parameters, and converting the scaled result into a quantized integer representation of the target bit width.

[0102] To ensure the accuracy and numerical stability of the quantization conversion results, the Cat operator unit performs rounding and saturation processing on the conversion results during the requantization process, so that the output data falls within the effective representation range of the target quantized integer.

[0103] In this embodiment, the Cat operator unit adopts a multi-stage pipeline structure to implement the above-mentioned equivalent inverse quantization and equivalent requantization processes. By delaying and aligning the input valid signal, it ensures that the data stream and its corresponding quantization parameter selection information are strictly synchronized in each stage of the pipeline, thereby achieving high throughput data processing capability.

[0104] In this way, the Cat operator unit can perform a unified quantization domain conversion on feature map data from different computation branches with different quantization parameters without adding additional physical input ports, and output the processed feature map data to subsequent hardware operator units for fusion processing.

[0105] This solution further proposes that the hardware operator unit includes an Add operator unit, which is configured to: first perform quantization parameter alignment and equivalent dequantization on two input feature map data streams with independent quantization parameters, then perform element-wise addition, and finally quantize the result into a feature map data stream with unified quantization parameters for output.

[0106] In this embodiment, the Add operator unit is used to perform element-wise addition operations on feature map data from different computational branches in the neural network. The Add operator unit is suitable for high-speed parallel addition processing of quantized feature map data.

[0107] Specifically, in this embodiment, the Add operator unit receives two parallel input data streams. Each input data stream is transmitted using a fixed-width data format, such as a 128-bit wide data stream, which contains multiple quantized feature map data in parallel. In this embodiment, each data stream may contain 16 8-bit quantized data in parallel. The number and bit width are only an example implementation and do not constitute a limitation of the present invention.

[0108] The Add operator unit adopts a multi-channel parallel processing structure and pipeline architecture to perform element-wise addition operations on each quantized feature map data in the input data stream, so as to achieve high throughput data processing capability, enabling multiple sets of feature map data addition operations to be completed in parallel within each clock cycle.

[0109] In the specific calculation process, the Add operator unit first performs quantization parameter alignment processing on the two input data streams. Specifically, the Add operator unit performs an equivalent dequantization operation on the input quantized integers according to the quantization parameters corresponding to the two input data streams. The equivalent dequantization operation includes: subtracting the corresponding zero-point offset from the input quantized integer, and performing a fixed-point transformation in combination with the corresponding scaling factor to obtain intermediate representation data for subsequent calculations. In one embodiment, to avoid negative values ​​from adversely affecting subsequent calculations, input data with values ​​less than zero can also be truncated during the equivalent dequantization process.

[0110] After quantization parameter alignment is completed, the Add operator unit performs element-wise addition on the two equivalent dequantized intermediate representation data to obtain intermediate data of the addition result. The element-wise addition operation is performed simultaneously in multiple parallel channels, thereby improving the overall computational efficiency.

[0111] After the addition operation is completed, the Add operator unit performs equivalent requantization processing on the intermediate addition result. The equivalent requantization processing includes: performing an equivalent scaling operation on the addition result according to preset output quantization parameters, and converting the scaled result into a quantized integer representation of the target bit width. In one embodiment, the equivalent scaling operation can be implemented through multiplication, shifting, or table lookup.

[0112] To ensure the accuracy and numerical stability of the quantization results, the Add operator unit further performs rounding and saturation processing on the addition results during the equivalent requantization process. Specifically, by rounding the decimal part of the scaling result, the operation result is approximated to the integer part; at the same time, values ​​that exceed the target quantization integer representation range are clamped to ensure that the final output result falls within the allowed numerical range.

[0113] The Add operator unit adopts a multi-stage pipeline structure throughout the data processing path, and uses shift registers and other methods to delay and align the data path with the corresponding control signals to ensure that the output addition result data is synchronized with the valid identification signal, thereby ensuring the correctness and timing consistency of the data output.

[0114] In this way, the Add operator unit completes the equivalent dequantization, element-wise addition, and equivalent requantization of the quantized feature map data at the hardware level, realizing quantization addition operations with high parallelism and high throughput, and can output the calculation results to subsequent hardware operator units for further processing.

[0115] This solution further proposes that the hardware operator unit includes an Upsample operator unit, used to perform upsampling processing on the feature maps in the quantized neural network. The Upsample operator unit includes an input controller and an output controller. The Upsample operator unit is configured to: read source feature map data line by line from external DDR memory through the input controller, cache the line data, and have the output controller repeatedly write back the same line of cached data, thereby enlarging the spatial size of the source feature map data by a specified factor. This achieves feature map spatial size enlargement based on a nearest neighbor approach.

[0116] Specifically, in this embodiment, the Upsample operator unit is configured to perform a spatial size magnification operation based on a preset upsampling factor. In this embodiment, the upsampling factor is 2×, which copies the row and column data of the input feature map, making the output feature map double in both height and width relative to the input feature map. It should be noted that 2× upsampling is only an example implementation in this embodiment; in other embodiments, the upsampling factor can be configured according to actual needs.

[0117] The Upsample operator unit includes an input controller and an output controller, which work together to complete the data reading, buffering, and write-back operations during the upsampling process. The Upsample operator unit itself does not perform explicit arithmetic copying operations on pixel values, but rather achieves spatial magnification of the feature map by scheduling the access order of the data stream and the storage address.

[0118] In the specific implementation, the upsampling calculation process is initiated by the input controller. The input controller receives control instructions from the register control unit, which include parameters such as the starting address of the source feature map in external memory, the number of rows in the feature map, and the length of each row. Based on these parameters, the input controller generates a corresponding memory read access request and reads the source feature map data row by row from the external DDR memory via the bus interface.

[0119] When a row of source feature map data is read and input to the Upsample operator unit through the data interface, the input controller buffers the row of data for subsequent access by the output controller. The buffer can be implemented using a register array, FIFO, or on-chip memory, and its specific structure is not limited.

[0120] The row copying operation during the upsampling process is performed by the output controller. Once the output controller detects that the corresponding row data has been cached, it performs at least two sequential write-back operations on the same cached row data. Specifically, the output controller first writes the row data back to the corresponding storage location of the target feature map, and then writes the same row data back again to the adjacent target storage address, thereby generating two rows of identical feature map data in the output feature map to complete the row-wise upsampling process.

[0121] In the column direction, the Upsample operator unit achieves column replication through the organization of the data stream. In this embodiment, the input and output data streams use the same data bit width, for example, a 128-bit data stream, and each data packet contains multiple pixel data in parallel. When a row of data is repeatedly written back to the output feature map, the multiple pixel data contained in the data packet are also repeatedly written, thereby completing the pixel replication in the column direction in an equivalent sense and achieving the upsampling effect in the column direction.

[0122] During the write-back process, the output controller dynamically calculates the target storage address corresponding to each write-back operation based on the upsampling factor and the row and column size of the feature map, so as to ensure that the repeatedly written-back data can be correctly mapped to the corresponding spatial location in the output feature map.

[0123] After all source feature map rows have completed the process of reading once and writing back repeatedly, the Upsample operator unit outputs a calculation completion signal to indicate that the current upsampling operation is over and to trigger subsequent hardware operator units to further process the upsampled feature map data.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them; although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can still be made to the specific implementation methods of the present invention or equivalent substitutions can be made to some technical features without departing from the spirit of the technical solutions of the present invention, and all such modifications should be covered within the scope of the technical solutions claimed in the present invention.

Claims

1. An image acceleration device based on neural networks, characterized in that, include: The computing module is used to perform forward inference calculations for a low-bit quantized neural network model. The computing module includes multiple hardware operator units and a register control unit. The register control unit receives and parses control instructions from the host computer and dynamically schedules the execution order of the multiple hardware operator units, distributing start signals, address information, and calculation parameters to the multiple hardware operator units. Each hardware operator unit is used to perform a specific type of computational operation in the neural network model. The data link module includes multiple DMA controllers for data interaction with external DDR memory via a bus interface protocol. The DMA controllers include a first DMA for communication with the host computer, a second DMA for instruction interaction, a third DMA for writing data to external DDR memory, and a fourth DMA for reading data from external DDR memory. The calculation module performs a single integer calculation process based on the following fusion quantization formula: ； in, , , These are the input feature map data, quantization weights, and output quantized feature map data, respectively. K is a constant derived from the quantization factor, and BiasTerm is a constant term pre-calculated with the bias and zero-point correction terms. The hardware operator unit internally includes a lookup table implemented using the BlockRAM resources of the FPGA, which pre-stores values ​​related to the weights. The BiasTerm value corresponding to all possible values; during calculation, the hardware operator unit uses the current weight. The address is input into the lookup table, and the corresponding BiasTerm value is read directly for multiplication and addition operations.

2. The image acceleration device based on a neural network according to claim 1, characterized in that, Each hardware operator unit adopts a three-layer pipeline structure, including a sequentially connected input buffer, a computational core, and an output buffer; the input buffer and the output buffer adopt a ping-pong buffering mechanism.

3. The image acceleration device based on a neural network according to claim 1, characterized in that, The hardware operator unit includes a convolution operator unit, which is configured to perform parallel computation.

4. The image acceleration device based on a neural network according to claim 1, characterized in that, The hardware operator unit includes a Focus operator unit, which is configured to: reorganize multi-channel quantized feature map data at different spatial locations in the input feature map data according to a preset data arrangement rule, map at least some spatial dimension information to the channel dimension, generate a target feature map data stream with increased channel number and reduced spatial size, and output the target feature map data stream to the subsequent hardware operator unit for processing.

5. The image acceleration device based on a neural network according to claim 1, characterized in that, The hardware operator unit includes an SPPF operator unit, which is configured to: receive quantized feature map data from multiple parallel input feature map data streams, and perform element-wise maximum value comparison operations on the quantized feature map data on multiple parallel channels. The SPPF operator unit includes at least a two-stage pipelined maximum value comparison structure. The first-level maximum comparison structure is used to compare the maximum values ​​of the input feature map data stream and output multiple local maximum values. The second-level maximum value comparison structure compares multiple local maximum values ​​output by the first-level maximum value comparison structure to generate the global maximum value for the corresponding channel.

6. The image acceleration device based on a neural network according to claim 1, characterized in that, The hardware operator unit includes a Cat operator unit. The Cat operator receives quantized feature map data from at least two different computation branches and performs quantization parameter alignment processing on the quantized feature map data before the quantized feature map data enters the fusion operation to generate a feature map data stream output with unified quantization parameters.

7. The image acceleration device based on a neural network according to claim 1, characterized in that, The hardware operator unit includes an Add operator unit, which is configured to: first perform quantization parameter alignment and dequantization on two input feature map data streams with independent quantization parameters, then perform element-wise addition, and finally quantize the result back into a feature map data stream with unified quantization parameters for output.

8. The image acceleration device based on a neural network according to claim 1, characterized in that, The hardware operator unit includes an Upsample operator unit, which includes an input controller and an output controller. The Upsample operator unit is configured to: read source feature map data line by line from external DDR memory through the input controller, cache the line data, and have the output controller perform repeated write-back operations on the same line of cached data to enlarge the spatial size of the source feature map data by a specified factor.

Images