Inference accelerator and inference method for YOLOv8m network

Abstract

The application relates to the technical field of neural network models, and provides an inference accelerator and an inference method for a YOLOv8m network. Input activation data and weight data are read in parallel by two data loading units in a data loading module, and the accelerator runs according to an instruction stream generated based on fixed-point quantization, operator fusion and first-layer dimension reorganization, so that the data loading bandwidth utilization is improved, the floating-point calculation resource overhead is reduced, the inter-layer data transfer is reduced, and the hardware parallel degree is matched.

Description

Technical Field

[0001] This application relates to the field of neural network model technology, and in particular to an inference accelerator and inference method for YOLOv8m networks. Background Technology

[0002] When the YOLOv8m (You Only Look Once version 8 medium) object detection network is deployed in edge scenarios such as autonomous driving and drone detection, its numerous convolutional layers, multi-path C2f blocks, and cross-layer concatenation operations pose real-time inference challenges to hardware, including computational intensity, complex data dependencies, strong nonlinearity of activation functions, and limited edge resources. To address the characteristics of the YOLOv8m network structure, accelerators are required to possess high parallel multiply-accumulate capabilities, flexible data flow scheduling mechanisms, and low-complexity nonlinear processing units.

[0003] The programmable neural network inference accelerator based on memory access adopts a modular decoupled architecture. It manages data uniformly through an on-chip memory cell array, achieves task-level parallelism through an instruction dispatch module, and has no direct data path between modules. It also supports a single instruction multi-cycle synchronization mechanism and can achieve mapping of multiple network layers through instruction sequence control. In other words, it improves programmability and network adaptability through unified storage and instruction control.

[0004] However, when processing YOLOv8m, the mismatch between data flow scheduling and network structure leads to low computing power utilization, high resource overhead of floating-point computing units, and frequent data transfer between layers. In particular, the three-channel input of the first layer results in insufficient utilization of parallel computing array and data loading bandwidth. Summary of the Invention

[0005] This application provides an inference accelerator and inference method for YOLOv8m networks to address the problem of insufficient computing power and bandwidth utilization.

[0006] In a first aspect, this application provides an inference accelerator for YOLOv8m networks, comprising:

[0007] The data loading module includes two data loading units for reading input activation data and weight data in parallel;

[0008] The matrix calculation module includes multiple sets of parallel multiply-accumulate arrays. Each set of multiply-accumulate arrays is used to perform convolution operations in parallel based on the input activation data and the weight data, and output the convolution result.

[0009] The vector computation module includes a fixed-point operation unit, which is used to perform fixed-point activation operation, sampling operation and pooling operation on the convolution result and output the processing result.

[0010] The data write-back module is used to write the processing results back to the memory;

[0011] The inference accelerator is configured to run according to an instruction stream generated by the compiler, the instruction stream being generated based on fixed-point quantization, operator fusion, and first-layer dimension reorganization of the YOLOv8m network. The fixed-point quantization converts floating-point operations in the network into fixed-point operations, the operator fusion merges consecutive network layers to reduce intermediate data write-back, and the first-layer dimension reorganization performs dimensional transformation on the first-layer input activation data to match the hardware parallelism.

[0012] Secondly, this application provides an inference method for the YOLOv8m network, applied to an inference accelerator. The inference accelerator includes a data loading module, a matrix calculation module, a vector calculation module, and a data write-back module. The inference accelerator runs according to the instruction stream generated by the compiler. The instruction stream is generated based on fixed-point quantization, operator fusion, and first-layer dimension reorganization of the YOLOv8m network. The fixed-point quantization converts floating-point operations in the network into fixed-point operations. The operator fusion fuses consecutive network layers to reduce intermediate data write-back. The first-layer dimension reorganization transforms the dimensions of the first-layer input activation data to match the hardware parallelism.

[0013] The method includes:

[0014] The input activation data and weight data are read in parallel by two data loading units in the data loading module.

[0015] The matrix calculation module uses multiple parallel multiply-accumulate arrays to perform convolution operations in parallel based on the input activation data and the weight data, and outputs the convolution result.

[0016] The fixed-point operation unit in the vector calculation module performs fixed-point activation, sampling and pooling operations on the convolution result and outputs the processing result.

[0017] The processing result is written back to the memory via the data write-back module.

[0018] As can be seen from the above technical solutions, this application provides an inference accelerator and inference method for YOLOv8m networks. By having two data loading units in the data loading module read input activation data and weight data in parallel, and by having the accelerator run according to the instruction stream generated based on fixed-point quantization, operator fusion and first-layer dimension reorganization, the data loading bandwidth utilization is improved, the floating-point computing resource overhead is reduced, the inter-layer data transfer is reduced, and the hardware parallelism is matched.

[0019] Furthermore, by utilizing the two sets of digital signal processing arrays and the multiply-accumulate tree structure within the matrix computation module, convolution operations on a preset number of input and output channels are executed in parallel, improving the parallelism and computational efficiency of convolution computation. The matrix computation module and vector computation module are controlled by instruction flow to sequentially operate on multiple consecutive convolutional layers, loading only at the first layer and writing back at the last, reducing the storage access overhead of data transfer between layers. By quantizing the input activation data, weight data, convolution result, activation result, dequantization factor, and quantization factor into fixed-point numbers with specific bit widths, the data bit width and transmission bandwidth are reduced while maintaining computational accuracy. The vector computation module, controlled by instruction flow, reads the result from the on-chip cache after activation as sampling input and writes it back after sampling, reducing intermediate data transfer during the write-back and read-back of activation results. Attached Figure Description

[0020] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the network structure of YOLOv8m provided in the embodiments of this application;

[0022] Figure 2 This is a schematic diagram of the structure of a convolutional layer provided in an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of the structure of the upsampling layer provided in an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of the splicing layer provided in an embodiment of this application;

[0025] Figure 5 A schematic diagram of the bottleneck block provided in an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of the structure of the C2f block provided in an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of the structure of an SPPF block provided in an embodiment of this application;

[0028] Figure 8 A schematic diagram of the structure of an inference accelerator for the YOLOv8m network provided in an embodiment of this application;

[0029] Figure 9 A schematic diagram of the data block calculation process before and after optimization provided in the embodiments of this application;

[0030] Figure 10A schematic diagram illustrating the multiplication-accumulation operation performed by an addition tree provided in an embodiment of this application;

[0031] Figure 11 A schematic diagram illustrating the multiply-accumulate array convolution calculation provided in an embodiment of this application;

[0032] Figure 12 The flowchart of the activation operation calculation after quantization is provided for the embodiments of this application;

[0033] Figure 13 Execution provided for embodiments of this application Figure 12 Process flowchart;

[0034] Figure 14 A flowchart illustrating the calculation process before and after fusion of the activation sampling operator provided in an embodiment of this application;

[0035] Figure 15 This is a schematic diagram of the data flow after fusion of n convolutional layers provided in an embodiment of this application;

[0036] Figure 16 The flowchart for the first-layer dimension fusion provided in the embodiments of this application. Detailed Implementation

[0037] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following examples do not represent all embodiments consistent with this application.

[0038] Before explaining this embodiment, let's first introduce the network structure of YOLOv8m. Figure 1 This is a schematic diagram of the YOLOv8m network structure provided in the embodiments of this application. The main body of the entire YOLOv8m network includes convolutional layers, C2f blocks, SPPF blocks (Spatial Pyramid Pooling Fast blocks), sampling layers, splicing layers, bottleneck blocks, and detection heads.

[0039] exist Figure 1 In the diagram, the array in parentheses above each module has the format (C, H, W), which represents the number of channels, height, and width of the output feature map of that module, respectively. For example, (3, 640, 640) means that the initial input of the network is a 3-channel, 640×640 pixel image; (48, 320, 320) means that after "convolutional layer 1", the feature map becomes 48 channels and the spatial size is downsampled to 320×320.

[0040] In a convolutional layer, k represents the size of the convolution kernel, i.e., the kernel height and the kernel width. They have equal values ​​and are both represented by k. s represents the stride of the convolution operation, p represents the padding of the convolution operation, n represents the number of times the bottleneck structure is repeatedly stacked in the C2f block, and r represents the residual connection status within the bottleneck structure. r=0 represents that the residual connection path is closed, and r=1 represents that the residual connection path is open.

[0041] Convolutional layers are the most common part of the YOLOv8m network. During the entire inference process, the inference time of convolutional layers accounts for approximately 90% of the total time. Figure 2 As shown.

[0042] The input dimensions of the convolutional layer are: number of input channels c_i, height of input image h_i, and width of input image w_i. After convolution, normalization, and activation, the output dimensions are c_o, h_o, and w_o.

[0043] For convolutions within convolutional layers, the YOLOv8m network offers three types of convolutions with corresponding parameters: kernel size k = (1, 1), stride s = (1, 1), padding p = (0, 0); kernel size k = (3, 3), stride s = (1, 1), padding p = (1, 1); and kernel size k = (3, 3), stride s = (2, 2), padding p = (1, 1). For the first two types, h_o = h_i and w_o = w_i; for the third type, h_o = h_i / 2 and w_o = w_i / 2.

[0044] For normalization in convolutional layers, the normalization operation can be performed on the weights in advance, without needing to perform the normalization operation during inference.

[0045] For activation in convolutional layers: YOLOv8m uses the SILU (Sigmoid Linear Unit) activation function, and its corresponding mathematical expression is as follows:

[0046] ;

[0047] in, It represents a specific value in the feature map output by the previous layer (usually a convolutional layer).

[0048] The sampling layer is an upsampling layer, such as Figure 3 As shown, Figure 3 This is a schematic diagram of the upsampling layer provided in an embodiment of this application. The upsampling layer amplifies the input size through nearest neighbor upsampling operations. The relationship between the output size and the input size is: c_o = c_i, h_o = 2. h_i, w_o=2 w_i.

[0049] Figure 4 This is a schematic diagram of the stitching layer provided in an embodiment of this application. The stitching layer has two inputs, whose dimensions in the h and w dimensions must be equal, so that the two inputs are stitched together in the channel dimension c. The relationship between the output size and the input size is: c_o = c_i1 + c_i2, h_o = h_i, w_o = w_i.

[0050] Figure 5 The diagram below illustrates the structure of a bottleneck block provided in this embodiment. The bottleneck block is composed of two convolutional layers, designated as convolutional layer 1 and convolutional layer 2. The parameter r represents the residual connection status. r=0 indicates that the residual connection path is closed, and the calculation result of convolutional layer 2 is the output of the bottleneck block. r=1 indicates that the residual connection path is open, and the calculation result of convolutional layer 2 needs to be added to the input of the bottleneck block to obtain the output of the bottleneck block. Figure 5 In the diagram, 'a' represents the case where parameter 'r' is 0, indicating that the residual connection pathways are closed. The kernel size 'k' of convolutional layers 1 and 2 is 3, the stride 's' is 1, and the padding 'p' is 1. Figure 5 In the diagram, b is a schematic of the residual connection path when the parameter r is 1, i.e., the residual connection path is open. The kernel size k of convolutional layer 1 and convolutional layer 2 is 3, the stride s is 1, and the padding p is 1. Depending on the parameter r, the residual path will be opened or closed accordingly.

[0051] Figure 6 This is a schematic diagram of the structure of the C2f block provided in the embodiment of this application. The C2f block mainly consists of n bottleneck blocks, two convolutional layers, and splitting and splicing operations. The splitting operation splits the output of convolutional layer 1 in the channel dimension, and outputs feature maps c1 and c2, which are used as inputs to bottleneck block 1 and splicing operation, respectively. The kernel size k of convolutional layer 1 is 1, the stride s is 1, and the padding p is 0. After passing through n bottleneck blocks, the splicing operation splices the outputs c1 and c2 of the splitting operation with the outputs of the n bottleneck blocks in the channel dimension, and then outputs them after passing through convolutional layer 2. The kernel size k of convolutional layer 2 is 1, the stride s is 1, and the padding p is 0.

[0052] Figure 7 This is a schematic diagram of the structure of the SPPF block provided in the embodiment of this application. The SPPF block consists of two convolutional layers, three two-dimensional max pooling layers, and a splicing operation. The splicing operation splices the output of convolutional layer 1 with the output of the three two-dimensional max pooling layers in the channel dimension. The kernel size k of convolutional layer 1 and convolutional layer 2 is 3, the stride s is 1, and the padding p is 1.

[0053] As can be seen from the above, the structure of the YOLOv8m network includes a large number of convolutional layers, C2f blocks, and cross-layer splicing operations, which bring the following challenges to inference:

[0054] ① Computationally intensive: Convolution operations account for more than 90%, requiring hardware with high parallel multiplication and accumulation capabilities;

[0055] ② Complex data dependencies: Residual joins and cross-layer splicing lead to complex data flow scheduling and frequent storage access;

[0056] ③ The activation function has strong nonlinearity: SILU activation function has high computational complexity, which is not conducive to efficient hardware implementation;

[0057] ④ Edge-side resource constraints: On FPGA (Field Programmable Gate Array) or edge AI chips, memory bandwidth, DSP resources (Digital Signal Processor resource), on-chip storage, etc. are all limited.

[0058] To address the aforementioned issues, some solutions employ a modular decoupling architecture. This involves unified data management through an on-chip memory array and task-level parallelism via an instruction dispatch module, thereby enhancing the accelerator's programmability and network adaptability. Its core features include:

[0059] ① There is no direct data path between modules; data exchange relies on on-chip storage.

[0060] ② Supports single-instruction multi-cycle synchronization mechanism, simplifying dependencies;

[0061] ③ The mapping of multiple network layers can be controlled through instruction sequences.

[0062] However, when processing YOLOv8m, the mismatch between data flow scheduling and network structure leads to low computing power utilization, high resource overhead of floating-point computing units, and frequent data transfer between layers. In particular, the three-channel input of the first layer will result in insufficient utilization of parallel computing array and data loading bandwidth.

[0063] This application provides an inference accelerator for YOLOv8m networks in some embodiments, such as... Figure 8 As shown, it includes: a data loading module, a matrix calculation module, a vector calculation module, and a data write-back module.

[0064] The inference accelerator of this embodiment is described in detail below.

[0065] The data loading module includes two data loading units for parallel reading of input activation data and weight data. For example, it includes data loading unit 0 and data loading unit 1. These two units are physically independent and can simultaneously obtain different types of data from off-chip memory, thereby eliminating serial waiting for data reading.

[0066] The matrix calculation module includes multiple sets of parallel multiply-accumulate arrays. Each set of multiply-accumulate arrays is used to perform convolution operations in parallel based on the input activation data and the weight data, and output the convolution result.

[0067] The vector computation module includes a fixed-point operation unit, which performs fixed-point activation operation, sampling operation and pooling operation on the convolution result and outputs the processing result.

[0068] The data write-back module is used to write the processing results back to the memory.

[0069] In some embodiments, the inference accelerator further includes an instruction fetching unit and a synchronization module; the instruction fetching unit is connected to the data loading module, matrix calculation module, vector calculation module and data write-back module, and is used to distribute the read instruction stream to each module; the synchronization module is connected to the instruction fetching unit, data loading module, matrix calculation module, vector calculation module and data write-back module, and is used to control the start and stop of each module according to the instruction stream.

[0070] The entire operation of the accelerator is controlled by the instruction stream generated by the compiler, which compiles the forward inference process of the network into accelerator instructions. The accelerator processes the network layer by layer according to the instruction stream until the forward inference of the network is completed.

[0071] The processing scheme for each layer is as follows: ① Due to the limited on-chip cache space, the input is divided into blocks; ② For each block, the input activation, weights, residual bias and other parameters are read in sequence; ③ For each block, if convolution calculation is required, convolution is performed; ④ For each block, activation or pooling is performed; ⑤ For each block, the result is written back to the corresponding memory address.

[0072] The inference accelerator is configured to run according to an instruction stream generated by the compiler, the instruction stream being generated based on fixed-point quantization, operator fusion, and first-layer dimension reorganization of the YOLOv8m network. The fixed-point quantization converts floating-point operations in the network into fixed-point operations, the operator fusion merges consecutive network layers to reduce intermediate data write-back, and the first-layer dimension reorganization performs dimensional transformation on the first-layer input activation data to match the hardware parallelism.

[0073] Specifically, before actual deployment, the trained YOLOv8m network is analyzed and optimized using a compiler. The compiler reads the network structure and performs fixed-point quantization on each layer, converting the input activation data, weight data, and intermediate results into fixed-point numbers with specific widths. At the same time, the compiler identifies continuous layers in the network that can be fused. For example, it merges the activation layer after a convolutional layer with the subsequent sampling layer into a single compound operation, avoiding the need to write intermediate results back to the off-chip.

[0074] To address the issue of insufficient input channels in the first layer of the network, the compiler restructures the input image dimensions, rearranging the width and channel dimensions to match the number of data channels in the restructured image with the hardware parallelism. After these optimizations, the compiler generates a series of instructions that specify the operation type, data address, computation parameters, and synchronization relationships between modules for each module.

[0075] When the accelerator starts, it reads the instruction stream from the instruction cache and distributes it to each module in sequence. It coordinates the start and stop of each module according to the synchronization information in the instructions, thereby ensuring the correctness of the data stream.

[0076] Taking the processing of a convolutional layer as an example, the instruction first controls the two units of the data loading module (data loading unit 0 and data loading unit 1) to work in parallel. For example, data loading unit 0 reads the input activation data block from outside the chip, and data loading unit 1 reads the corresponding weight data block at the same time. The read data can be temporarily stored in the on-chip cache.

[0077] The matrix computation module retrieves this data from the on-chip cache, and its multiple multiply-accumulate arrays start convolution computation simultaneously. Each multiply-accumulate array is responsible for part of the computation of one output channel. Multiple arrays work in parallel to quickly generate convolution results.

[0078] The fixed-point arithmetic unit of the vector computation module first dequantizes the convolution result, adds a bias, and then performs ReLU activation based on the sign bit of the data. If the sign bit is 0, it is output directly; if it is 1, it is output as 0. If the current layer is followed by a sampling operation and has been fused by the compiler, the fixed-point arithmetic unit then performs nearest-neighbor sampling on the activation result to obtain the sampled data. Finally, this data undergoes quantization to convert it into a fixed-point number with a preset bit width (low bit width) required for the next layer, and is written back to off-chip memory by the data write-back module. If the instruction stream indicates that a pooling operation should be performed, the vector computation module performs max pooling or average pooling on the input data, also using fixed-point arithmetic.

[0079] Throughout the inference process, the four modules of data loading, matrix calculation, vector calculation, and data write-back can perform pipelined operations. That is, while a set of data is being processed by the matrix calculation module, the data loading module can simultaneously read the next set of data, thereby achieving parallelism in time.

[0080] The inference accelerator of this embodiment will be described in further detail below.

[0081] During the operation data reading process, the input activation data and weight data account for most of the data reading time. Therefore, if FPGA resources allow, an additional data loading unit is added to distribute the data loading tasks that originally needed to be completed by one data loading unit to two decoupled data loading units.

[0082] like Figure 9 As shown, located Figure 9 The upper part of the process is the calculation process before optimization, located in Figure 9 The following section describes the optimized computation process. Through a reasonable concurrency strategy, the two units can work in parallel to acquire weight data and input activation data respectively, thereby greatly reducing the time spent on data loading.

[0083] After receiving the input activation data and weight data from the parallel data loading unit, the matrix calculation module continues to process them. In the graph neural network taking YOLOv8m as an example, convolution operations account for the majority of the network's computation. Optimizing the matrix calculation module can improve the computational power of convolution, which is also the key to improving the accelerator's inference speed.

[0084] By combining the structure of the YOLOv8m network with the FPGA hardware resources saved through quantization, the 1×32×32×1 floating-point multiply-accumulate array is optimized into a 2×3×32×32×1 multiply-accumulate array for convolution calculation.

[0085] In some embodiments, the matrix calculation module includes two sets of digital signal processing arrays, the digital signal processing arrays include three sets of multiply-accumulate arrays, and the multiply-accumulate arrays include a first preset number of multiply-accumulate trees.

[0086] The multiply-add tree includes a first preset number of multipliers and a first preset number of adders, used to accumulate the product of a first preset number of input activation data and a first preset number of weight data, so as to perform convolution operations on a first preset number of input channels and a first preset number of output channels in parallel.

[0087] In this embodiment, the first preset quantity is 32, that is, each multiply-accumulate array consists of 32 multiply-accumulate trees. Each multiply-accumulate tree includes 32 multipliers and an adder tree containing 32 adders. Each multiply-accumulate tree calculates the product of 32 pairs of input activation data and weight data in parallel. Then, these products and on-chip intermediate results are fed into the adder tree for accumulation. According to the information in the instruction stream provided by the compiler, if the current multiply-accumulate operation only completes the multiply-accumulate operation of a part of the input channel, the obtained multiply-accumulate result will be used as an intermediate result to participate in the next multiply-accumulate operation; if the current multiply-accumulate operation completes all the multiply-accumulate operations of the channel, the final multiply-accumulate result of the channel is obtained.

[0088] like Figure 10As shown, each multiply-accumulate tree performs a multiply-accumulate operation of a0×b0+a1×b1+……+a31×b31+c. This structure enables each multiply-accumulate tree to process convolution calculations for 32 input channels simultaneously. A set of multiply-accumulate arrays contains 32 such multiply-accumulate trees. Therefore, convolution operations for 32 input channels and 32 output channels can be performed in parallel.

[0089] Figure 10 In the middle, a0, a1, …, a 31 This represents a set of input activation data (i.e. feature map data), with a total of 32 input channels. During convolution calculation, the data from these 32 input channels will be broadcast to all output channels, meaning they are shared by the convolution kernels of all output channels.

[0090] b 00 , b 01 , ..., b 310 This represents the weight data corresponding to one output channel, b. 00 To b 310 Together, these constitute the weight data for the first output channel (channel 0) spanning all 32 input channels. In actual computation, the complete weight dataset corresponding to the 96 output channels is: b 00 ~b 310 (Output channel 0), b 01 ~b 311 (Output channel 1), ..., b 095 ~b 3195 (Output channel 95), each set of weighted data and input activation a0~a 31 Perform multiplication and accumulation to generate the result of one output channel.

[0091] c0 represents the partial sum for one output channel, and o0 represents the convolution calculation result for one output channel.

[0092] P0, P1, and P2 represent the outputs of three DSP multipliers deployed in a digital signal processor using the DSP-Packing algorithm (Digital Signal Processor Packaging), where input a0 is simultaneously multiplied by weight b. 01 b 02 b 03 Multiplication operations are performed in different DSP multipliers within a digital signal processor, outputting multiplication results P0, P1, and P2 respectively. In this way, parallelism is achieved across 96 output channels using 32 sets of digital signal processing arrays. The LUT adders and DSP multipliers work together to form a complete programmable multiply-accumulate operation logic.

[0093] Here, w4a6 represents the input data format of the multiply-accumulate array: the weight data is quantized into a 4-bit integer, i.e., w4, the input activation data is quantized into a 6-bit integer, i.e., a6, 10 bits, 11 bits, 12 bits, 13 bits, 14 bits represent the bit width of the intermediate results of each stage in the multiply-accumulate calculation process, and 23 bits represent the final result in the multiply-accumulate calculation process.

[0094] In some embodiments, the two sets of digital signal processing arrays include a first processing array and a second processing array. The two sets of digital signal processing arrays achieve efficient hardware layout through a DSP-Packing algorithm. The first and second processing arrays have identical structures and both support simultaneous computation of multiple output channels through data packing. The DSP-Packing algorithm allows three sets of multiply-accumulate arrays to be placed within a single set of digital signal processing arrays. These three sets of multiply-accumulate arrays share the same input activation data but use different weight data, which correspond to different output channels.

[0095] With this layout, a single digital signal processing array can simultaneously compute convolution operations with 96 output channels and 32 input channels. The number of output channels in most convolutional layers of the YOLOv8m network is a multiple of 96, except for the first two convolutional layers and some layers in the first C2f block, which have 48 output channels. Therefore, for most convolutional layers except for these few, a single digital signal processing array can maintain full-load operation.

[0096] During convolution computation, the first processing array performs convolution operations on multiple output channels in parallel at the same spatial location of the input activation data, i.e., it processes the computation of multiple output channels simultaneously, with each output channel corresponding to a set of weight data. The second processing array performs convolution operations on different spatial locations on the same set of output channels of the input activation data, i.e., it processes data blocks at different spatial locations in the input activation data simultaneously. Through this parallel strategy, the two sets of digital signal processing arrays can work together, enabling the multiply-accumulate array in the matrix computation module to operate at full capacity in most convolutional layer computations.

[0097] To ensure the second processing array achieves the same utilization as the first, the block size of the input activation data in both the height and width directions is a multiple of 2. The compiler adjusts the block strategy of the input activation data when generating the instruction stream to ensure this condition is met. When the block size is a multiple of 2, the second processing array can fully utilize its computational resources in the spatial dimension, complementing the parallelism of the first processing array in the output channel dimension.

[0098] Figure 11 The schematic diagram of multiply-accumulate array convolution calculation provided in the embodiments of this application is as follows: Figure 11 The left side shows the unoptimized solution, which uses a single floating-point multiply-accumulate array with FP16 inputs and weights and FP32 outputs. This solution has limited computing power and consumes a lot of resources.

[0099] Figure 11 The right side of the image shows the optimized solution of this application. By reducing the data bit width through quantization, two processing arrays can be integrated under the same hardware resources. Each array contains three multiply-accumulate arrays: computation array 0, computation array 1, and computation array 2. The three multiply-accumulate arrays in the first processing array operate in parallel along the output channel dimension, simultaneously computing three different output channel groups (output block 0, output block 1, and output block 2), each group containing 32 channels, achieving parallel computation of 96 output channels. The second processing array is configured to operate in parallel along the spatial dimensions (height and width) of the input activation data, complementing the first processing array. When the number of output channels is not a multiple of 192, the second processing array can still maintain full-load operation through spatial parallelism, as long as the input block size is a multiple of 2. Overall computing power utilization is significantly improved, with over 90% of convolutional layers achieving full-load computation.

[0100] Where ih and iw represent the spatial dimensions of the input block in a single computation, namely the input height and the input width. For example, (ih, iw, 32) represents an input block with an input height of ih, an input width of iw, and 32 input channels.

[0101] kh and kw represent the spatial dimensions of the convolution kernel in the convolution weight block, namely the kernel height and kernel width. For example, (kh, kw, 32, 32) represents a weight block with a kernel height of kh, a kernel width of kw, and 32 output and input channels, respectively.

[0102] oh and ow represent the spatial dimensions of the output block generated after one calculation, namely the output height and output width. For example, (oh, ow, 32) represents an output block with an output height of oh, an output width of ow, and 32 output channels. Output block 0, output block 1, and output block 2 correspond to the results of three different output channel groups obtained by parallel calculations of the three computing arrays in the first processing array.

[0103] INT4, INT6, and INT23 are 4-bit, 6-bit, and 23-bit signed fixed-point integers, respectively. The input block is INT6, the weight block is INT4, and the convolution result is INT23.

[0104] If the current convolutional layer output channels are forty-eight, then only two of the three multiply-accumulate arrays in the first processing array are needed to complete the calculation, leaving the third multiply-accumulate array idle. This situation where a few layers cannot be fully utilized only exists in the first two convolutional layers and some layers of the first C2f block in the YOLOv8m network, and has a relatively small impact on the overall computing power utilization.

[0105] When optimizing the YOLOv8m network, the compiler performs fixed-point quantization, operator fusion, and first-layer dimension reorganization operations, and provides these optimizations to the accelerator as an instruction stream. Fixed-point quantization converts floating-point operations in the network into fixed-point operations, operator fusion merges consecutive network layers to reduce intermediate data write-back, and first-layer dimension reorganization transforms the dimensions of the first-layer input activation data to match the hardware parallelism.

[0106] After the convolution calculation is completed, the convolution result is sent to the fixed-point arithmetic unit of the vector calculation module for further processing. Fixed-point quantization involves specific bit width configurations of input activation data, weight data, convolution result, activation result, dequantization factor, and quantization factor.

[0107] In some embodiments, the fixed-point arithmetic unit is configured to perform fixed-point multiplication and addition operations on the convolution result and the dequantization factor to obtain a dequantization result, wherein the dequantization factor is used to restore the convolution result to a fixed-point decimal with a preset precision; it is also configured to perform a selection operation based on the sign bit of the dequantization result and output an activation result; wherein, when the sign bit is 0, the dequantization result is determined to be an activation result, and when the sign bit is 1, 0 is determined to be an activation result; and to perform fixed-point multiplication and addition operations on the activation result and the quantization factor to output a quantization result.

[0108] Wherein, the fixed-point decimal with preset precision is a high-precision fixed-point decimal, the inverse quantization factor and the quantization factor are parameters that are inverse operations of each other, the quantization factor is used to compress the activation result into a fixed-point integer with preset bit width, and the fixed-point integer with preset bit width is a low-bit-width fixed-point integer.

[0109] In some embodiments, the input activation data is quantized into a six-digit integer, the weight data is quantized into a four-digit integer, the convolution result is quantized into a twenty-three-digit fixed-point integer, the activation result is quantized into a twenty-five-digit fixed-point decimal, and the inverse quantization factor and the quantization factor are quantized into sixteen-digit fixed-point decimals.

[0110] When performing convolutional layer computation, the fixed-point arithmetic unit receives the convolution result from the matrix computation module. The convolution result is quantized into a 23-bit signed fixed-point integer, and the corresponding inverse quantization factor is read from the on-chip cache. Since the inverse quantization factors obtained are all <1... 2 -3Therefore, the dequantization factor can be represented as 0.000x_xxxx_xxxx_xxxx_xxx(b), where x is the valid data part. Only 16 bits of valid data are needed to fully represent the value of the dequantization factor, and the calculation also uses 16 bits of valid data. Finally, the decimal point is repositioned in the result.

[0111] like Figure 12 , Figure 13 As shown, Figure 13 25, 18, and 48 all represent the bit width of the data line, corresponding to 25 bits, 18 bits, and 48 bits.

[0112] The fixed-point arithmetic unit sends the 23-bit convolution result to the digital signal processing unit as input A. Before sending it, two sign bits are padded in the high-order bits to expand the data to 25 bits. The 16-bit dequantization factor is sent to the digital signal processing unit as input B. Before sending it, two zero bits are padded in the high-order bits to expand it to 18 bits.

[0113] The input D of the digital signal processing unit (DSP) is set to all zeros. The DSP performs the operation P = (A + D) × B + C, and outputs a 48-bit signed fixed-point multiply-accumulate result after one clock cycle. Since the DSP supports high-bit-width retention, this multiply-accumulate operation retains all bits, and the output result contains 20 integer bits and 19 fractional bits, with an actual effective number of bits of 39. This step is called dequantization, which restores the low-bit-width convolution result to a high-precision fixed-point fractional number so that the subsequent bias addition and activation operations do not lose precision.

[0114] After dequantization, the fixed-point arithmetic unit reads the bias data from the cache. The bias is a 16-bit signed fixed-point fractional, including 5 integer bits and 11 fractional bits. To add the bias data with the dequantized result in fractional alignment, the bit width needs to be adjusted. Since the dequantized output should contain 19 fractional bits, but the bias only has 11 fractional bits, 8 zeros need to be padded to the lower bits of the bias to achieve fractional alignment.

[0115] Simultaneously, to match the input bit width of the digital signal processing unit (DSP), 24 sign bits need to be padded in the high-order bits, extending the bias to 48 bits before serving as the DSP's input C. The fixed-point arithmetic unit adds the dequantization result to the bias, obtaining a 48-bit result. This result is then rounded, with the lower 25 bits used as the output, resulting in a 25-bit signed fixed-point fractional number, containing 6 integer bits and 19 fractional bits. The rounding operation adjusts the data bit width to a range acceptable to subsequent processing units while retaining sufficient precision.

[0116] Next, the fixed-point arithmetic unit performs the activation operation. In this embodiment, the activation function is ReLU, which selects the activation based on the sign bit of the data. The fixed-point arithmetic unit checks the sign bit (the most significant bit) of the aforementioned 25-bit signed fixed-point fractional. If the sign bit is 0, it indicates that the value is ≥ 0, and the data is directly output as the activation result. If the sign bit is 1, it indicates that the value is < 0, and all zeros are output as the activation result. After ReLU activation, the output activation result is a 25-bit unsigned fixed-point fractional, including 6 integer bits and 19 fractional bits. This implementation avoids the complex floating-point exponentiation required by the traditional SILU activation function, and can be completed by only using a single sign bit, simplifying the hardware design.

[0117] If residual connections exist in the current convolutional layer, the fixed-point arithmetic unit (FLU) also needs to perform a residual addition operation. The residual data is a 6-bit unsigned fixed-point integer, and the corresponding residual dequantization factor is a 16-bit unsigned fixed-point fractional, including zero integer bits and 16 fractional bits. The FLU first sends the residual data to the digital signal processing unit (DSP) as input A, sends the residual dequantization factor as input B, and uses the 25-bit activation result obtained from ReLU activation as input C. At this time, input D is set to all zeros. The DSP executes the operation P=(A+D)×B+C, directly obtaining the sum of the activation result and the dequantized residual within one clock cycle, outputting a 48-bit result. The sum is then rounded to a 25-bit unsigned fixed-point fractional, which is used as the final activation result output. These 25 bits also include 6 integer bits and 19 fractional bits.

[0118] After activation, the fixed-point arithmetic unit needs to quantize the activation result to generate the input data required for the next layer of the network. The fixed-point arithmetic unit reads the quantization factor from the buffer. The quantization factor is a 16-bit unsigned fixed-point fractional number, including 6 integer bits and 10 fractional bits. The fixed-point arithmetic unit sends the 25-bit activation result to the digital signal processing unit as input A, fills it with the 16-bit quantization factor and uses it as input B, and sets input C to all zeros.

[0119] The digital signal processing unit (DSP) performs multiplication and addition operations, outputting a 48-bit result, which includes 19 integer bits and 29 fractional bits. The fixed-point arithmetic unit (FPA) then rounds this result, converting it into a 6-bit unsigned integer. The rounding process uses a rounding-to-the-end method for the fractional part and performs a carry-over check on the integer part. Based on statistical analysis of the data distribution of each layer of the YOLOv8m network, the quantized activation result, after layer-by-layer calibration, typically falls within the valid range of a 6-bit unsigned integer, rarely exceeding 63 after a carry-over. To ensure the integrity of the hardware circuitry, the FPA incorporates an overflow protection mechanism. If the carry-over value is greater than 63, the FPA constrains the output value to 63, the maximum value that a 6-bit unsigned integer can represent. If the carry-over value is less than or equal to 63, the value is directly output. The final output 6-bit unsigned integer becomes the input activation data for the next layer of the network.

[0120] Throughout the entire calculation process described above, except for the sign bit determination during ReLU activation and the rounding constraints during final quantization, all intermediate calculations fully retain decimal precision without introducing additional errors, ensuring that the precision loss during fixed-point quantization is controlled within an acceptable range.

[0121] Building upon fixed-point quantization, the compiler further optimizes the data flow through operator fusion. In some embodiments, the operator fusion includes activation sampling operator fusion and convolutional layer fusion. First-layer dimension reorganization transforms the format of the first-layer input activation data, and activation sampling operator fusion is used to process the combination of activation operations after convolutional layers and subsequent sampling operations in the YOLOv8m network.

[0122] like Figure 14 In the upper part of the process shown, without fusion, the data flow path of the convolutional layer is as follows: the data loading module moves the segmented data to the input buffer, weight buffer, and bias buffer; the matrix calculation module reads data from the buffer, performs convolution calculation, and updates the convolution result to the intermediate result buffer; the vector calculation module reads data from the intermediate result buffer, performs activation operation, and updates the activation result to the output buffer; the data write-back module writes the activation result in the output buffer back to memory; then the sampling layer begins execution; the data loading module reads the activation result from memory as the input of the sampling layer and moves it to the input buffer; the vector calculation module reads data from the input buffer, performs sampling operation, and updates the sampling result to the output buffer; the data write-back module writes the sampling result in the output buffer back to memory. In this process, the activation result needs to be transferred between memory and buffer once, increasing the data transfer volume.

[0123] like Figure 14In the following process, after the compiler implements the activation sampling operator fusion, the generated data flow path instruction stream changes. The data loading module still moves the block-wise data to the input buffer, weight buffer, and bias buffer. The matrix calculation module reads the required parameters, performs convolution calculation, and updates the convolution result to the intermediate result buffer. The accelerator is configured to: control the vector calculation module to read the activation result from the on-chip buffer as the input of the sampling operation after completing the activation operation, and control the data write-back module to write the processing result back to memory after the sampling operation is completed.

[0124] The vector computation module reads data from the intermediate result buffer and performs activation operations. However, the activation result is not written back to the output buffer at this point. Instead, based on the address definition in the instruction, the activation result is updated to the input buffer. The vector computation module then reads the newly written activation result from the input buffer, performs sampling operations, updates the output buffer with the sampling result, and the data write-back module writes the sampling result from the output buffer back to memory. Throughout the entire process, the activation result does not leave the on-chip buffer; it flows directly from the activation operation to the sampling operation, reducing the number of memory accesses.

[0125] When implementing activation sampling operator fusion, the compiler makes three modifications to the instruction generation logic. First, it modifies the data end address of the convolutional layer activation instruction, replacing the original address pointing to the output buffer with an address pointing to the input buffer, ensuring the activation result is stored in the input buffer, not the output buffer. Second, it inserts a sampling instruction between each generated activation instruction and write-back instruction in the convolutional layer. The starting address of the sampling instruction is set to the address where the activation result is stored in the input buffer, and the ending address is set to the output buffer address. Third, it adjusts the concurrency flow to ensure that activation and sampling operations can be completed consecutively within the processing cycle of the same data block, avoiding waiting due to improper module scheduling.

[0126] Convolutional layer fusion is used to process combinations of multiple consecutive convolutional layers in the YOLOv8m network. There are multiple consecutive convolutional layer sequences in the YOLOv8m network. For example, convolutional layer 1, convolutional layer 2 and the first convolutional layer in the first C2f block constitute a three-layer consecutive sequence.

[0127] In some embodiments, the accelerator is configured such that: the data loading module moves the segmented data to the input buffer, weight buffer, and bias buffer required by the first convolutional layer; and through the instruction stream, controls the matrix calculation module and the vector calculation module to sequentially perform convolution and activation operations on multiple consecutive convolutional layers.

[0128] like Figure 15As shown, specifically, the matrix calculation module reads the parameters and performs the convolution operation of the first convolutional layer, updating the convolution result to the intermediate result cache. The vector calculation module reads the data from the intermediate result cache, performs the activation operation, and updates the activation result to the input cache. At this time, the write-back operation is not performed, but the data in the input cache is used as the input of the second convolutional layer.

[0129] The matrix calculation module reads the weights and biases of the second convolutional layer, performs convolution operations on the data in the input buffer, and updates the intermediate result buffer again. The vector calculation module then performs activation operations again and updates the result to the input buffer. This process is repeated until the last convolutional layer of the fusion is completed, i.e., the number of convolutional layers (count) is n. The activation result of the last convolutional layer is updated to the output buffer, and then written back to memory by the data write-back module.

[0130] Throughout the process, the input data is loaded by the data loading module only before the first convolutional layer, and the processing result is written back to the memory by the data write-back module after the last convolutional layer. The input and output data of the intermediate convolutional layers are all processed in the on-chip cache without accessing off-chip memory.

[0131] When implementing convolutional layer fusion, the compiler needs to consider the choice of the number of fusion layers. The number of consecutive convolutional layers in the YOLOv8m network cannot be fused indefinitely. The weight cache capacity is limited. The weights of a single convolutional layer can be fully stored in the cache, but if the number of fusion layers is too large, the sum of the weights of all layers will exceed the cache capacity, and the weights will have to be loaded repeatedly, which will increase the amount of data transmission.

[0132] Convolutional computation involves scale transformation. For example, a convolutional layer with a stride of 2 will change the feature map size, resulting in a mismatch in the block size of different layers. The existence of residual connections also complicates fusion because residual paths require the input and output sizes to be the same. Considering these factors, the compiler chooses to limit the number of fusion layers to three and divides the continuous convolutional layers in the YOLOv8m network into six groups of fusion layers. Through these six groups of fusion, the original eighteen convolutional layers that needed to be processed separately are merged into six fusion layers, significantly reducing the amount of data transmission.

[0133] In addition to operator fusion, the compiler also performs dimensional reorganization on the first-layer input activation data to match the hardware parallelism. In some embodiments, the first-layer dimensional reorganization includes: edge padding of the first-layer input activation data to obtain padded input activation data; dividing the width dimension of the padded input activation data into blocks to obtain four-dimensional block input activation data with a second preset number of width blocks; transposing the four-dimensional block input activation data, placing the second preset number of width block dimensions before the channel dimensions to obtain transposed input activation data; and fusing the second preset number of width block dimensions and the channel dimensions in the transposed input activation data to obtain fused input activation data, wherein the number of channels in the fused input activation data is equal to the input parallelism of the matrix calculation module.

[0134] like Figure 16 As shown, the specific process of first-layer dimension reorganization includes five steps. The first step is to fill the edges of the input image. The data dimension of the input image is (h, w, 3), where h is the image height, w is the image width, and 3 represents the number of channels c is 3. Since the kernel size of the first convolutional layer and subsequent convolutional layers is 3, the stride is 2, and the padding is 1, it is necessary to increase the height and width of the input image by three pixels each, that is, to fill from h rows and w columns to h + three rows and w + three columns, while keeping the number of channels unchanged. At this time, the data dimension is (h+3, w+3, c=3).

[0135] The second step is to split the width dimension of the filled image into ten blocks. The width of each block is the original width divided by 10 plus 3. There is an overlapping area between adjacent blocks to ensure that edge pixels can also be convolved correctly. After splitting, the dimensions of the data become height plus 3, 10 blocks, width of each block, and three channels. At this time, the data dimension is (h+3, 10, w_block_size, c=3), and the width of each block is w_block_size = (w / 10) + 3.

[0136] The third step is to transpose the data, moving the dimension of 10 blocks before the channel dimension, resulting in a height plus 3, the width of each block, the 10 blocks, and the arrangement of the 3 channels. The fourth step is to merge the last two dimensions, that is, to combine the 10 blocks and the 3 channels into 30 channels, resulting in a height plus 3, the width of each block, and 30 channels. At this point, the data dimension is (h+3, w_block_size, c=30).

[0137] The fifth step involves edge padding again, adding two pixels to the channel dimension to bring the number of channels to 32, perfectly matching the input parallelism of the matrix calculation module. After these five steps, the effective number of channels in the first-layer input activation data increases from 3 to 30, and after padding, it reaches 32, enabling subsequent data loading and matrix calculations to be performed at full bandwidth. At this point, the data dimension is (h+3, w_block_size=(w / 10)+3, c=32).

[0138] When the data after the first-level dimension reorganization is fed into the accelerator, the hardware decoding logic needs to be adjusted accordingly. In normal mode, the 32 channels of the input data are directly stored in the on-chip cache. In the first-level dimension fusion mode, the 32 channels of the input data are divided into 10 groups, with each group consisting of 3 valid channels and 2 filler channels to form a 32-channel data block. These 10 data blocks are stored in different areas of the on-chip cache for the matrix calculation module to read as needed in subsequent calculations. The entire dimension reorganization process is pre-completed by the compiler at the software level and does not occupy the accelerator's online inference time.

[0139] In summary, the inference accelerator provided in this embodiment, based on the structural characteristics of the YOLOv8m network, has been designed and optimized for these structural characteristics in terms of network, hardware, and compilation tools. It not only retains the decoupling of the existing accelerator's computation module and data access module, thus possessing high programmability, but also has the following advantages:

[0140] ① Significantly improved computing power and energy efficiency: Through fixed-point quantization, hardware structure optimization and operator fusion, the effective computing power and effective data loading rate of the YOLOv8m network inference are increased by nearly 6 times under similar resource consumption.

[0141] ② High hardware utilization: The computing array and data loading strategy designed for the characteristics of YOLOv8m structure enable the full load rate of the convolutional layer computing array to exceed 90%, and the parallelism of data loading is significantly improved;

[0142] ③ Reduce deployment complexity: By integrating layers and operators, reduce intermediate data movement and storage access, and simplify address planning and instruction scheduling;

[0143] ④ Edge-friendly: Full fixed-point computation and lightweight activation function replacement significantly reduce hardware resource consumption and power consumption, making it more suitable for edge deployment scenarios using FPGA.

[0144] Based on the aforementioned inference accelerator for the YOLOv8m network, some embodiments of this application also provide an inference method for the YOLOv8m network, applied to the inference accelerator. The inference accelerator includes a data loading module, a matrix calculation module, a vector calculation module, and a data write-back module. The inference accelerator runs according to the instruction stream generated by the compiler. The instruction stream is generated based on fixed-point quantization, operator fusion, and first-layer dimension reorganization of the YOLOv8m network. Fixed-point quantization converts floating-point operations in the network into fixed-point operations. Operator fusion fuses consecutive network layers to reduce intermediate data write-back. First-layer dimension reorganization transforms the dimensions of the first-layer input activation data to match the hardware parallelism.

[0145] The method includes:

[0146] The input activation data and weight data are read in parallel by two data loading units in the data loading module.

[0147] The matrix calculation module uses multiple parallel multiply-accumulate arrays to perform convolution operations in parallel based on the input activation data and the weight data, and outputs the convolution result.

[0148] The fixed-point operation unit in the vector calculation module performs fixed-point activation, sampling and pooling operations on the convolution result and outputs the processing result.

[0149] The processing result is written back to the memory via the data write-back module.

[0150] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.

Claims

1. An inference accelerator for YOLOv8m networks, characterized in that, include: The data loading module includes two data loading units for reading input activation data and weight data in parallel; The matrix calculation module includes multiple sets of parallel multiply-accumulate arrays. Each set of multiply-accumulate arrays is used to perform convolution operations in parallel based on the input activation data and the weight data, and output the convolution result. The vector computation module includes a fixed-point operation unit, which is used to perform fixed-point activation operation, sampling operation and pooling operation on the convolution result and output the processing result. The data write-back module is used to write the processing results back to the memory; The inference accelerator is configured to run according to an instruction stream generated by the compiler, the instruction stream being generated based on fixed-point quantization, operator fusion, and first-layer dimension reorganization of the YOLOv8m network, the fixed-point quantization converting floating-point operations in the network into fixed-point operations, the operator fusion fusing consecutive network layers to reduce intermediate data write-back, and the first-layer dimension reorganization performing dimensional transformation on the first-layer input activation data to match the hardware parallelism. The fixed-point arithmetic unit is further configured to perform fixed-point multiplication and addition operations on the convolution result and the dequantization factor to obtain the dequantization result, and add the dequantization result to the bias to obtain the result with the bias, and the dequantization factor is used to restore the convolution result to a fixed-point decimal with a preset precision. When performing the fixed-point multiply-add operation, the internal digital signal processing unit generates and retains the intermediate operation results with a full bit width; The fixed-point arithmetic unit is further configured to: perform a selection operation based on the sign bit of the dequantization result and output an activation result; wherein, when the sign bit is 0, the dequantization result is determined to be an activation result, and when the sign bit is 1, 0 is determined to be an activation result; and, if there are residual connections in the current convolutional layer, perform a residual addition operation, add the dequantized residual to the activation result, output the addition result, perform a fixed-point multiplication and addition operation on the addition result and the quantization factor, and output the quantization result; Wherein, the inverse quantization factor and the quantization factor are parameters that are inverse operations of each other. The quantization factor is used to compress the activation result into a fixed-point integer with a preset bit width, where the preset bit width represents low precision. The activation function of the activation operation is ReLU. The fixed-point quantization is specifically configured as follows: The input activation data is quantized into a six-digit integer, the weight data into a four-digit integer, the residual data into a six-digit unsigned fixed-point integer, the convolution result into a twenty-three-digit fixed-point integer, and the activation result into a twenty-five-digit fixed-point decimal, the twenty-five-digit fixed-point decimal including 6 integer digits and 19 decimal digits; and the bias, inverse quantization factor, and quantization factor are quantized into sixteen-digit fixed-point decimals, wherein the bias is a 16-digit signed fixed-point decimal, including 5 integer digits and 11 decimal digits; the inverse quantization factor of the convolution result includes 16 decimal digits, and the quantization factor of the activation result includes 6 integer digits and 10 decimal digits.

2. The inference accelerator for YOLOv8m networks according to claim 1, characterized in that, The matrix calculation module includes two sets of digital signal processing arrays, each set of digital signal processing arrays includes three sets of multiply-accumulate arrays, and each multiply-accumulate array includes a first preset number of multiply-accumulate trees. The multiply-add tree includes a first preset number of multipliers and a first preset number of adders, used to accumulate the product of a first preset number of input activation data and a first preset number of weight data, so as to perform convolution operations on a first preset number of input channels and a first preset number of output channels in parallel.

3. The inference accelerator for YOLOv8m networks according to claim 2, characterized in that, The two sets of digital signal processing arrays include a first processing array and a second processing array. The first processing array and the second processing array have the same structure and both support the simultaneous calculation of multiple output channels through data packaging. The first processing array is used to perform convolution operations on multiple output channels in parallel at the same spatial location of the input activation data, and the second processing array is used to perform convolution operations on different spatial locations in parallel on the same set of output channels of the input activation data.

4. The inference accelerator for YOLOv8m networks according to claim 2 or 3, characterized in that, The operator fusion includes convolutional layer fusion; The inference accelerator is configured to control the matrix calculation module and the vector calculation module through the instruction stream to sequentially perform convolution and activation operations on multiple consecutive convolutional layers; Specifically, before the first convolutional layer, the input data is loaded by the data loading module, and after the last convolutional layer, the processing result is written back to the memory by the data write-back module.

5. The inference accelerator for YOLOv8m networks according to claim 1, characterized in that, The operator fusion includes activation sampling operator fusion; The accelerator is configured to: control the vector calculation module to read the activation result from the on-chip cache as the input of the sampling operation after completing the activation operation, via the instruction stream; And control the data write-back module to write the processing result back to the memory after the sampling operation is completed.

6. The inference accelerator for YOLOv8m networks according to claim 1, characterized in that, The first-layer dimension reorganization includes: edge padding of the input activation data of the first layer to obtain padded input activation data; The width dimension of the filled input activation data is divided into blocks to obtain four-dimensional block input activation data with a second preset number of width blocks; The input activation data after the four-dimensional block is transposed by placing the second preset number of width block dimensions before the channel dimensions to obtain the transposed input activation data. The second preset quantity width block dimension and channel dimension in the transposed input activation data are fused to obtain fused input activation data. The number of channels in the fused input activation data is equal to the input parallelism of the matrix calculation module.

7. The inference accelerator for YOLOv8m networks according to claim 1, characterized in that, It also includes an instruction fetch unit and a synchronization module; The instruction fetching unit is connected to the data loading module, matrix calculation module, vector calculation module and data write-back module, and is used to distribute the read instruction stream; The synchronization module is connected to the instruction fetching unit, data loading module, matrix calculation module, vector calculation module, and data write-back module, and is used to control the start and stop of the module according to the instruction flow.

8. An inference method for YOLOv8m networks, applied to an inference accelerator, characterized in that, The inference accelerator includes a data loading module, a matrix calculation module, a vector calculation module, and a data write-back module. The inference accelerator runs according to the instruction stream generated by the compiler. The instruction stream is generated based on the YOLOv8m network after performing fixed-point quantization, operator fusion, and first-layer dimension reorganization. The fixed-point quantization converts floating-point operations in the network into fixed-point operations. The operator fusion merges consecutive network layers to reduce intermediate data write-back. The first-layer dimension reorganization transforms the dimensions of the first-layer input activation data to match the hardware parallelism. The method includes: The input activation data and weight data are read in parallel by two data loading units in the data loading module. The matrix calculation module uses multiple parallel multiply-accumulate arrays to perform convolution operations in parallel based on the input activation data and the weight data, and outputs the convolution result. The fixed-point operation unit in the vector calculation module performs fixed-point activation, sampling and pooling operations on the convolution result and outputs the processing result. The processing result is written back to the memory via the data write-back module. The fixed-point arithmetic unit performs fixed-point multiplication and addition operations on the convolution result and the dequantization factor to obtain the dequantization result. The dequantization result is added to the bias to obtain the result with the bias added. The dequantization factor is used to restore the convolution result to a fixed-point decimal with a preset precision. When performing the fixed-point multiplication and addition operation, the internal digital signal processing unit is controlled to generate and retain intermediate operation results with a full bit width. The fixed-point arithmetic unit is further configured to: perform a selection operation based on the sign bit of the dequantization result and output an activation result; wherein, when the sign bit is 0, the dequantization result is determined to be an activation result, and when the sign bit is 1, 0 is determined to be an activation result; and, if there are residual connections in the current convolutional layer, perform a residual addition operation, add the dequantized residual to the activation result, output the addition result, and perform a fixed-point multiplication and addition operation on the activation result and the quantization factor to output the quantization result; Wherein, the inverse quantization factor and the quantization factor are parameters that are inverse operations of each other. The quantization factor is used to compress the activation result into a fixed-point integer with a preset bit width, where the preset bit width represents low precision. The activation function of the activation operation is ReLU. The fixed-point quantization is specifically configured as follows: quantizing the input activation data into a six-bit integer, quantizing the weight data into a four-bit integer, quantizing the residual data into a six-bit unsigned fixed-point integer, quantizing the convolution result into a twenty-three-bit fixed-point integer, and quantizing the activation result into a twenty-five-bit fixed-point decimal, wherein the twenty-five-bit fixed-point decimal includes six integer bits and 19 decimal bits; and quantizing the bias, inverse quantization factor, and quantization factor into sixteen-bit fixed-point decimals, wherein the bias is a 16-bit signed fixed-point decimal, including five integer bits and 11 decimal bits; the inverse quantization factor of the convolution result includes 16 decimal bits, and the quantization factor of the activation result includes six integer bits and 10 decimal bits.

Images