A hardware accelerator architecture suitable for deep neural network models

Abstract

This invention provides a hardware accelerator architecture suitable for deep neural network models, comprising: a CPU and an FPGA wirelessly connected to each other; the CPU is used for normalizing and adaptively adjusting the size of the feature maps input to the neural network model, processing the inference results of the network model to determine the final calculation result, and logically controlling the core computing unit of the FPGA acceleration module; the FPGA is used for accelerated computation and data transmission of data from different network layers; the FPGA includes: a data loading module, a write-back module, and a computation module; the data loading module is used to load data from off-chip into an on-chip buffer, the computation module is used to perform calculations based on the data in the on-chip buffer to obtain the calculation result, and the write-back module is used to write the calculation result back to off-chip DDR. This invention solves the problems of insufficient acceleration performance and high power consumption of existing hardware accelerators for deep neural network models.

Description

Technical Field

[0001] This invention relates to the field of neural network hardware design technology, and in particular to a hardware accelerator architecture suitable for deep neural network models. Background Technology

[0002] In recent years, with the rapid development of big data and hardware computing power, deep learning-based neural network models have gradually attracted the attention of researchers worldwide and have been applied in various fields. Their superior performance has shone brightly, gradually creating a significant impact on traditional fields. Especially in the field of computer vision, various fundamental neural network models have emerged, from the initial LeNet and AlexNet to Vgg16, then to ResNet, ShuffleNet, and MobileNet, and finally to VIT (VisionTransformer). With the deepening and optimization of network structures, the performance of these network models has been greatly improved.

[0003] Taking object detection as an example, in the field of computer vision, object detection is one of the most fundamental and challenging problems, and convolutional neural network (CNN) models are the most important and extensively studied part of it. Among these, the YOLO series of algorithms has gained widespread favor from researchers and industry due to its relatively low model complexity and high model inference performance. The YOLO algorithm fundamentally solves the real-time requirement in object detection by unifying classification and localization problems, greatly reducing the detection complexity of the network model. In subsequent iterations, the YOLO algorithm has been continuously optimized, for example, by adding routing operations and removing fully connected layers, thus overcoming the limitation of image size during detection; adding a feature pyramid structure to achieve multi-scale feature fusion, further improving the network model's detection performance for small targets; in addition, introducing residual structures to the model improves its ability to extract abstract features, while also further reducing the training difficulty. After multiple iterations, the current YOLO series of algorithm models can fully demonstrate its "artificial intelligence" capabilities, with extremely high recognition and localization accuracy, thus better serving human production and life in various fields.

[0004] However, despite their high detection accuracy, these deep neural network models suffer from complex structures and numerous layers. For example, the smallest branch of YOLOv5, YOLOv5s, has over 200 layers, with more than 60 of them being the most computationally complex convolutional layers. This increased layer depth and complexity pose significant challenges to practical deployment and application. Relying solely on CPU computing power for inference tasks results in substantial inference latency, failing to meet the computational demands of current high-performance neural network models. Furthermore, while GPUs are widely used for training and inference in various AI models, and their streaming cores are well-suited to the computational characteristics of different network models, thus meeting the inference computational requirements of deep neural network models to some extent, they also introduce extremely high power consumption, making them unsuitable for some mobile applications.

[0005] For current deep neural network models, the main computational modules include convolution, pooling, activation, fully connected layers, softmax, shortcuts, and concat. The YOLOv5 network model primarily uses convolution, pooling, activation, shortcuts, and concat. Among these modules, convolution, fully connected layer computation, and activation operations involve the highest computational cost. Therefore, the focus of hardware acceleration architectures for neural networks on FPGAs is on the parallelization and dataflow optimization of these computational modules. The remaining computational modules in the network structure, due to their lower computational complexity, do not require overly complex designs.

[0006] Currently, most related research in China still focuses on accelerating relatively shallow and simple neural network models. The implemented neural network accelerators are mainly designed for VGG16 and YOLOv2. In recent years, accelerator designs for slightly deeper models such as YOLOv3 and v4 have begun to emerge, but the design ideas are similar to previous designs. The optimization directions of various accelerators mainly focus on model quantization and compression research, simplification of key calculations (e.g., using matrix multiplication to replace convolution multiplication and addition operations), and optimization design of some neural network accelerator architectures (systolic arrays, parallelization design, inter-layer pipelines, intra-layer pipelines, etc.).

[0007] The proposed accelerator architectures have focused more on optimizing certain hardware computing modules, neglecting the massive amount of data required for model computation. Due to the sheer volume of data and the limitations of the FPGA's storage resources, not all the data needed for the entire model computation process can be stored on-chip. A block-based strategy is required to load and write back data in batches, leading to frequent data interaction between on-chip and off-chip resources. In actual computation, the computational latency of the computing resources is relatively low, but the processes of moving data from off-chip to on-chip or from on-chip to off-chip have extremely high latency and energy consumption, thus affecting actual computational efficiency. Furthermore, due to the computational characteristics of some network layers (such as padding operations), the block-based strategy frequently interrupts overall data transmission, affecting the efficiency of burst transmissions. Simultaneously, the continuous transmission of the same redundant data during computation increases the accelerator's power consumption.

[0008] Although these neural network accelerators are designed for specific neural networks, they still lack fine-grained optimization analysis and general design for some internal computational details. As a result, the actual acceleration performance of the accelerators is still insufficient, and the power consumption during operation has not been optimized. Summary of the Invention

[0009] To overcome the shortcomings of the prior art, the purpose of this invention is to provide a hardware accelerator architecture suitable for deep neural network models. This invention solves the problems of insufficient acceleration performance and high power consumption of hardware accelerators for deep neural network models in the prior art.

[0010] To achieve the above objectives, the present invention provides the following solution:

[0011] A hardware accelerator architecture suitable for deep neural network models includes:

[0012] CPUs and FPGAs that are wirelessly connected to each other;

[0013] The CPU is used to normalize and adaptively adjust the size of the feature maps input to the neural network model, process the inference results of the network model to determine the final calculation results, and perform logic control on the FPGA acceleration core computing unit. The FPGA is used to accelerate the calculation and data transmission of data from different network layers.

[0014] The FPGA includes:

[0015] Data loading module, write-back module, and calculation module;

[0016] The data loading module is used to load data from off-chip into the on-chip buffer, the calculation module is used to perform calculations based on the data in the on-chip buffer to obtain calculation results, and the write-back module is used to write the calculation results back to the off-chip DDR.

[0017] Preferably, the FPGA further includes:

[0018] The Shortcut and Concat modules are interconnected;

[0019] The Shortcut module is used to implement the shortcut operator function, and the Concat module is used to implement the concat operator function.

[0020] Preferably, the Shortcut module and the Concat module are a streaming processing architecture.

[0021] Preferably, the data loading module includes:

[0022] The module includes an input feature map loading submodule, a weight loading submodule, and a bias value loading submodule.

[0023] The feature map loading submodule is used to input feature maps into the on-chip BRAM, the reloading submodule is used to input weights into the on-chip BRAM, and the bias value loading submodule is used to input biases into the on-chip BRAM.

[0024] Preferably, the calculation module includes:

[0025] Convolution calculation submodule, pooling calculation submodule, activation calculation submodule, Upsample calculation submodule;

[0026] The convolution calculation submodule is used to perform convolution calculations, the pooling calculation submodule is used to perform pooling calculations, the activation calculation submodule is used to perform Silu activation operations and Sigmoid activation operations, and the upsample calculation submodule is used to perform upsample calculation operations.

[0027] Preferably, the computing module further includes:

[0028] The padding_check submodule and the reindex submodule;

[0029] The padding_check submodule is used to perform padding checks on the data during the calculation process, and the reindex submodule is used to perform indexing and locating operations on the data during the calculation process.

[0030] Preferably, the communication between the FPGA, CPU, and DDR is the AXI-BUS protocol.

[0031] Preferably, the convolution calculation submodule includes:

[0032] Enable inter-block multiplexing units and multiplexing register units;

[0033] The inter-block reuse unit is used to repeat the calculation operations of the convolution calculation submodule, and the reuse register unit is used to store the results calculated by the inter-block reuse unit.

[0034] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0035] This invention provides a hardware accelerator architecture suitable for deep neural network models, which consists of a CPU and an FPGA that are wirelessly connected to each other.

[0036] The CPU is used for normalizing and adaptively adjusting the size of feature maps input to the neural network model, processing the inference results of the network model to determine the final calculation result, and logically controlling the FPGA-accelerated core computing unit. The FPGA is used for accelerated computation and data transmission of data from different network layers. The FPGA includes a data loading module, a write-back module, and a computation module. The data loading module loads data from off-chip into an on-chip buffer. The computation module performs calculations based on the data in the on-chip buffer to obtain the calculation result. The write-back module writes the calculation result back to off-chip DDR. This invention optimizes the on-chip and off-chip communication transmission modes, optimizes its burst transmission mode, and simultaneously optimizes the data reuse strategy between on-chip memory and computing units, reducing access to redundant off-chip data, reducing the number of interactions between on-chip and off-chip storage units, and reducing power consumption. Based on this, the design of each key computing unit is further optimized from the perspective of data flow optimization, thereby improving the computational throughput of the hardware acceleration architecture. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 An overall architecture diagram of the accelerator provided in an embodiment of the present invention;

[0039] Figure 2 This is a diagram of the CAMU-IPcore architecture provided in an embodiment of the present invention;

[0040] Figure 3This is a schematic diagram of the convolution calculation submodule structure provided in an embodiment of the present invention;

[0041] Figure 4 A diagram of the convolutional PE architecture provided in an embodiment of the present invention;

[0042] Figure 5 This is a schematic diagram of the activation calculation submodule structure provided in an embodiment of the present invention;

[0043] Figure 6 This is a schematic diagram of the pooling computation submodule structure provided in an embodiment of the present invention;

[0044] Figure 7 This is a schematic diagram of the Upsample calculation module structure provided in an embodiment of the present invention;

[0045] Figure 8 The diagram shows the SC-IPcore architecture provided in this embodiment of the invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] The purpose of this invention is to provide a hardware accelerator architecture suitable for deep neural network models. This invention solves the problems of insufficient acceleration performance and high power consumption of existing hardware accelerators for deep neural network models.

[0048] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] like Figure 1 As shown, this invention provides a hardware accelerator architecture suitable for deep neural network models, comprising:

[0050] CPUs and FPGAs that are wirelessly connected to each other;

[0051] The CPU is used to normalize and adaptively adjust the size of the feature maps input to the neural network model, process the inference results of the network model to determine the final calculation results, and perform logic control on the FPGA acceleration core computing unit. The FPGA is used to accelerate the calculation and data transmission of data from different network layers.

[0052] The FPGA includes:

[0053] Data loading module, write-back module, and calculation module;

[0054] The data loading module is used to load data from off-chip into the on-chip buffer, the calculation module is used to perform calculations based on the data in the on-chip buffer to obtain calculation results, and the write-back module is used to write the calculation results back to the off-chip DDR.

[0055] Furthermore, the FPGA also includes:

[0056] The Shortcut and Concat modules are interconnected;

[0057] The Shortcut module is used to implement the shortcut operator function, and the Concat module is used to implement the concat operator function.

[0058] Furthermore, the Shortcut and Concat modules are based on a streaming processing architecture.

[0059] Specifically, this neural network hardware accelerator adopts a CPU+FPGA heterogeneous acceleration architecture. The CPU part is mainly responsible for the pre-processing and post-processing of the neural network model, as well as the logic control of the FPGA acceleration core computing unit. The FPGA part is a hardware acceleration module, which consists of two hardware IP cores, CAMU-IPcore and SC-IPcore, which are used to implement different network layer operators.

[0060] The former mainly includes a data loading module, a write-back module, and a computation module, while the latter mainly adopts a streaming processing architecture, which mainly includes a Shortcut module and a Concat module.

[0061] Furthermore, the data loading module includes:

[0062] The module includes an input feature map loading submodule, a weight loading submodule, and a bias value loading submodule; the computation module includes:

[0063] The system includes a convolution calculation submodule, a pooling calculation submodule, an activation calculation submodule, and an upsample calculation submodule. The convolution calculation submodule is used to perform convolution calculations, the pooling calculation submodule is used to perform pooling calculations, the activation calculation submodule is used to perform Silu activation operations and Sigmoid activation operations, and the upsample calculation submodule is used to perform upsample calculation operations.

[0064] The feature map loading submodule is used to input feature maps into the on-chip BRAM, the reloading submodule is used to input weights into the on-chip BRAM, and the bias value loading submodule is used to input biases into the on-chip BRAM.

[0065] For CAMU-IPcore, its data loading module mainly consists of three parts: an input feature map loading submodule, a weight loading submodule, and a bias value loading submodule, which are responsible for the efficient transmission of input feature maps, weights, and bias values ​​to the on-chip, respectively. The output feature map write-back module shares the data transmission channel with the data loading module and adopts a multi-channel data transmission mode. After loading the input feature map and related model parameters into the on-chip BRAM, the data flow is allocated according to the specific operator of the network layer. If it is a convolutional layer, the convolution calculation submodule is enabled, and an output feature map reuse strategy is adopted to repeatedly load input feature maps of the same position but different input channels and pass them to the convolution calculation submodule until all output feature maps of that part are calculated before they are all written back off-chip. In the convolution calculation module, the calculation is completed according to the block strategy of the input feature map, and a three-layer parallel calculation strategy is adopted, namely the feature map data width, input channels, and output channels. Simultaneously, a data reuse strategy between blocks is applied, storing the computation results of reusable data between blocks for reuse iteration in the next block, ultimately completing all convolution computations for that block and transmitting them to the activation computation submodule. Since the padding operation was removed from the data loading module, a padding_check module and a reindex module were added to the computation module. These modules are responsible for padding checks and indexing during the computation process, respectively, to ensure the correctness of the computation results. For pooling operations, the pooling computation submodule is enabled. Pooling operations employ two parallel computation strategies: feature map width and output channel. The computation results are directly transmitted to the activation computation module. If the computation of a network layer requires activation operations, the activation computation module is enabled to perform the corresponding activation operations. These activation operations mainly consist of two parts: Silu activation and Sigmoid activation, both sharing the same nonlinear operator. To reduce computational performance, this module is pipelined. The final calculation result is transmitted to the output feature map write-back module, and then the write-back module is enabled to write the final output feature map result back to the external DDR. If it is an Upsample operation, it is directly transmitted to the corresponding processing module, and after the calculation is completed, it is directly transmitted to the external DDR.

[0066] The SC-IPcore is primarily responsible for Shortcut and Concat computations. Since these two computation modules are relatively simple and do not require random access, they directly employ a streaming processing architecture. Data transmission is directly controlled by DMA. Due to the streaming burst transmission strategy, a large number of dedicated on-chip registers are unnecessary; instead, a FIFO is used to complete the entire data loading and writing process. Both computation modules are invoked via the enable signal of the top-level IP core. If the current computation network layer is the Shortcut layer, the Shortcut computation module is enabled. To improve computational performance, parallel computation is designed, simultaneously transmitting data of corresponding parallelism to the FIFO, and then transmitting the completed computation to the output FIFO. The Concat module is similar to the Shortcut module, sharing its streaming transmission channel; only the intermediate computation process differs.

[0067] Specifically, data preprocessing mainly includes normalizing the input feature maps and adaptively adjusting their size, which is primarily related to the network model. Post-processing mainly deals with the inference results of the network model (the final output feature maps) to determine the final computation result. Both IP cores process feature maps of the network model, which are actually tensors, and their specific size is determined by the network model configuration. The off-chip DDR is mainly responsible for storing the feature map data and related model parameters (model weights and biases) during the computation process. Communication between the FPGA, CPU, and DDR is primarily accomplished using the AXI-BUS protocol. The transmission of control signals between the FPGA and CPU is handled by S_AXILite, while the transfer of feature map data and model weights is accomplished via burst transmission through the M_AXI port. In the CAMU-IPcore, since some input feature maps require repeated addressing and loading, the M_AXI protocol is used directly; while the SC-IPcore does not require re-addressing or repeated loading, so the AXIS protocol is used for burst transmission in data stream form. Furthermore, to further improve transmission efficiency, DMA is used to control the entire transmission process. The acceleration of network inference tasks by hardware accelerators mainly relies on the control signals of the top-level CPU to repeatedly call various hardware acceleration modules on the FPGA.

[0068] from Figure 1 As can be seen from the diagram, the CAMU-IPcore in this invention mainly comprises six parts: a data loading module, a convolution calculation module, a pooling calculation module, an activation calculation module, an upsample calculation module, and an output feature map write-back module. Its overall structure is shown in the diagram below. Figure 2As shown. Both the loading module and the output feature map write-back module employ multi-channel parallel burst transmission, with each transmission channel operating independently and without interference. Each M_AXI transmission port is assigned its own AXI adapter to control the transmission transactions on that port. Since the AXI protocol specifies that read and write channels within a port are independent, the data loading module and the output feature map write-back module share the M_AXI transmission port.

[0069] The overall structure diagram of the data loading module and the output feature map write-back module is as follows: Figure 2 As shown, the data loading module comprises three parts: an input feature map loading module, a model weight loading module, and a model bias parameter loading module. These three modules execute independently but in parallel during actual accelerated computation.

[0070] Furthermore, the computing module also includes:

[0071] The padding_check submodule and the reindex submodule;

[0072] The padding_check submodule is used to perform padding checks on the data during the calculation process, and the reindex submodule is used to perform indexing and locating operations on the data during the calculation process.

[0073] Specifically, in the input feature map loading module, each input feature map transmission port corresponds to a block of input channels. Padding operations are not considered during transmission; instead, they are integrated with the convolution and pooling computation modules. Furthermore, inter-block reuse strategies are implemented in each computation module. Therefore, transmission interruptions caused by padding operations and data re-addressing are eliminated throughout the transmission process. This also avoids the additional overhead of transmitting redundant data after each block. See the convolution and pooling computation modules for details. Thus, each transmission port can adopt incremental transmission mode, requiring only one addressing in each transmission transaction, significantly improving transmission efficiency. In addition, to maximize bandwidth utilization and further improve transmission efficiency, the data bit width is extended to 512 bits, greatly adapting to changes in parameter bit width and network layer parameters.

[0074] The model's weight data is only used during convolution calculations. Therefore, in the weight loading module, due to the block-based strategy, it's unnecessary to transmit the entire model on-chip for storage; only the data to be calculated needs to be pre-loaded into the on-chip buffer. During actual transmission, the amount of data transmitted each time is directly determined by the block parameters and the convolution kernel size. Considering that each network layer's output channels are multiples of 32, the weight data is rearranged, arranging the weights corresponding to the output channels together to expand the transmission bit width. For convolutional layers with fewer than 32 output channels, the weights are set to 0. Since the weight rearrangement is done off-chip and is a preprocessing step, this padding process does not affect weight transmission.

[0075] The model's bias parameters are only used during convolution calculations, and the size of the output channels after segmentation directly determines the amount of data transmitted during transmission. Since the number of output channels is generally small, the weight loading module and the bias parameter loading module share the data transmission channels, and access is mutually exclusive during transmission. The specific access scheduling is handled by the corresponding M_AXI adapter.

[0076] The output feature map write-back module uses the same transmission optimization strategy as the input feature map module to move feature map data. The difference is that the number of parallel transmission channels for data write-back is limited by the number of ports, while the number of output channels is generally determined by the top-level configuration TM. Therefore, based on multi-channel parallel transmission, multiple re-addressing transmissions are required. However, since the total number is small and each transmission transaction can maximize the use of transmission bandwidth, the impact of re-addressing is minimal.

[0077] To ensure the continuity and efficiency of the entire data loading, computation, and write-back process, the input feature map loading module, weight loading module, and output feature map write-back module all employ a ping-pong double-buffer optimization strategy to minimize the latency caused by data loading, computation, and write-back, achieving quasi-parallel execution of the three modules. The three computation modules within the IP core share the data loading and write-back modules, and also share on-chip buffer storage units to maximize storage resource utilization.

[0078] Furthermore, the convolution calculation submodule includes:

[0079] Enable inter-block multiplexing units and multiplexing register units;

[0080] The inter-block reuse unit is used to repeat the calculation operations of the convolution calculation submodule, and the reuse register unit is used to store the results calculated by the inter-block reuse unit.

[0081] Specifically, Figure 3The diagram shows the convolutional computation module, which is the most computationally intensive and time-consuming part of most neural network models. Acceleration of this module primarily relies on parallel design, data flow optimization, and pipeline design. In this invention, from a top-level structural perspective, its parallelism design is mainly achieved through multiple Processing Engines (PEs) within the main computation module. Parallel computation is mainly divided into two aspects: PE-level parallelism and PE-internal parallelism. The overall parallel computation unfolds along three dimensions: output channels (TM), feature map size dimension (PC), and input channels (TN). PE-level parallelism corresponds to the organic fusion of the first two parallel methods (TM*PC), while PE-internal parallelism corresponds to the unfolded computation of input channels (TN). In the convolutional computation module, there are a total of TM*PC PEs. TM represents the number of output channels in blocks, and PC represents the parallelism within the feature map size dimension. Within the PE, parallel design is mainly focused on the input channels TN, simultaneously computing TN data points with the same index but different input channels.

[0082] Figure 4 This is a diagram of the internal structure of the PE (Programmer) in the convolution computation module. To ensure the correctness of the calculation, the convolution computation operation and the padding operation are integrated within the convolution PE by adding a `padding_check` module and a `reindex` module. The former is mainly responsible for checking the padding of the data to be computed, while the latter's main function is to determine the indices in the current input buffer and output buffer. If padding is required, the loading of data in the buffer is disabled; if padding is not required, data is loaded from the on-chip buffer into each parallel PE based on the indices in the input and output buffers determined in the `reindex` module to complete the convolution multiplication and addition computation. In the PE of this invention, the multiplication and addition computation adopts a multiplication and addition separation strategy. After TN parallel multiplications are completed, the result is directly passed into the addition tree and then accumulated with the temporary result at the corresponding position in the temporary storage buffer. At the same time, the model weights and bias parameters under this block configuration are loaded. In addition, the computational structure is optimized at the dataflow level. The entire dataflow computation process involved in the convolution computation is broken down into fine-grained components, including the padding_check module, reindex module, multiplication unit, addition tree unit, and temporary result accumulation and write-back unit. By analyzing and decomposing the entire computational dataflow, a pipelined design is implemented. Furthermore, the outer loops (feature map size dimension and convolution kernel size dimension) are flattened to increase the overall pipeline length, reduce iteration intervals during the iterative computation process, and ensure that a computation result is obtained in each clock cycle. This improves the throughput of the convolution computation module while increasing hardware utilization.

[0083] Finally, inter-block data reuse is additionally designed into the entire convolutional computation module. Since the convolutional kernel size is greater than 1, after processing the entire input feature map into blocks, a certain amount of feature map data always needs to be repeatedly loaded. To reduce the transmission of this redundant data, inter-block data reuse is designed. To avoid affecting the pipeline in the previous data flow optimization process, this data reuse is implemented by extending the pipeline. The specific implementation strategy is as follows: A dedicated register module is added to the convolutional computation module. Its capacity is related to the convolutional kernel size K and the block size TC, and its capacity can be expressed as (K-1)*TC. After completing the required results for the corresponding output feature map in this iteration, the inter-block reuse module is enabled, and the previous computation process continues, using the same computational data flow. The only difference is that the computation results are stored in the inter-block reuse register module. When the next block is computed, the temporary result is calculated together with the current result according to the reuse strategy and then written back to the current reuse location.

[0084] The above describes the computation structure of a single feature data block after segmentation. For the entire convolutional layer, this convolutional computation module needs to be called multiple times, cooperating with the data loading and writing-back modules, until all output feature map results are obtained. In this invention, to increase data reusability, an output feature map reuse method is adopted. That is, before obtaining the complete segmented output feature map, the convolutional computation module does not write the intermediate output feature map results back to the off-chip DDR, but keeps them in the on-chip output buffer. The next data loaded into the on-chip input buffer is the input feature map at the same position in the input channel of the next segment. After the input feature map of this segment is loaded, it is passed to the convolutional computation module and accumulated with the previous calculation result. This process is repeated until the output feature map of the corresponding segment is completely calculated, and then it is written back to the off-chip. Then the calculation continues until the entire output feature map is completely calculated. In this process, the corresponding input feature map needs to be repeatedly loaded to ensure the correctness of the output feature map calculation under the segmentation strategy.

[0085] Figure 5The diagram shows the activation calculation module, which mainly includes the implementation of two activation functions: the Sigmoid function and the SiLU function. These two activation functions are related; the SiLU function uses the Sigmoid operator during calculation. Therefore, this invention directly integrates the two activation functions. During the calculation of the SiLU function result, the result of the Sigmoid function is directly called. If the activation operator of this layer is the Sigmoid function, its result is multiplied by 1 and then output; if it is the SiLU activation function, its result is multiplied by the activation input and then output. By adding a multiplier, the two activation operators are integrated, reusing the Sigmoid operator calculation module and maximizing the utilization of on-board hardware computing resources. Simultaneously, through data flow optimization, the hardware computing resources in the entire calculation module are always in a computing state, improving the module's computing performance. To further improve computing efficiency, a parallel design is added to this operator, with multiple activation operator modules computing in parallel. The specific parallelism design depends on the on-board computing resources; the optimal parallelism corresponds to the PE parallelism design in the convolution module.

[0086] Figure 6 The diagram shows the pooling computation module, which implements max pooling, performing pooling calculations according to the input pooling window size. This invention employs two parallel computing modes: output channel parallelism (TM) and feature map size parallelism (PC). Both parallel computing designs are implemented using PE-level parallelism, with the PE primarily handling simple comparison operations. The total parallelism can be represented as TM*PC. Since the pooling and convolution computation modules share data input and output modules, and pooling computation typically requires padding of the input feature map, a padding_check module and a reindex module are added to the pooling computation module, integrating them with the pooling PE. Their working principle is the same as in the convolution computation module: after processing by the padding_check and reindex modules, the corresponding input feature map data is obtained and then transmitted to the pooling comparator. The final result is then written back to the on-chip output buffer. The structure of the pooling computation module and the specific structure of its internal PE are shown in the figure. In this multidimensional parallel computing mode, fine-grained parallel data flow optimization is performed. The computations involved in the entire pooling computation module are pipelined, and the overall computational load is based on the output feature map size and pooling kernel size after block division. This completes the fine-grained pipeline scheduling, ensuring that each cycle yields a corresponding number of computation results (equal to the parallelism design). This extends the overall pipeline length and significantly improves the utilization of hardware computing resources.

[0087] Figure 7The Upsample module primarily implements upsampling operations. In this invention, only the "near" mode upsampling calculation strategy is implemented. The loading of the input feature map and the writing back of the output feature map in this module use the same calling logic as the convolution and pooling calculation modules, sharing their on-chip buffers. The loading and writing back processes will not be elaborated upon here. Its core calculation module also adopts a multi-dimensional parallel mode, namely, output channel parallelism (TM) and output feature map size dimension parallelism (PC), i.e., TM*PC. The specific implementation strategy is shown in the figure. Due to the computational characteristics of upsampling, the outer layer has a total of PEs of TM*PC / 2, and each PE can obtain 2 results simultaneously. Based on this parallel result, data flow optimization is performed, decomposing each basic operation and implementing a pipelined design to ensure that the corresponding parallelism calculation result can be obtained in each clock cycle.

[0088] Figure 8 The diagram shows the SC-IPcore architecture, which mainly consists of two parts: the Shortcut computation module and the Concat module. Neither module reuses data; therefore, the entire computation process can be completed directly using a streaming processing strategy. There's no need to add an additional data loading and writing-back module. Instead, input and output FIFOs are set up using AXIS ports to ensure the continuity of data loading, computation, and writing-back, while also guaranteeing efficient data transmission. The data processing of the key modules is described below.

[0089] In the Shortcut computation module, its main responsibility is to fuse the features calculated by the previous convolution module with the features calculated by the previous layer. The computation only involves adders, which are all implemented directly using LUTs. Parallel computation is combined with burst transmission to increase the data transmission bit width. In pipelined burst transmission mode, it is ensured that the corresponding amount of data can be loaded into the on-chip FIFO in each clock cycle. Then, the FIFO provides computation data to the accelerator's parallel computing unit for computation. The computation results are first temporarily stored in the output FIFO, and then written back to the off-chip via a write-back request. The write-back process is also similar. Since the transmission ports use streaming transmission, the two input channels and the output channel are mutually exclusive, using a total of 3 AXIS transmission ports.

[0090] The Concat module primarily performs copying and moving of key feature map data stored in off-chip DRAM. Currently, residual structures in the network architecture can generally be handled by pre-setting addresses using a hardware-software collaborative approach. However, for some specialized splicing operations, data migration is necessary. The Concat module also employs a streaming computing architecture with two AXIS ports: one for loading and the other for writing back. These two ports operate independently and with mutually exclusive access. To ensure efficient and continuous transmission, an intermediate FIFO buffer is added, and the entire data migration process is completed under DMA control. This eliminates the need for CPU-based mapping and migration to virtual memory, significantly reducing the accelerator's overall dependence on the CPU.

[0091] The beneficial effects of this invention are as follows:

[0092] 1. This invention achieves maximum acceleration of the entire neural network algorithm's mapping in hardware by implementing various operators in the neural network model in hardware. Furthermore, it employs a combined hardware and software design, enabling dynamic data flow configuration during computation based on network layer parameters, thereby improving computational performance.

[0093] 2. Based on dynamic data stream configuration, this invention optimizes the burst transmission mode during on-chip and off-chip interaction, further improving transmission efficiency and maximizing the utilization of transmission bandwidth.

[0094] 3. In convolution and pooling operations in neural network algorithms, there are some padding operations, which can affect the performance and efficiency of burst transmission to a certain extent. This invention optimizes these padding operations by integrating the calculation process of operators in the CAMU-IPcore with this operation and adding an inter-block reuse strategy. Without affecting the calculation efficiency of operators, it avoids the repeated loading of invalid off-chip data, eliminates the impact of padding operations on transmission efficiency, and reduces the power consumption of the accelerator during operation to a certain extent.

[0095] 4. Based on the computational characteristics of the CAMU-IPcore, this invention further designs an inter-block data reuse strategy, which improves the efficiency of data utilization.

[0096] 5. This invention applies parallel optimization design in three dimensions in the CAMU-IPcore, which can provide higher computing power support when computing resources are sufficient.

[0097] 6. The hardware IP core in this invention has high module reusability. It is mainly designed for specific operators, while taking into account the variability of neural network models. Therefore, a certain degree of universality is considered in the hardware design of operators. Only some top-level calling parameters need to be changed to accelerate other models.

[0098] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0099] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A hardware accelerator architecture suitable for use with a deep neural network model, the architecture comprising: The application relates to a neural network model acceleration system. The CPU is used for normalization processing and size self-adaptive adjustment of feature maps of input of a neural network model, processing of inference results of the network model, determination of final calculation results and logical control of an FPGA acceleration core calculation unit, and the FPGA is used for accelerated calculation and data transmission of different network layer data. The FPGA comprises a data loading module, a write-back module and a calculation module. The data loading module is used for loading data from outside a chip to a buffer on the chip, the calculation module is used for calculating a calculation result according to data in the buffer on the chip, and the write-back module is used for writing the calculation result back to a DDR outside the chip. The calculation module further comprises a padding_check sub-module and a reindex sub-module. The padding_check sub-module is used for padding check operation of data in a calculation process, and the reindex sub-module is used for index positioning operation of data in the calculation process. The FPGA further comprises a Shortcut module and a Concat module connected with each other. The Shortcut module is used for shortcut operator function, and the Concat module is used for concat operator function. The Shortcut module and the Concat module are stream processing architectures. 2.The hardware accelerator architecture suitable for deep neural network model according to claim 1, wherein, The data loading module comprises an input feature map loading sub-module, a weight loading sub-module and a bias value loading sub-module. The feature map loading sub-module is used for inputting feature maps to BRAM on a chip, the weight loading sub-module is used for inputting weights to BRAM on the chip, and the bias value loading sub-module is used for inputting bias to BRAM on the chip. The calculation module comprises a convolution calculation sub-module, a pooling calculation sub-module, an activation calculation sub-module and an Upsample calculation sub-module.

3. The hardware accelerator architecture suitable for a deep neural network model according to claim 2, wherein, The convolution calculation sub-module is used for convolution calculation, the pooling calculation sub-module is used for pooling calculation, the activation calculation sub-module is used for Silu activation operation and Sigmoid activation operation, and the Upsample calculation sub-module is used for Upsample calculation operation. 4.The hardware accelerator architecture suitable for deep neural network model of claim 1, wherein, Communication between the FPGA, the CPU and the DDR is AXI-BUS protocol. The convolution calculation sub-module comprises an inter-block multiplexing unit and a multiplexing register unit. The inter-block multiplexing unit is used for repeated convolution calculation sub-module calculation operation, and the multiplexing register unit is used for storing results calculated by the inter-block multiplexing unit.

5. The hardware accelerator architecture suitable for deep neural network model according to claim 1, wherein, ​ ​ ​ 6. The hardware accelerator architecture suitable for a deep neural network model according to claim 1, wherein, ​ 7. The hardware accelerator architecture suitable for a deep neural network model according to claim 5, wherein, ​ ​ ​

Images