Convolutional neural network acceleration system with configurable computing structure based on Systolic array

Abstract

The invention belongs to the technical field of semiconductor integrated circuit technology and deep learning hardware acceleration, and relates to a computing structure configurable convolutional neural network acceleration system based on a Systolic array, which comprises a control processor and a coprocessor, the control processor is connected with the coprocessor through a high-speed expansion interface and a low-speed signal interface; the control processor is used for sending self-defined computing structure configuration data and image data to the coprocessor through the high-speed expansion interface; the control processor is further configured to send a layer indication signal, a convolution mode signal, and a computational mode signal to the coprocessor over the low speed signal interface. According to the method, parallel calculation of a plurality of basic calculation paths can be completed at the same time, calculation tasks under different input channel dimensions and different output channel dimensions in the convolution calculation process can be completed after calculation structure configuration, and the method has the advantages of being high in calculation throughput and low in energy consumption ratio and is very suitable for the high calculation power requirement of the convolutional neural network.

Description

Technical Field

[0001] This invention belongs to the fields of semiconductor integrated circuit technology and deep learning hardware acceleration technology, and more specifically, relates to a hardware acceleration technology for convolutional neural networks based on semiconductor integrated circuits. Background Technology

[0002] Convolutional Neural Networks (CNNs), as one of the most representative algorithms in deep learning, are widely used in the field of computer vision. In edge devices such as drones and intelligent surveillance systems, computational tasks have stringent requirements for real-time performance and power consumption control. To reduce latency introduced by remote data transmission, trained CNN models need to be deployed to local devices for real-time data processing. However, due to limitations in energy consumption, size, and other physical factors, the low-power chips used in edge devices have significant constraints in terms of computing power and storage resources. They are typically used as coprocessors or customized CNN model circuits to improve inference computation efficiency.

[0003] In the field of convolutional neural network acceleration, research directions can be divided into two categories: algorithm-level optimization and hardware-level optimization. Mainstream algorithm-level optimization strategies can be summarized as reducing computational complexity and reducing actual computational load. Early work focused on introducing classical numerical algorithms into neural networks to improve computational performance. Michael et al. pioneered the introduction of the Fast Fourier Transform (FFT) into convolutional computation, significantly improving training and inference efficiency on GPU platforms. Besides frequency domain methods, the Winograd minimum filtering algorithm has also become a research hotspot for convolution acceleration in recent years. The Winograd algorithm was first used for FIR (Finite Impulse Response) filter optimization. Lavin et al. extended it to deep convolutions, significantly reducing the number of multiplication operations, especially in scenarios with small convolutional kernels. Building on this, Lin et al., by analyzing the redundancy between neurons, proposed a strategy of reusing similar computational results in real-time during inference to further reduce redundant computation.

[0004] Hardware-level optimization methods often combine with algorithmic optimization methods for acceleration. Abtahi et al. proposed a structured optimization method to address the memory consumption problem of FFT on embedded platforms, enabling the deployment of FFT convolutions, which involve a large amount of floating-point computation, on resource-constrained devices. Wu et al. reused similar redundant computational results in the Winograd algorithm to design the DREU model, achieving significant inference acceleration on embedded platforms. To address timing and latency control issues in deep learning models, the Systolic array structure is widely used in accelerator design. This structure achieves efficient parallel computing and pipelined control through regularly arranged processing elements (PEs). Google's Tensor Processing Unit (TPU), developed based on this, significantly improves the performance of matrix operations in neural networks, achieving orders-of-magnitude speedups compared to traditional CPU and GPU platforms.

[0005] When accelerating CNN inference on edge devices in specific application scenarios, there are usually strict requirements for power consumption control and real-time response. Hardware accelerator designs can be broadly divided into two categories: custom and general-purpose. Custom accelerators are designed for specific CNN models, employing high parallelism and pipeline optimization, and typically achieve excellent computational performance. General-purpose accelerators, on the other hand, utilize programmable processor soft cores to flexibly schedule computation and data flow based on instructions, possessing stronger model adaptability and suitable for application scenarios that support acceleration of different CNN models. However, with the continuous evolution of CNN model structures, custom acceleration circuits designed for specific CNN models lack flexibility; while general-purpose acceleration circuits implemented based on instructions increase design complexity and introduce additional instruction processing and data transmission latency, limiting the overall performance of the acceleration system. Summary of the Invention

[0006] To address the aforementioned shortcomings or improvement needs of existing technologies, a configurable convolutional neural network acceleration system based on a systolic array is provided. The aim is to ensure applicability to accelerating CNN models with different layer structures, while integrating multiple algorithm optimization methods and fully leveraging the advantages of low latency, high computational performance, and low power consumption of hardware acceleration. This solves the technical problems of existing convolutional neural network hardware acceleration products that only target single network acceleration, do not consider the use of algorithm optimization methods, and have insufficient hardware optimization.

[0007] To achieve the above objectives, according to one aspect of the present invention, a configurable convolutional neural network acceleration system based on a Systolic array is provided, comprising: a control processor and a coprocessor, wherein the control processor and the coprocessor are connected via a high-speed expansion interface and a low-speed signal interface. The coprocessor internally includes configurable units and configurable data flow scheduling methods. The configurable units include registers, task RAM, and selector MUX designed based on a dual-buffer mechanism. The configurable units are used to store convolution parameters of different convolutional layers and different task information, as well as to determine the data source of the basic computing units. The configurable data flow scheduling methods include a controller and task RAM designed based on a message queue, which are used to guide the flow of different data flows between modules. During system operation, the control processor is used to send custom computing structure configuration data and image data to the coprocessor through a high-speed expansion interface; The control processor is also used to send layer indication signals, convolution mode signals, and computation mode signals to the coprocessor via a low-speed signal interface; wherein, the layer indication signal indicates the layer information including the input layer, intermediate layer, and output layer; the convolution mode signal includes depth-parallel convolution mode and Winograd convolution mode; and the computation mode signal includes iterative mode and non-iterative mode. The coprocessor is used to send status signals to the control processor via a low-speed signal interface; the status signals include configurable status, configuration ready status, computation in progress status, and output status.

[0008] Furthermore, the coprocessor is used to handle the computational tasks involved in the input layer, intermediate layer, and output layer of different convolutional neural network models; When the system processes the input layer computation task indicated by the layer indication signal, the control processor sends computation structure configuration data and image data to the coprocessor. The coprocessor automatically starts the input layer convolution computation task based on the currently received convolution mode signal and computation mode signal, and sends status signals to the control processor through a low-speed signal structure. The coprocessor is also used to cache intermediate data generated during the computation based on the current layer indication signal. The intermediate data includes the output feature data generated by the input layer and intermediate layer convolution computation. When the system processes the input layer and intermediate layer computing tasks indicated by the layer indicator signals, the control processor is also used to obtain status signals from the coprocessor through the low-speed signal interface; when the configurable status signal transmitted by the coprocessor is set, the control processor sends the next layer computing structure configuration data and computing mode signal to the coprocessor through the high-speed expansion interface; thereafter, the next layer computing task is automatically executed. When the system processes the output layer indicated by the layer indicator signal, the coprocessor is also used to send the output data to the control processor through a high-speed extension interface; wherein, the output data includes the output feature map generated by the convolution calculation of the output layer.

[0009] Furthermore, it also includes two methods to accelerate convolutional layers with different convolutional parameters: depth-parallel convolution and Winograd convolution; where convolutional parameters include kernel size, weight parameters, and stride parameters. The deep parallel convolution mode is based on decomposing the original convolution kernel into several 1*1*1 convolution kernels, and performing convolution calculation and calculation merging of the above 1*1*1 convolution kernels through the computation structure configuration information. Winograd convolution mode is based on converting convolution computation into matrix multiplication, and performs its computation process through a computational structure built using computational structure configuration information.

[0010] Furthermore, the coprocessor includes a control and interface module, a system configuration module, a layer input processing module, a high-performance computing module, and a layer output processing module; The control and interface module is connected to the control processor through a high-speed expansion interface and a low-speed signal interface. The control and interface module is used to transmit data with the control processor to receive input computational structure configuration data, image data, layer indicator signals, convolution mode signals and computation mode signals. Based on the receiving progress of computational structure configuration data and image data, the module controls the generation of corresponding enable signals and status signals in combination with the layer indicator signals. The enable signals are then transmitted to the corresponding modules in the system, and the status signals are output to the control processor through the low-speed signal interface. The system configuration module is used to parse the computing structure configuration data received by the control and interface module from the control processor, and distribute the parsed computing structure configuration data to the corresponding modules through the configuration data flow bus; The layer input processing module is used to read the input feature data of the current layer from the external memory according to the layer input task determined by the computational structure configuration data. Then, it processes the input feature data based on the convolution mode signal received by the control and interface module and sends it to the high-performance computing module through the internal data flow bus. The high-performance computing module receives input feature data from the layer input processing module, performs convolution operations on the computing structure constructed based on the computing structure configuration data, and sends the computation output data obtained from the convolution calculation to the layer output processing module through the internal data flow bus. The layer output processing module receives computation output data from the high-performance computing module, processes the input feature data based on the computation mode signal received by the control and interface module, and then stores the output data to the external memory through the external data stream bus according to the layer output task determined by the computation structure configuration data.

[0011] Furthermore, the layer input processing module includes a Winograd input conversion module, a layer input task RAM, a layer input task controller, a layer input data buffer FIFO, and a computation input task controller; The layer input task RAM is used to store the layer input task information saved in the computational structure configuration information; the layer input task information includes the layer input task number, read start address, read offset, and computational input task index; The layer input task controller is used to poll the layer input tasks. When a layer input task is executed, the layer input task controller reads the corresponding layer input task information from the layer input task RAM, and then reads the input feature data from the external memory through the external data flow bus based on the read start address and read offset. The Winograd input transformation module is used to perform input transformation operations in the Winograd fast algorithm and cache them to the layer input data cache FIFO; The computation input task controller is used to send the data in the layer input data buffer FIFO and the computation input task index read from the layer input task RAM to the high-performance computing module.

[0012] Furthermore, the high-performance computing module includes a computing input control module, a configurable Systolic array, an array control module, and a computing output processing module; The computation input control module is used to receive computation input data sent from the layer input processing module, and synchronously send the computation input data to the reconfigurable Systolic array for computation based on the computation input task index; Configurable Systolic arrays are used to perform convolution calculations on computational input data based on the weight and bias parameters in the computational structure configuration data. The array control module is used to send column control signals to the configurable Systolic array group based on the step size parameter in the computational structure configuration data; The computation output processing module receives the output data from the Systolic array group after convolution calculation, processes and buffers it, and sends it to the layer output processing module through the internal data flow bus.

[0013] Furthermore, the PE unit of the configurable Systolic array group includes a C_PE processing unit and an O_PE processing unit. Both the C_PE processing unit and the O_PE processing unit have a delay mechanism, a double buffer mechanism, and a control signal pulsation mechanism. The configurable Systolic array group is used to complete computational tasks under different convolution modes, including depth-parallel convolution computation and Winograd convolution computation. Each column of the configurable Systolic array group has a basic computational path, which is formed by a combination of several C_PE processing units arranged in succession and an O_PE processing unit. Among them, the C_PE and O_PE have the same structure. The C_PE unit is only used for computation, and the O_PE unit is used for computation and outputting the computation result to the output processing module. When the calculation bit width is 8 bits, the processing unit can perform two multiply-add operations in one clock cycle. When the calculation bit width is 16 bits, the processing unit can perform one multiply-add operation in one clock cycle. The sign and bit width conversion module in the PE unit is used to perform the corresponding sign and bit width conversion during 8-bit bit width operations to ensure the correctness of the calculation results. The delay mechanism of the PE unit is used to delay the computation input data received in the current clock cycle for several cycles before computation. This mechanism is used to adjust the position of the weight parameters in the sparse convolution kernel in the Systolic array. The dual-buffering mechanism of the PE unit is used to synchronously receive the weight parameters and bias parameters from the configuration data of the next layer of computing structure during the computation process; The control signal pulsation mechanism of the PE unit is used to pass the control signal from top to bottom to the immediately adjacent PE unit below in each clock cycle, and to serve as the calculation output enable signal of the O_PE unit.

[0014] Furthermore, the array control module generates corresponding column control signals based on the step size parameters in the computational structure configuration data; The column control signal is used to control the calculation process of all PE cells in the corresponding column of the Systolic array based on the control signal pulsation mechanism of the PE cell, so as to skip the corresponding calculation operation when the step size parameter is not 1.

[0015] Furthermore, the calculation input control module includes a line input FIFO, a calculation input controller, a calculation input data buffer, and a line input delay module; The row input FIFO is used to buffer the row input data of the Systolic array; The computation input data buffer is used to cache computation input data output from the layer input processing module; The computation input controller is used to forward the computation input data cached in the computation input data buffer to the corresponding row input FIFO according to the computation input task index from the layer input processing module, and issue a read enable signal for the row input FIFO; The row input delay module is used to control the delay period of the row input FIFO read enable signal based on the row input delay information in the computational structure configuration data.

[0016] Furthermore, the calculation output processing module includes a column accumulator, an inter-column accumulator, an inter-column FIFO accumulator, an inter-group accumulator, an intra-group FIFO, an inter-group FIFO, a calculation output controller, and a calculation output task RAM; The intra-group FIFO is used to cache processed computation data from different basic computation paths under the same column in the Systolic array; Inter-group FIFO is used to cache the processed computation data of the basic computation paths under different columns in the Systolic array; the inter-group FIFO is divided into levels, and the division is based on the number of common factors of the total number of basic computation paths under the same column of the Systolic array. The column accumulator is used to accumulate the outputs of different basic computation paths in the same column of a Systolic array. The inter-column accumulator is used to accumulate the output of the column accumulator from the current column in the Systolic array with the output of the inter-column FIFO accumulator from the previous column. The intra-column FIFO accumulator is used to accumulate the output of the inter-column accumulator from the current column and the intra-column FIFO output from the previous column within the same Systolic array group. The inter-column FIFO accumulator is used to accumulate the FIFO outputs from the same level of array groups in different Systolic array groups from the current column. The computation output task RAM is used to store computation output task information from the computation structure configuration information; the computation output task information includes the computation input task number and the data cache FIFO index; The computation output controller is used to poll the computation output tasks in the computation output task RAM, and read the data from the corresponding array group intra-FIFO or inter-array group FIFO according to the data cache FIFO index, and send it to the layer output processing module through the internal data flow bus; the computation output controller is also used to send the computation output task number to the layer output processing module. The layer output processing module includes a layer output task RAM, an iteration task RAM, a layer output message generation module, an iteration task controller, a layer output task controller, a Winograd input conversion module, an iteration accumulation module, an accumulation data cache module, a layer output data cache FIFO, an activation arithmetic unit, and a data rearrangement module; The layer output task RAM is used to store the layer output task information in the computing structure configuration information. The layer output task information includes the layer output task number, write start address, and write offset. The Iteration Task RAM is used to store the iteration task information in the computational structure configuration information. The iteration task information includes the iteration task number, read start address, and read offset. The layer output message generation module is used to generate corresponding layer output task messages and iteration task messages based on the calculation output task number; The iterative task controller is used to read the corresponding iterative task information from the iterative task RAM according to the iterative task message, and then read the iterative data from the external memory through the external data flow bus based on the read start address and read offset. The Winograd input conversion module is used to process iterative data based on the read mode in the current iterative task information; The iterative accumulation module is used to determine whether to accumulate the iterative data and the calculation output data based on the current calculation mode, and then temporarily store the processed data in the accumulation data buffer FIFO; The activation unit is used to activate the output data of the accumulated data buffer FIFO; The data rearrangement module is used in Winograd computation mode to rearrange the final output feature data so that the elements in the data are mapped to external memory in a left-to-right and top-to-bottom order; The layer output task controller is used to read the corresponding layer output task information from the layer output task RAM according to the layer output task message, and then store the data buffer FIFO to the external memory through the external data flow bus based on the write start address and write offset.

[0017] In summary, the beneficial effects achieved by the above-described technical solutions conceived in this invention compared with the prior art are as follows: (1) The present invention adopts a high-performance computing module based on a Systolic array computing architecture. After fixing the weight parameters, it can simultaneously complete the parallel computing of multiple basic computing paths. After the computing structure is configured, these computing paths can complete the computing tasks under different input channel dimensions and different output channel dimensions during the convolution computing process. The acceleration system designed in this way can achieve a high operating frequency with low power consumption. Therefore, it has the characteristics of high computing throughput and low energy consumption ratio, which is very suitable for the high computing power requirements of convolutional neural networks.

[0018] (2) The online configurable computing structure adopted in this invention is implemented by configurable units within the coprocessor and a configurable data flow scheduling method. Both adopt a dual-buffer design, and the configuration process and the working process do not interfere with each other. Compared with instruction-based accelerators, this invention does not require additional instruction access and decoding overhead. Compared with dedicated accelerators, this invention supports the acceleration of convolutional neural networks with different layer structures. Relatively speaking, the workflow of this invention is simple and efficient, while also possessing good flexibility and scalability.

[0019] (3) In the deep parallel convolution mode adopted in this invention, the row input delay in the computation input control module and the delay mechanism in the PE unit are combined. The position of the sparse convolution kernel on the Systolic array can be adjusted to give it a richer layout space. This improves the utilization of hardware computing resources in sparse convolution computation.

[0020] (4) In the deep parallel convolution mode adopted in this invention, each processing unit in the Systolic array group can be regarded as a 1*1*1 convolution kernel performing a convolution operation on the local input feature map. Under this premise, by calculating the data index of the input processing module and integrating the data of the output processing module, the input feature map is continuously read from the external memory by the layer input module in the order of row, column, and input channel, and forwarded to the high-performance computing module for calculation. This method improves the utilization rate of storage bandwidth between the coprocessor and the external memory, improves the reuse rate of input feature map data and weight parameters, and eliminates the need for additional indexing, rearrangement, etc., of the input feature map data cached on the chip, simplifying the design process and saving logic resources and on-chip storage resources.

[0021] (5) In the Winograd convolution mode adopted in this invention, the high-performance computing module is reused to perform matrix multiplication operations. The Winograd algorithm can be further optimized for the 3*3 convolution calculation with a stride of 1, so that this invention has better acceleration performance. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of a configurable convolutional neural network acceleration system based on a Systolic array, as proposed in one embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of a deep parallel unfolding optimization method in one embodiment of the present invention.

[0024] Figure 3 This is a system workflow in one embodiment of the present invention.

[0025] Figure 4This is a schematic diagram of the coprocessor structure in one embodiment of the present invention.

[0026] Figure 5 This is a schematic diagram of the calculation input module in one embodiment of the present invention.

[0027] Figure 6 This is a data flow scheduling mechanism proposed in one embodiment of the present invention.

[0028] Figure 7 This is a schematic diagram of the high-performance computing module in one embodiment of the present invention.

[0029] Figure 8 This is a schematic diagram of the calculation input module in one embodiment of the present invention.

[0030] Figure 9 This is a schematic diagram of the structure of a configurable Systolic array in one embodiment of the present invention.

[0031] Figure 10 This is a schematic diagram of the PE unit in one embodiment of the present invention.

[0032] Figure 11 This is a schematic diagram of the pulsation mechanism of the PE unit in one embodiment of the present invention.

[0033] Figure 12 This is a schematic diagram of the delay mechanism of the PE unit in one embodiment of the present invention.

[0034] Figure 13 This is a schematic diagram of the layer output processing module in one embodiment of the present invention.

[0035] Figure 14 This is a schematic diagram of the deep parallel convolution mode in one embodiment of the present invention.

[0036] Figure 15 This is a schematic diagram of the structure of the layer output module in one embodiment of the present invention.

[0037] Figure 16 This is a schematic diagram of the Winograd input conversion process in one embodiment of the present invention.

[0038] Figure 17 This is a schematic diagram illustrating the matrix multiplication implementation of Winograd dot product calculation in one embodiment of the present invention.

[0039] Figure 18 This is a schematic diagram of the complete parallel computation process of Winograd dot product calculation in one embodiment of the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0041] This invention provides a configurable convolutional neural network acceleration system based on a Systolic array, comprising a heterogeneous hardware system consisting of a control processor and a coprocessor, which implements an online configurable computing architecture. The control processor and the coprocessor are connected via a high-speed expansion interface and a low-speed signal interface. Figure 1 As shown, the configurable convolutional neural network acceleration system based on a systolic array of the present invention comprises a heterogeneous hardware system consisting of a control processor and a coprocessor. The control processor is responsible for monitoring the working state of the coprocessor, sending control signals, and providing the image data and computational structure configuration data required during the computation process. The coprocessor is responsible for performing convolution calculations based on the control signals, computational structure configuration data, and image data provided by the control processor, and outputting the final calculated output feature map data to the control processor. The two are connected via a high-speed signal interface and a low-speed signal interface.

[0042] The online configurable computing architecture is implemented by configurable units within the coprocessor and a configurable data flow scheduling method. The configurable units include registers, task RAM, and a selector MUX based on a dual-buffer mechanism; they store convolution parameters for different convolutional layers. The configurable units also store different task information, including layer input tasks, layer output tasks, computation input tasks, computation output tasks, and iteration tasks. Furthermore, the configurable units determine the data source for the basic computing units. The configurable data flow scheduling method consists of a message queue-based controller and task RAM, guiding the flow of different data streams between modules.

[0043] During system operation, the control processor is used to send custom computing structure configuration data and image data to the coprocessor through a high-speed expansion interface.

[0044] The control processor is also used to send layer indication signals, convolution mode signals, and computation mode signals to the coprocessor through a low-speed signal interface; wherein, the layer indication signal indicates the layer information including the input layer, intermediate layer, and output layer; the convolution mode signal includes depth-parallel convolution mode and Winograd convolution mode; and the computation mode signal includes iterative mode and non-iterative mode.

[0045] The coprocessor is used to send status signals to the control processor via a low-speed signal interface; the status signals include configurable status, configuration ready status, computation in progress status, and output status.

[0046] The coprocessor in this invention can handle the computational tasks involved in the input layer, intermediate layer, and output layer of different convolutional neural network models; When the system processes the input layer computation task indicated by the layer indicator signal, after the control processor has finished sending the computation structure configuration data and image data to the write processor, the coprocessor automatically starts the input layer convolution computation task based on the currently received convolution mode signal and computation mode signal, and sends a status signal to the control processor through a low-speed signal structure; the coprocessor is also used to cache the intermediate data generated during the computation based on the current layer indicator signal; the intermediate data includes the output feature data generated by the input layer and intermediate layer convolution computation.

[0047] When the system processes the input layer and intermediate layer computation tasks indicated by the layer indicator signals, the control processor also receives status signals from the coprocessor via a low-speed signal interface. When the configurable status signal transmitted by the coprocessor is set, the control processor sends the next-layer computational structure configuration data and computation mode signal to the coprocessor via a high-speed expansion interface. The next-layer computation task is then executed automatically.

[0048] When the system processes the output layer indicated by the layer indicator signal, the coprocessor is also used to send the output data to the control processor through a high-speed extended interface; wherein, the output data includes the output feature map generated by the convolution calculation of the output layer.

[0049] The acceleration method in this invention supports two acceleration methods: depth-parallel convolution and Winograd convolution for convolutional layers with different convolution parameters; wherein, the convolution parameters include convolution kernel size, weight parameters, and stride parameters.

[0050] The deep parallel convolution mode is based on decomposing the original convolution kernel into several 1*1*1 convolution kernels, and performing convolution calculations and merging calculations of the above 1*1*1 convolution kernels through a computational structure constructed by the computational structure configuration information.

[0051] Winograd convolution mode is based on converting convolution computation into matrix multiplication, and performs its computation process through a computational structure built using computational structure configuration information.

[0052] Preferably, the coprocessor includes a control and interface module, a system configuration module, a layer input processing module, a high-performance computing module, and a layer output processing module.

[0053] The control and interface module is connected to the control processor through a high-speed expansion interface and a low-speed signal interface. The control and interface module is used to transmit data with the control processor to receive input computational structure configuration data, image data, layer indicator signals, convolution mode signals and computation mode signals. Based on the receiving progress of computational structure configuration data and image data, combined with the layer indicator signals, the module controls the generation of corresponding enable signals and status signals. Then, the enable signals are transmitted to the corresponding modules in the system, and the status signals are output to the control processor through the low-speed signal interface. The system configuration module is used to parse the computing structure configuration data received by the control and interface module from the control processor, and distribute the parsed computing structure configuration data to the corresponding modules through the configuration data flow bus.

[0054] The layer input processing module is used to read the input feature data of the current layer from the external memory according to the layer input task determined by the computational structure configuration data. Then, based on the convolution mode signal received by the control and interface module, the input feature data is processed and sent to the high-performance computing module through the internal data flow bus.

[0055] The high-performance computing module receives input feature data from the layer input processing module, performs convolution operations on the computing structure constructed based on the computing structure configuration data, and sends the computational output data obtained from the convolution calculation to the layer output processing module through the internal data flow bus.

[0056] The layer output processing module receives computation output data from the high-performance computing module, processes the input feature data based on the computation mode signal received by the control and interface module, and then stores the output data to the external memory through the external data stream bus according to the layer output task determined by the computation structure configuration data.

[0057] Preferably, the layer input processing module includes a Winograd input conversion module, a layer input task RAM, a layer input task controller, a layer input data buffer FIFO, and a computation input task controller.

[0058] The layer input task RAM is used to store the layer input task information saved in the computational structure configuration information; the layer input task information includes the layer input task number, read start address, read offset, and computation input task index.

[0059] The layer input task controller is used to poll the layer input tasks. When a layer input task is executed, the layer input task controller reads the corresponding layer input task information from the layer input task RAM, and then reads the input feature data from the external memory based on the read start address, read offset and external data flow bus.

[0060] The Winograd input transformation module is used to perform input transformation operations in the Winograd fast algorithm and cache them to the layer input data cache FIFO.

[0061] The computation input task controller is used to send the data in the layer input data buffer FIFO and the computation input task index read from the layer input task RAM to the high-performance computing module.

[0062] Preferably, the high-performance computing module includes a computing input control module, a configurable Systolic array, an array control module, and a computing output processing module.

[0063] The computation input control module receives computation input data sent from the layer input processing module and synchronously sends the computation input data to the reconfigurable Systolic array for computation based on the computation input task index.

[0064] Configurable Systolic arrays are used to perform convolution calculations on computational input data based on the weight and bias parameters in the computational structure configuration data.

[0065] The array control module is used to send column control signals to the Systolic array group based on the step size parameters in the computational structure configuration data.

[0066] The computation output processing module receives the output data from the Systolic array group after convolution calculation, processes and buffers it, and sends it to the layer output processing module through the internal data flow bus.

[0067] Preferably, the reconfigurable Systolic array consists of C_PE processing units and O_PE processing units. Both types of processing units are designed based on delay mechanisms, double buffering mechanisms, and control signal pulsation mechanisms.

[0068] The reconfigurable systolic array is used to perform computational tasks under different convolutional modes, including depthwise parallel convolution and Winograd convolution. In each column of the reconfigurable systolic array, several C_PE processing units are arranged consecutively, followed by one O_PE processing unit. This combination is referred to as the basic computational path. The C_PE and O_PE units have the same structure, but the C_PE units are only used for computation, while the O_PE units are also used to output the computation results to the output processing module.

[0069] When the calculation bit width is 8 bits, the processing unit can perform two multiply-accumulate operations per clock cycle; when the calculation bit width is 16 bits, the processing unit can perform one multiply-accumulate operation per clock cycle. The sign-to-bit-width conversion module in the PE unit is used to perform the corresponding sign-to-bit-width conversion during 8-bit bit-width operations to ensure the correctness of the calculation results.

[0070] The delay mechanism of the PE unit is used to delay the computation input data received in the current clock cycle for several cycles before computation. This mechanism is used to adjust the position of the weight parameters in the sparse convolution kernel in the Systolic array.

[0071] The dual-buffering mechanism of the PE unit is used to synchronously receive the weight parameters and bias parameters from the configuration data of the next layer of computing structure during the computation process.

[0072] The control signal pulsation mechanism of the PE unit is used to transmit the control signal from top to bottom to the immediately adjacent PE unit below in each clock cycle.

[0073] Preferably, the array control module generates corresponding column control signals based on the step size parameters in the computational structure configuration data.

[0074] The column control signal is used to control the calculation process of all PE cells in the corresponding column of the Systolic array based on the control signal pulsation mechanism of the PE cell; this mechanism is used to skip the corresponding calculation operation when the step size parameter is not 1.

[0075] Preferably, the calculation input control module includes a row input FIFO, a calculation input controller, a calculation input data buffer, and a row input delay module.

[0076] The row input FIFO is used to buffer the row input data of the Systolic array.

[0077] The computation input data buffer is used to cache computation input data output from the layer input processing module.

[0078] The computation input controller is used to forward the computation input data cached in the computation input data buffer to the corresponding row input FIFO according to the computation input task index from the layer input processing module, and issue the read enable signal of the row input FIFO.

[0079] The row input delay module is used to control the delay period of the row input FIFO read enable signal based on the row input delay information in the computational structure configuration data.

[0080] Preferably, the calculation output processing module includes a column accumulator, an inter-column accumulator, an inter-column FIFO accumulator, an inter-group accumulator, an intra-group FIFO, an inter-group FIFO, a calculation output controller, and a calculation output task RAM; The intra-group FIFO is used to cache processed computation data from different basic computation paths under the same column in the Systolic array; Inter-group FIFO is used to cache the processed computation data of the basic computation paths under different columns in the Systolic array; the inter-group FIFO is divided into levels, and the division is based on the number of common factors of the total number of basic computation paths under the same column of the Systolic array. The column accumulator is used to accumulate the outputs of different basic computation paths in the same column of a Systolic array. The inter-column accumulator is used to accumulate the output of the column accumulator from the current column in the Systolic array with the output of the inter-column FIFO accumulator from the previous column. The intra-column FIFO accumulator is used to accumulate the output of the inter-column accumulator from the current column and the intra-column FIFO output from the previous column within the same Systolic array group.

[0081] The inter-column FIFO accumulator is used to accumulate the FIFO outputs of the same level array group from the current column under different Systolic array groups.

[0082] The computation output task RAM is used to store computation output task information from the computation structure configuration information; the computation output task information includes the computation input task number and the data cache FIFO index.

[0083] The computation output controller polls the computation output tasks in the computation output task RAM, reads the data from the corresponding intra-array FIFO or inter-array FIFO according to the data cache FIFO index, and sends it to the layer output processing module via the internal data flow bus. The computation output controller is also used to send the computation output task number to the layer output processing module.

[0084] Preferably, the layer output processing module includes a layer output task RAM, an iteration task RAM, a layer output message generation module, an iteration task controller, a layer output task controller, a Winograd input conversion module, an iteration accumulation module, an accumulation data cache module, a layer output data cache FIFO, an activation arithmetic unit, and a data rearrangement module.

[0085] The layer output task RAM is used to store the layer output task information in the computing structure configuration information. The layer output task information includes the layer output task number, write start address, and write offset.

[0086] The Iteration Task RAM is used to store the iteration task information in the computing structure configuration information. The iteration task information includes the iteration task number, read start address, and read offset.

[0087] The layer output message generation module is used to generate corresponding layer output task messages and iteration task messages based on the calculation output task number.

[0088] The iterative task controller is used to read the corresponding iterative task information from the iterative task RAM based on the iterative task message, and then read the iterative data from the external memory through the external data flow bus based on the read start address and read offset.

[0089] The Winograd input transformation module is used to process iterative data based on the read mode in the current iteration task information.

[0090] The iterative accumulation module is used to determine whether to accumulate the iterative data and the calculation output data based on the current calculation mode, and then temporarily store the processed data in the accumulated data cache FIFO.

[0091] The activation unit is used to activate the output data of the accumulated data buffer FIFO.

[0092] The data rearrangement module is used in Winograd computation mode to rearrange the final output feature data so that the elements in the data are mapped to external memory in a left-to-right and top-to-bottom order.

[0093] The layer output task controller is used to read the corresponding layer output task information from the layer output task RAM according to the layer output task message, and then store the data buffer FIFO to the external memory through the external data flow bus based on the write start address and write offset.

[0094] The convolutional neural network acceleration system according to this invention supports inference acceleration for CNN models with different layer structures through configurable computational structures and data flow scheduling. The configuration process is simple and efficient, avoiding the complex process of instruction decoding and execution and the additional latency it introduces. All configurable units adopt a dual-buffered design, supporting parallel execution of the configuration and computation processes. Furthermore, this system is compatible with the Winograd fast algorithm for optimizing 3×3 convolutional kernels, while significantly improving the utilization of computational resources in sparse convolutions.

[0095] On the other hand, such as Figure 2As shown, this system employs an optimization strategy based on deep parallel unfolding of convolution operations, applied in a deep parallel convolution mode. This strategy decomposes the original convolution kernel into several 1*1*1 convolution kernels. It should be noted that the convolution regions of these 1*1*1 convolution kernels on the input feature map differ from the original kernel. This means that computational tasks involving different 1*1*1 convolution kernels must consider strict temporal relationships when merging them. Under these two conditions, the input feature data only needs to be calculated in a left-to-right, top-to-bottom order. Compared to traditional convolution computation, no additional input transformation process is required. The specific embodiments described later will illustrate how to perform the convolution computation and merging of the aforementioned 1*1*1 convolution kernels using a computational structure configuration information. Furthermore, for convolution computations with different size and stride parameters, no additional input transformation process is needed. This invention employs a unique control signal pulsation mechanism to control the processing unit to skip the corresponding computation process.

[0096] The following is in conjunction with the appendix Figures 3-18 The invention will be described in more detail with reference to specific embodiments.

[0097] Example 1: In such Figure 3 In the illustrated process, the computation and configuration processes of this system can be performed simultaneously. This system divides the convolutional layers of the processed convolutional neural network into three categories: input layer, intermediate layer, and output layer, and uses layer indicator signals to indicate these three layers respectively.

[0098] When processing the input layer, the control processor first sends the input layer's computational structure configuration data to the coprocessor. After configuration, the control processor sends image data to the coprocessor and simultaneously sends corresponding control signals to initiate the coprocessor's computation process. The intermediate results generated by the coprocessor are temporarily stored in an externally connected memory. When processing the output layer, the output feature map data generated by the coprocessor is output to the control processor. The system's online configurable computational structure is implemented through configurable units within the coprocessor and a configurable data flow scheduling method. The configurable units include registers based on a dual-buffer mechanism, task RAM, and a selector MUX; they store convolution parameters for different convolutional layers. The configurable units also store various task information, including layer input tasks, layer output tasks, computation input tasks, computation output tasks, and iteration tasks. Furthermore, the configurable units determine the data source for the basic computational units. The configurable data flow scheduling method, comprised of a message queue-based controller and task RAM, guides the flow of different data streams between modules. Therefore, when processing the input, intermediate, and output layers, the control processor can provide configuration data and corresponding control signals for the next layer's computational structure during the current layer's computation. Once the next layer's computational structure is configured and the current layer's computational task is completed, the next layer's computational task will automatically begin.

[0099] Furthermore, Figure 4 A basic block diagram of the coprocessor is provided, describing the data flow between its modules. In this embodiment, the coprocessor includes a control and interface module, a system configuration module, a layer input processing module, a high-performance computing module, and a layer output processing module. The configurable data flow scheduling method is implemented based on the coprocessor's internal configuration data flow bus, internal data flow bus, and external data flow bus. All three buses are based on the AXI4 protocol and are driven by DMA. The configuration data flow bus is used for data transfer of computing architecture configuration data between modules within the coprocessor. The internal data flow bus is used for data transfer of internal data between modules within the coprocessor; this internal data includes output data from internal modules of the coprocessor. The external data flow bus is used for the transfer of internal and external data between the coprocessor and external memory; this external data includes data stored in external memory.

[0100] To better illustrate the working process of this invention, let CI be the number of input feature map channels, with its index ci; CO be the number of output feature map channels, with its index co; K be the convolution kernel size; R and C be the row height and column width of the output feature map, respectively; and S be the convolution stride.

[0101] Figure 5 This is a block diagram of the layer input module. During configuration, the layer input tasks included in the computation structure configuration data are updated by writing the information stored in the layer input task RAM. Each input feature channel ci corresponds to one layer input task. During computation, such as... Figure 6The example shown is based on a configurable data flow approach (the figure shows three layer input tasks, numbered 0, 1, and 2). The layer input task control state machine polls each layer input task. Specifically, it reads the contents of the layer input task RAM address pointed to by the current layer input task index, and initiates a DMA request using information such as the read start address, read offset, and calculated input task index to read the data from the external memory. In deep convolution mode, the layer input data is directly forwarded to the high-performance computing module; in Winograd convolution mode, the layer input data is first processed by the Winograd input conversion module before being forwarded to the high-performance computing module.

[0102] like Figure 7 As shown, the high-performance computing module includes a computational input processing module, an array control and configuration module, a configurable Systolic array group, and a computational output processing module. The high-performance computing module receives input feature data from the layer input processing module, performs convolution operations on the computational structure constructed based on the computational structure configuration data, and sends the computational output data obtained from the convolution calculation to the layer output processing module via an internal data flow bus.

[0103] In such Figure 8 In the computational input processing module shown, the layer input data input by the layer input processing module is cached in the computational input buffer. The computational input controller forwards the data in the computational input buffer to the corresponding row input FIFO via the module's internal bus according to the computational input task index. During the computation process, the computational input data cached in each row input FIFO is delayed by the delay module for a pre-set delay period before entering the configurable Systolic group for computation.

[0104] In specific convolution calculations, taking a 5x5 Systolic array based on 5x5 PE units as an example, it can effectively accelerate 5x5 convolution calculations. However, when accelerating 3x3 convolution calculations, it actually uses only 9 PE units, meaning 16 PE units are idle. Furthermore, this Systolic array does not support convolution calculations larger than 5. Designing Systolic arrays of various sizes in the accelerator to support convolution calculations of different sizes can alleviate the aforementioned generality problem to some extent, but it will result in even more idle PE processing units. For example, when performing 3x3 convolution calculations, all PE units within 5x5 and 7x7 Systolic arrays will be idle.

[0105] Therefore, to support convolution calculations with different convolution parameters and improve the versatility and PE unit utilization of the acceleration system across various computational tasks, this system designs a configurable Systolic array group such as... Figure 9As shown, it is used to complete computational tasks under different convolution modes, including depthwise parallel convolution computation and Winograd convolution computation. In each column of the reconfigurable Systolic array, several C_PE processing units are arranged consecutively, followed by one O_PE processing unit. This invention refers to such a combination as the basic computation path. The C_PE and O_PE units have the same structure, but the C_PE units are only used for computation, while the O_PE units are also used to output the computation results to the output processing module.

[0106] Specifically, Figure 10 This is a structural diagram of a PE unit designed based on a configurable, delay mechanism, and a dual-buffering mechanism. The delay mechanism of the PE unit is used to postpone the computational input data received in the current clock cycle for a certain number of cycles before computation. In this embodiment, the maximum number of delay cycles is 3. This mechanism is used to adjust the position of the weight parameters in the sparse convolution kernel within the Systolic array. The dual-buffering mechanism of the PE unit is used to synchronously receive the weight parameters and bias parameters from the configuration data of the next layer's computational structure during the computation process. The control signal pulsation mechanism of the PE unit is used to pass the control signal from top to bottom to the immediately adjacent PE unit in each clock cycle.

[0107] Specifically, when the calculation bit width is 8 bits, the processing unit can perform two multiply-accumulate operations per clock cycle; when the calculation bit width is 16 bits, the processing unit can perform one multiply-accumulate operation per clock cycle. The sign-to-bit-width conversion module within the PE unit performs the corresponding sign-to-bit-width conversion during 8-bit bit-width operations to ensure the correctness of the calculation results. During operation, configuration process data, including delay parameters, weight parameters, and bias parameters, is input from the upper PE unit or array control module, latched into a register determined by the strobe data, and the data latched in the register in the previous clock cycle is transferred to the lower PE unit.

[0108] During the calculation process, in each clock cycle, the intermediate data register latches the intermediate data calculated and output from the upper PE unit; the control signal register latches the column control signals input from the upper PE unit or the array control module and outputs them to the lower PE unit; the calculated data is input from the left PE unit or the calculation input control module, and is delayed and timed by the calculated data register, and then output to the right PE unit in the next clock cycle.

[0109] When the step size S=1, the input feature data a( ci , l ) sequentially enter the Systolic array group, and with the weight parameter w( k_ ci , k_x , k_y Convolution is performed, where lInput sequence numbers for feature data. k_ci These are the coordinates of the three-dimensional convolution kernel channels. k_x 3D convolution kernel x coordinate, k_y 3D convolution kernel y Coordinates. For example... Figure 11 As shown, based on the column control signal pulsation mechanism, the input feature data is continuously input, and the control signal propagating longitudinally in the PE unit precisely controls the multiplication and addition calculation of each weight parameter with the corresponding input feature data. The control signal of the O_PE unit is its output enable signal.

[0110] The delay register within the PE unit delays and increments the data in the calculation data register. In each clock cycle, the delay parameter in the delay parameter register, the intermediate data register, and each parameter register transmit data to the multiply-accumulator. The multiply-accumulator performs calculations based on the enable signal output from the control signal register and outputs the calculated intermediate data to the PE unit below. For example... Figure 12 As shown, after pruning, the 3*3 convolution kernel leaves only the weight parameters marked in gray. Its layout on the Systolic array (left) results in the waste of 6 PE units (marked in dark gray). After adjusting using the delay mechanism of PE units (right), the delay values ​​of the PE units containing w(0,0,0) and w(0,1,1) are set to 2 and 1 respectively. The three weight parameters can be laid out on a basic computation path, and the empty PE units (marked in white) can be used to lay out other weight parameters.

[0111] like Figure 13As shown, in this embodiment, the computation output processing module includes a column accumulator, an inter-column accumulator, an inter-column FIFO accumulator, an inter-group accumulator, an intra-group FIFO, an inter-group FIFO, a computation output controller, and a computation output task RAM. The intra-group FIFO is used to cache the processed computation data of different basic computation paths under the same column in the Systolic array. The inter-group FIFO is used to cache the processed computation data of basic computation paths under different columns in the Systolic array. The inter-group FIFO is hierarchically divided based on S... The ystolic array is calculated by counting the common factors of the total number of basic computation paths in the same column. Column accumulators are used to accumulate the outputs of different basic computation paths in the same column of the ystolic array. Inter-column accumulators are used to accumulate the output of the column accumulator from the current column within the same ystolic array group with the output of the inter-column FIFO accumulator from the previous column. Intra-column FIFO accumulators are used to accumulate the output of the inter-column accumulator from the current column within the same ystolic array group with the intra-array FIFO output from the previous column. Inter-column FIFO accumulators are used to accumulate the FIFO outputs from the same level array groups within the current column of different ystolic array groups. The computation output task RAM stores computation output task information from the computation structure configuration information; this information includes the computation input task number and the data cache FIFO index. The computation output controller polls the computation output tasks in the computation output task RAM, reads the data from the corresponding intra-array FIFO or inter-array FIFO according to the data cache FIFO index, and sends it to the layer output processing module via the internal data flow bus. The computation output controller is also used to send the computation output task number to the layer output processing module.

[0112] The computation output control module uses different types of accumulators to merge the computational processes involved in the basic computational paths within the Systolic array group through accumulation calculations, thereby reducing excessive on-chip memory usage. Specifically, Figure 14 A specific example of convolution calculation with 3 input channels, 2 output channels, and a 3*3 kernel size is given. After calculation by the Systolic array group, the computation data generated by different basic computation paths enters the computation output processing module and is accumulated by each accumulator. The accumulator marked by the gray box only serves as a delay for timing alignment. Finally, the computation output data is buffered in the FIFO of the two array groups. The computation output controller polls different computation output tasks and sends the computation output data to the layer output processing module for processing.

[0113] like Figure 15As shown, the layer output processing module includes a layer output task RAM, an iteration task RAM, a layer output message generation module, an iteration task controller, a layer output task controller, a Winograd input conversion module, an iteration accumulation module, an accumulation data cache module, a layer output data cache FIFO, an activation arithmetic unit, and a data rearrangement module. In non-iterative computation mode, the computation output data from the high-performance computing module corresponds to a complete output feature map for one output feature channel, without needing to accumulate with historical data. In this case, the computation output data directly enters the accumulation data cache FIFO for buffering via the multiplexer MUX. When data transmission is complete, the layer output task message cache FIFO receives a transmission message from the high-performance computing module, including the computation output task number and transmission length. Based on this message field, the layer output controller searches for the corresponding external memory write address and offset in the layer output task RAM and initiates a write request to the external data stream DMA to write the buffered data to the external memory.

[0114] In iterative computation mode, the current computation output data needs to be added to the intermediate results generated in the previous iteration, i.e., the iterative data, to obtain the final output. Therefore, the computation output data first enters the computation output data FIFO. Simultaneously, the layer output information generation module generates two types of task messages based on the messages transmitted by the high-performance computing module: iterative task messages and layer output task messages, which are written into their respective message FIFOs. Subsequently, the iterative task controller initiates a read request to the external data stream DMA based on the message content, reading the data generated in the previous iteration from memory (its reading mechanism is consistent with the layer input processing module). The read iterative data is then sequentially entered into the iterative task data FIFO and added to the current computation data. The subsequent data storage process is consistent with the non-iterative mode.

[0115] Example 2: The Winograd convolution mode supported by this invention aims to optimize 3x3 convolution operations using the Winograd fast algorithm. In implementation, the hardware structure of Embodiment 1 is reused. Compared with the depth-parallel convolution mode, the changes involve the layer input processing module, the high-performance computing module, and the layer output processing module.

[0116] The Winograd fast algorithm consists of three steps: input transformation, dot product calculation, and output transformation. The input to the input transformation process is a tile containing 4×4 input feature data. Adjacent tiles overlap by 2 elements. Therefore, to reduce redundant data reading, as follows... Figure 16 As described, each time a 2×4 block (represented by input feature map data of different gray levels) is read from the input feature map, these basic blocks are stitched together to form the required tile.

[0117] The tile input is represented by matrix D as follows:

[0118] Input conversion process It can be expanded as

[0119]

[0120] Due to the transformation matrix and The elements consist of 0, 1, and -1, and the above calculation process can be implemented through simple addition. The entire input conversion process is completed by the Winograd input conversion module in the layer input processing module and the layer output processing module. That is, within one clock cycle, the 2×4 input feature data block read from the external memory is concatenated into a tile in the shift register group, and then passes through the input conversion module composed of several adders to form the layer input data of the high-performance computing module. .

[0121] For convenience, Two-dimensional subscript Convert to one-dimensional subscript ,get Its representative input feature channel Next The matrix obtained after input transformation of each tile will be Represented in matrix form

[0122] Matrix elements middle This represents its position number within the matrix.

[0123] For a convolutional layer with CI input feature channels and CO output feature channels, Winograd contains CO dot product calculation processes. The dot product calculation process for the p-th tile is denoted as... ,in For input feature channels With output features The weight matrix corresponding to the convolution kernel after weight transformation:

[0124] Systolic arrays offer high computational efficiency for matrix multiplication; therefore, when using high-performance computing modules to accelerate the dot product calculation process, it is necessary to... The calculation is converted into matrix multiplication form, at which point... The calculation formula can be written as:

[0125] The above matrix Inner element Its calculation process can also be obtained through matrix multiplication.

[0126]

[0127] Extend the above process to Dot product calculation for each output channel, and considering p consecutive tile inputs in the CI input channels, we have:

[0128] This calculation can be performed by fixing the right-multiplication matrix as a weight parameter in the systolic array, and then calculating the left-multiplication matrix by entering it into the systolic array column by column. Specifically, using... CO= 6, CI= Taking 3 as an example, the calculation process of the above formula on a Systolic array is as follows: Figure 17 As shown. At this time, the row input of the Systolic array is In the same position in the matrix The column outputs of the sequence, the Systolic array, are the output channels. co Down sequence.

[0129] A complete Winograd dot product calculation requires the simultaneous output of 16... sequence( =0,1,2…15). Therefore, considering the configurable input and output characteristics of the systolic arrays in the high-performance computing module designed in this paper, in order to maximize the pipelining and parallelization of the computation process, 16 of the above-mentioned systolic arrays are needed to jointly complete the dot product calculation task. Figure 18 The calculation process of the Winograd dot product on a configurable Systolic array is described in detail.

[0130] Output transformation process in the Winograd fast algorithm Due to the matrix With matrix The elements in the dot product are also composed only of 0, 1, and -1, and the result is calculated during the dot product calculation. M The elements are output at the same time, so in the calculation output processing module of Embodiment 1, this addition operation can be completed by different forms of accumulators.

[0131] Considering only the position number n in the dot product result M, the output conversion process can be calculated using the following formula:

[0132] in:

[0133]

[0134]

[0135]

[0136] In Winograd convolution mode, the order of the computational output data generated by the high-performance computing module in one clock cycle differs from that in depth-parallel convolution mode. Therefore, it is necessary to adjust the computational output matrix. The elements in the layer are rearranged to meet the requirement of sequential storage in external memory. This process is completed by the rearrangement module in the layer output processing module.

[0137] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A configurable convolutional neural network acceleration system based on a systolic array computing structure, characterized in that, include: The control processor and coprocessor are connected via a high-speed expansion interface and a low-speed signal interface. The coprocessor internally includes configurable units and configurable data flow scheduling methods. The configurable units include registers, task RAM, and selector MUX designed based on a dual-buffer mechanism. The configurable units are used to store convolution parameters of different convolutional layers and different task information, as well as to determine the data source of the basic computing units. The configurable data flow scheduling methods include a controller and task RAM designed based on a message queue, which are used to guide the flow of different data flows between modules. During system operation, the control processor is used to send custom computing structure configuration data and image data to the coprocessor through a high-speed expansion interface; The control processor is also used to send layer indication signals, convolution mode signals, and computation mode signals to the coprocessor via a low-speed signal interface; wherein, the layer indication signal indicates the layer information including the input layer, intermediate layer, and output layer; the convolution mode signal includes depth-parallel convolution mode and Winograd convolution mode; and the computation mode signal includes iterative mode and non-iterative mode. The coprocessor is used to send status signals to the control processor via a low-speed signal interface; the status signals include configurable status, configuration ready status, computation in progress status, and output status.

2. The configurable convolutional neural network acceleration system based on a Systolic array according to claim 1, characterized in that, Coprocessors are used to handle computational tasks involved in the input, intermediate, and output layers of different convolutional neural network models; When the system processes the input layer computation task indicated by the layer indication signal, the control processor sends computation structure configuration data and image data to the coprocessor. The coprocessor automatically starts the input layer convolution computation task based on the currently received convolution mode signal and computation mode signal, and sends status signals to the control processor through a low-speed signal structure. The coprocessor is also used to cache intermediate data generated during the computation based on the current layer indication signal. The intermediate data includes the output feature data generated by the input layer and intermediate layer convolution computation. When the system processes the input layer and intermediate layer computing tasks indicated by the layer indicator signals, the control processor is also used to obtain status signals from the coprocessor through the low-speed signal interface; when the configurable status signal transmitted by the coprocessor is set, the control processor sends the next layer computing structure configuration data and computing mode signal to the coprocessor through the high-speed expansion interface; thereafter, the next layer computing task is automatically executed. When the system processes the output layer indicated by the layer indicator signal, the coprocessor is also used to send the output data to the control processor through a high-speed extension interface; wherein, the output data includes the output feature map generated by the convolution calculation of the output layer.

3. The configurable convolutional neural network acceleration system based on a Systolic array according to claim 1, characterized in that, It also includes two acceleration methods: depth-parallel convolution and Winograd convolution for convolutional layers with different convolution parameters; where convolution parameters include kernel size, weight parameters, and stride parameters; The deep parallel convolution mode is based on decomposing the original convolution kernel into several 1*1*1 convolution kernels, and performing convolution calculation and calculation merging of the above 1*1*1 convolution kernels through the computation structure configuration information. Winograd convolution mode is based on converting convolution computation into matrix multiplication, and performs its computation process through a computational structure built using computational structure configuration information.

4. The configurable convolutional neural network acceleration system based on a Systolic array according to claim 1, characterized in that, The coprocessor includes a control and interface module, a system configuration module, a layer input processing module, a high-performance computing module, and a layer output processing module; The control and interface module is connected to the control processor through a high-speed expansion interface and a low-speed signal interface. The control and interface module is used to transmit data with the control processor to receive input computational structure configuration data, image data, layer indicator signals, convolution mode signals and computation mode signals. Based on the receiving progress of computational structure configuration data and image data, the module controls the generation of corresponding enable signals and status signals in combination with the layer indicator signals. The enable signals are then transmitted to the corresponding modules in the system, and the status signals are output to the control processor through the low-speed signal interface. The system configuration module is used to parse the computing structure configuration data received by the control and interface module from the control processor, and distribute the parsed computing structure configuration data to the corresponding modules through the configuration data flow bus; The layer input processing module is used to read the input feature data of the current layer from the external memory according to the layer input task determined by the computational structure configuration data. Then, it processes the input feature data based on the convolution mode signal received by the control and interface module and sends it to the high-performance computing module through the internal data flow bus. The high-performance computing module receives input feature data from the layer input processing module, performs convolution operations on the computing structure constructed based on the computing structure configuration data, and sends the computation output data obtained from the convolution calculation to the layer output processing module through the internal data flow bus. The layer output processing module receives computation output data from the high-performance computing module, processes the input feature data based on the computation mode signal received by the control and interface module, and then stores the output data to the external memory through the external data stream bus according to the layer output task determined by the computation structure configuration data.

5. The configurable convolutional neural network acceleration system based on a Systolic array according to claim 4, characterized in that, The layer input processing module includes the Winograd input conversion module, layer input task RAM, layer input task controller, layer input data buffer FIFO, and computation input task controller; The layer input task RAM is used to store the layer input task information saved in the computational structure configuration information; the layer input task information includes the layer input task number, read start address, read offset, and computational input task index; The layer input task controller is used to poll the layer input tasks. When a layer input task is executed, the layer input task controller reads the corresponding layer input task information from the layer input task RAM, and then reads the input feature data from the external memory through the external data flow bus based on the read start address and read offset. The Winograd input transformation module is used to perform input transformation operations in the Winograd fast algorithm and cache them to the layer input data cache FIFO; The computation input task controller is used to send the data in the layer input data buffer FIFO and the computation input task index read from the layer input task RAM to the high-performance computing module.

6. The configurable convolutional neural network acceleration system based on a Systolic array according to claim 4, characterized in that, The high-performance computing module includes a computing input control module, a configurable Systolic array, an array control module, and a computing output processing module; The computation input control module is used to receive computation input data sent from the layer input processing module, and synchronously send the computation input data to the reconfigurable Systolic array for computation based on the computation input task index; Configurable Systolic arrays are used to perform convolution calculations on computational input data based on the weight and bias parameters in the computational structure configuration data. The array control module is used to send column control signals to the configurable Systolic array group based on the step size parameter in the computational structure configuration data; The computation output processing module receives the output data from the Systolic array group after convolution calculation, processes and buffers it, and sends it to the layer output processing module through the internal data flow bus.

7. The configurable convolutional neural network acceleration system based on a Systolic array according to claim 6, characterized in that, The PE unit of the configurable Systolic array group includes a C_PE processing unit and an O_PE processing unit. Both the C_PE processing unit and the O_PE processing unit have a delay mechanism, a double buffer mechanism, and a control signal pulsation mechanism. The configurable Systolic array group is used to complete computational tasks under different convolution modes, including depth-parallel convolution computation and Winograd convolution computation. Each column of the configurable Systolic array group has a basic computational path, which is formed by a combination of several C_PE processing units arranged in succession and an O_PE processing unit. Among them, the C_PE and O_PE have the same structure. The C_PE unit is only used for computation, and the O_PE unit is used for computation and outputting the computation result to the output processing module. When the calculation bit width is 8 bits, the processing unit can perform two multiply-add operations in one clock cycle. When the calculation bit width is 16 bits, the processing unit can perform one multiply-add operation in one clock cycle. The sign and bit width conversion module in the PE unit is used to perform the corresponding sign and bit width conversion during 8-bit bit width operations to ensure the correctness of the calculation results. The delay mechanism of the PE unit is used to delay the computation input data received in the current clock cycle for several cycles before computation. This mechanism is used to adjust the position of the weight parameters in the sparse convolution kernel in the Systolic array. The dual-buffering mechanism of the PE unit is used to synchronously receive the weight parameters and bias parameters from the configuration data of the next layer of computing structure during the computation process; The control signal pulsation mechanism of the PE unit is used to pass the control signal from top to bottom to the immediately adjacent PE unit below in each clock cycle, and to serve as the calculation output enable signal of the O_PE unit.

8. A configurable convolutional neural network acceleration system based on a Systolic array according to claim 7, characterized in that, The array control module generates corresponding column control signals based on the step size parameters in the computational structure configuration data. The column control signal is used to control the calculation process of all PE cells in the corresponding column of the Systolic array based on the control signal pulsation mechanism of the PE cell, so as to skip the corresponding calculation operation when the step size parameter is not 1.

9. The configurable convolutional neural network acceleration system based on a Systolic array according to claim 6, characterized in that, The calculation input control module includes a line input FIFO, a calculation input controller, a calculation input data buffer, and a line input delay module; The row input FIFO is used to buffer the row input data of the Systolic array; The computation input data buffer is used to cache computation input data output from the layer input processing module; The computation input controller is used to forward the computation input data cached in the computation input data buffer to the corresponding row input FIFO according to the computation input task index from the layer input processing module, and issue a read enable signal for the row input FIFO; The row input delay module is used to control the delay period of the row input FIFO read enable signal based on the row input delay information in the computational structure configuration data.

10. The configurable convolutional neural network acceleration system based on a Systolic array according to claim 6, characterized in that, The calculation output processing module includes a column accumulator, an inter-column accumulator, an inter-column FIFO accumulator, an inter-group accumulator, an intra-group FIFO, an inter-group FIFO, a calculation output controller, and a calculation output task RAM; The intra-group FIFO is used to cache processed computation data from different basic computation paths under the same column in the Systolic array; Inter-group FIFO is used to cache the processed computation data of the basic computation paths under different columns in the Systolic array; the inter-group FIFO is divided into levels, and the division is based on the number of common factors of the total number of basic computation paths under the same column of the Systolic array. The column accumulator is used to accumulate the outputs of different basic computation paths in the same column of a Systolic array. The inter-column accumulator is used to accumulate the output of the column accumulator from the current column in the Systolic array with the output of the inter-column FIFO accumulator from the previous column. The intra-column FIFO accumulator is used to accumulate the output of the inter-column accumulator from the current column and the intra-column FIFO output from the previous column within the same Systolic array group. The inter-column FIFO accumulator is used to accumulate the FIFO outputs from the same level of array groups in different Systolic array groups from the current column. The computation output task RAM is used to store computation output task information from the computation structure configuration information; the computation output task information includes the computation input task number and the data cache FIFO index; The computation output controller is used to poll the computation output tasks in the computation output task RAM, and read the data from the corresponding array group intra-FIFO or inter-array group FIFO according to the data cache FIFO index, and send it to the layer output processing module through the internal data flow bus; the computation output controller is also used to send the computation output task number to the layer output processing module. The layer output processing module includes a layer output task RAM, an iteration task RAM, a layer output message generation module, an iteration task controller, a layer output task controller, a Winograd input conversion module, an iteration accumulation module, an accumulation data cache module, a layer output data cache FIFO, an activation arithmetic unit, and a data rearrangement module; The layer output task RAM is used to store the layer output task information in the computing structure configuration information. The layer output task information includes the layer output task number, write start address, and write offset. The Iteration Task RAM is used to store the iteration task information in the computational structure configuration information. The iteration task information includes the iteration task number, read start address, and read offset. The layer output message generation module is used to generate corresponding layer output task messages and iteration task messages based on the calculation output task number; The iterative task controller is used to read the corresponding iterative task information from the iterative task RAM according to the iterative task message, and then read the iterative data from the external memory through the external data flow bus based on the read start address and read offset. The Winograd input conversion module is used to process iterative data based on the read mode in the current iterative task information; The iterative accumulation module is used to determine whether to accumulate the iterative data and the calculation output data based on the current calculation mode, and then temporarily store the processed data in the accumulation data buffer FIFO; The activation unit is used to activate the output data of the accumulated data buffer FIFO; The data rearrangement module is used in Winograd computation mode to rearrange the final output feature data so that the elements in the data are mapped to external memory in a left-to-right and top-to-bottom order; The layer output task controller is used to read the corresponding layer output task information from the layer output task RAM according to the layer output task message, and then store the data buffer FIFO to the external memory through the external data flow bus based on the write start address and write offset.

Images