General convolution computing hardware architecture based on row data flow and its implementation method

Abstract

This invention discloses a hardware architecture for conventional convolutional computing based on row data streams and its implementation method, which is applied to chips to implement conventional convolution operations for neural network mapping. The method utilizes a serial-parallel conventional convolutional computing architecture to achieve delay-free, real-time serial-parallel computing of input data streams. This invention provides a method for optimizing time, area, and power consumption for conventional convolutional computing solutions for data streams of arbitrary granularity.

Description

Technical Field

[0001] The present invention relates to a common convolution computing hardware architecture based on row data stream and its implementation method, belonging to the field of microelectronic integrated circuit design and the direction of neural network customized hardware acceleration design. Background Art

[0002] With the rapid development of artificial intelligence (AI), convolutional neural networks (CNNs) have demonstrated powerful feature extraction capabilities in fields such as computer vision and natural language processing. As the core computational unit of CNNs, convolutional calculations typically account for over 70% of computing resources during model inference. Traditional implementations based on general-purpose processors (CPUs / GPUs) suffer from low energy efficiency and poor real-time performance, making them difficult to meet the low-power, high-real-time requirements of edge computing scenarios. Consequently, academia and industry have begun exploring specialized hardware acceleration architectures for convolutional calculations.

[0003] Existing hardware acceleration solutions mainly adopt two technical routes: one is a reconfigurable architecture based on FPGA, which implements parallel computing of fixed-size convolution kernels through hardware description language; the other is a customized design based on ASIC, which uses structures such as systolic arrays to achieve high-throughput computing.

[0004] However, these architectures generally have three technical bottlenecks:

[0005] First, existing designs mostly focus on fixed-granularity data input (such as 32×32 pixel blocks). When processing irregular data or dynamically changing data streams, additional data padding or cropping operations are required, resulting in wasted computing resources and increased latency.

[0006] Secondly, the traditional pipeline design lacks dynamic scheduling capabilities. When processing different convolutional layers (such as a mixture of 3×3 and 5×5 kernels), the hardware resource utilization rate drops significantly.

[0007] Furthermore, the data reuse mechanism of the existing architecture is relatively simple, usually using a fixed-size input cache window, which makes it difficult to adapt to the increasingly complex new operators in modern networks, such as cross-channel convolution and depth-wise separable convolution.

[0008] Especially in edge computing scenarios, input data is often highly heterogeneous: on the one hand, front-end sensors may generate image / video streams of arbitrary resolution; on the other hand, multimodal fusion applications require processing feature maps of varying dimensions. This leads to two major challenges for traditional hardware architectures: first, fixed data pathways struggle to adapt to dynamically changing input granularity, and frequent data reorganization reduces the proportion of effective computing cycles; second, traditional caching mechanisms do not match the bandwidth between computing units, making computing unit starvation a common phenomenon when processing fine-grained data streams.

[0009] In summary, existing convolutional computing hardware architectures suffer from low hardware resource utilization, poor dynamic adaptability, and suboptimal energy efficiency when processing arbitrary-granularity input data streams. These issues severely restrict the real-time processing capabilities of intelligent edge devices in complex scenarios. Therefore, there is an urgent need to develop a new convolutional architecture that can effectively address these issues. Summary of the Invention

[0010] The purpose of the present invention is to provide a common convolution computing hardware architecture based on row data stream and its implementation method. The technical solution adopted by this method is as follows:

[0011] A common convolution computing hardware architecture is constructed by adopting a 3x3 convolution kernel size, which includes a first input port, a second input port, a first output port, a second output port, two row data cache units, a weight control unit, a plurality of single-output channel convolution computing units, a serial output unit, and an activation and quantization unit;

[0012] The weight control unit includes a RAM unit for storing the weight data required for ordinary convolution calculation and the write and read control logic corresponding to the unit; each single-output channel convolution calculation unit includes three three-stage pipeline multiplication and accumulation units, where each three-stage pipeline multiplication and accumulation unit includes a three-stage pipeline multiplication unit, an accumulation unit and a cache control unit; the activation and quantization unit includes an adder, a multiplier, a binary symbol number judgment logic corresponding to the adder and multiplier, and a multi-stage pipeline structure with a shift function.

[0013] The first input port is connected to two row cache units, the second input port is connected to the weight control unit, and the first input port is also directly connected to the single output channel convolution calculation unit; the outputs of the two row cache units and the weight control unit are simultaneously connected to several single output channel convolution calculation units; another output of the weight control unit is connected to the second output port, the output of the single output channel convolution calculation unit is connected to the activation and quantization unit through the serial output unit, and the output of the activation and quantization unit is connected to the first output port.

[0014] The first input port inputs pooling data and pooling data valid signals, the second input port inputs write enable, read enable, weight address and write data; the first output port outputs convolution data and convolution data valid signals, the second output port outputs read data and read data valid signals;

[0015] Before receiving the multi-channel data stream input in channel order or out of order serially through the row data cache unit, this ordinary convolution computing hardware architecture will first judge the data stream and input the data stream into the row data cache unit according to the judgment result; the weight control unit synchronously outputs the 3x3 convolution kernel weight data of the corresponding channel, and the three-level pipeline multiplication and accumulation unit in the single output channel convolution calculation unit processes three consecutive rows of input data respectively, and realizes the parallel calculation of the 3x3 convolution kernel through the splicing weight distribution and overlapping data slicing mechanism. Finally, the full-precision intermediate results of the three processing units are point-added and accumulated, and the final output is formed after processing by the activation function and quantization unit.

[0016] Another object of the present invention is to provide a method for implementing ordinary convolution calculation based on data stream, the specific method comprising:

[0017] The first input port adopts a configurable bit width design of M*N bit / clk, where M is the granularity of the input data, and both M and N can be adjusted through parameterization. The input data stream controls the transmission timing through the data valid signal, allowing adjacent input channel data to be separated by one or more clock cycles. The weight control unit adopts a dual-mode data access mechanism, which not only supports off-chip reading and writing of traditional RAM interfaces, but also can achieve timing synchronization alignment of weight data and input channel data streams through internal control logic, ensuring that the 3x3 convolution kernel weights remain strictly synchronized with the input data during the continuous input of a single input channel data.

[0018] The three-stage pipeline multiplication unit in the three-stage pipeline multiplication and accumulation processing unit uses a three-stage pipeline structure to optimize the 3x3 convolution calculation. Each stage of the pipeline is configured with weights W1, W2, and W3 respectively. The input weight data Wrow is composed of the three weights {W3, W2, W1} of the convolution kernel in a single row, that is, it is assigned to {W3, W2, W1}, where W1 is a low N w Bit weight, W2 is the middle N w Bit weight, W3 is high N wBit weight. In this three-stage pipeline multiplication unit, assuming that the three-stage registers are R1, R2, and R3, the excitation input In1 of the first-stage pipeline multiplication unit is assigned to {R1[(M-1)*N-1:0], R2[M*N-1:(M-1)*N]}, the excitation input In2 of the second-stage pipeline multiplication unit is assigned to R2, and the excitation input In3 of the third-stage pipeline multiplication unit is assigned to {R2[N-1:0], R3[M*N-1:N]}. Among them, the weight input We1 of the first-stage pipeline multiplication unit is assigned to W1, the weight input We2 of the second-stage pipeline multiplication unit is assigned to W2, and the weight input We3 of the third-stage pipeline multiplication unit is assigned to W3. The sliding of the row-wise data window is realized through registers R1-R3, where M is the granularity of the input data and N is the number of bits of the input data; the problem of data overlap at the boundary during the convolution operation windowing process is effectively avoided through the accumulation unit.

[0019] The three-stage pipeline multiplication unit performs the signed point multiplication operation and uses the bit width expansion technology to process the intermediate results. The final output bit width is M*(N w After the three three-stage pipeline multiplication and accumulation processing units complete the calculation and feedback the Rdy signal, the system triggers cross-unit point addition accumulation to generate the full-precision convolution calculation result of the output channel.

[0020] In terms of multi-output channel expansion, this general convolution computing hardware architecture achieves output channel expansion by parallel instantiating multiple single-output channel convolution computing units, and manages data transmission timing through a unified control interface of the serial output unit. It adopts a row data multiplexing mechanism and supports prefetching data from the row data cache unit for real-time pipeline calculation, avoiding unnecessary data access and transmission delays. The data multiplexing mechanism is optimized to implement intelligent prefetching of overlapping calculation window areas in the row data cache unit, significantly reducing data access latency.

[0021] The hardware implementation of this general convolution computing hardware architecture uses a parameterized design, supporting flexible configuration of the number of input and output channels. The zero padding and stride parameters are fixed at 1. The weight storage unit optimizes the storage structure for the characteristics of a 3x3 convolution kernel. For a convolution kernel size of 3, the number of input channels is A, and the number of output channels is B, where A and B are both positive integers. The weight parameter size is 3 × 3 × A × B, effectively improving weight access efficiency.

[0022] Furthermore, the parameterized design method package is one of window sliding, scale transformation or cluster analysis.

[0023] In terms of timing control, this conventional convolution hardware architecture innovatively employs a distributed ready signal mechanism. Each three-stage pipeline multiplication unit generates an Rdy signal after completing processing a single row of multiple input channel data in the current computation window. When all three three-stage pipeline multiplication units are simultaneously ready, the result accumulation operation is triggered. This mechanism, combined with a global state machine, enables precise pipeline control, ensuring strict alignment of all stages of the 3x3 convolution computation.

[0024] The quantization unit in this standard convolutional computing hardware architecture utilizes shifting and data truncation techniques to support flexible conversion from full-precision results to the target bit width. Rounding strategies are optimized for the specific characteristics of 3x3 convolutions, minimizing hardware overhead while maintaining computational accuracy. The activation function unit is embedded within the quantization processing unit, enabling activation before quantization, ensuring consistent excitation data transmission across different computational layers.

[0025] In terms of hardware resource optimization, dedicated data paths are designed to maximize computing efficiency through weight reuse, data slicing, and pipeline bubble elimination.

[0026] The beneficial effects of the present invention are:

[0027] The present invention proposes a data stream-based general convolution computing hardware architecture and its implementation method. The architecture is constructed through a 3x3 convolution kernel and realizes real-time and efficient multiplication and accumulation calculations on data streams input channel by channel and row by row through time-division multiplexing of three-stage pipelined multiplication and accumulation units. It also avoids the problem of overlap (data stacking at the edges during the convolution windowing process) and enhances the dynamic adaptability of the computing architecture by designing configurable bit widths for the input ports. At the same time, through parallel calculation of multiple single-output channel convolution computing units, the computing efficiency is maximized, ultimately adapting to the computing requirements of different input and output channels of the convolution layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 This is a block diagram of the hardware architecture of a common convolution calculation based on row data stream provided in the first embodiment of the present invention;

[0030] Figure 2 This is a block diagram of another implementation of a common convolution computing hardware architecture based on row data stream provided in the second embodiment of the present invention;

[0031] Figure 3This is a schematic diagram of the calculation principle of a single three-stage pipelined multiplication-accumulation unit within a single-output channel convolution calculation unit in a common convolution calculation hardware architecture based on row data flow provided in the third embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of the working process of the internal cache control unit of the three-stage pipeline multiplication and accumulation unit in the ordinary convolution computing hardware architecture based on row data flow provided in the fourth embodiment of the present invention. DETAILED DESCRIPTION

[0033] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0034] Example 1

[0035] This embodiment provides a common convolution computing hardware architecture based on row data flow, as shown in the attached Figure 1 The architecture includes a first input port, a second input port, a first output port, a second output port, two row data cache units, a weight control unit, 16 single-output channel convolution calculation units, a serial output unit, and an activation and quantization unit;

[0036] The weight control unit includes a RAM unit for storing the weight data required for ordinary convolution calculation and a write and read control logic unit corresponding to the unit; each single-output channel convolution calculation unit includes three three-stage pipeline multiplication and accumulation units, wherein each three-stage pipeline multiplication and accumulation unit includes a three-stage pipeline multiplication unit, an accumulation unit and a cache control unit; the activation and quantization unit includes an adder, a multiplier, a binary symbol number judgment logic unit corresponding to the adder and the multiplier, and a three-stage pipeline structure unit with a shift function.

[0037] The first input port is connected to two row cache units, the second input port is connected to the weight control unit, and the first input port is also directly connected to the single output channel convolution calculation unit; the outputs of the two row cache units and the weight control unit are simultaneously connected to several single output channel convolution calculation units; another output of the weight control unit is connected to the second output port, the output of the single output channel convolution calculation unit is connected to the activation and quantization unit through the serial output unit, and the output of the activation and quantization unit is connected to the first output port.

[0038] The first input port inputs pooling data and pooling data valid signals, and the second input port inputs write enable, read enable, weight address and write data; the first output port outputs convolution data and convolution data valid signals, and the second output port outputs read data and read data valid signals.

[0039] Example 2

[0040] This embodiment provides another implementation of a common convolution computing hardware architecture based on data flow, such as Figure 2 shown.

[0041] The architecture includes a first input port, a second input port, a first output port, a second output port, two row data cache units, a weight control unit, 16 single-output channel convolution calculation units, 16 activation and quantization units, 16 single-port register files, and a serial output unit;

[0042] The weight control unit includes a RAM unit for storing the weight data required for ordinary convolution calculation and the write and read control logic corresponding to the unit; each of the 16 single-output channel convolution calculation units includes three three-stage pipeline multiplication and accumulation units, wherein the three-stage pipeline multiplication and accumulation unit includes a three-stage pipeline multiplication unit, an accumulation unit and a cache control unit; the activation and quantization unit includes an adder, a multiplier, a binary symbol number judgment logic unit corresponding to the adder and the multiplier, and a three-stage pipeline structure unit with a shift function.

[0043] Among them, the first input port is simultaneously connected to two row data cache units and a single-output channel convolution calculation unit, and the second input port is connected to the weight control unit; the outputs of the two row data cache units and the weight control unit are simultaneously connected to 16 single-output channel convolution calculation units; the outputs of the 16 single-output channel convolution calculation units are correspondingly connected to 16 activation and quantization units; the outputs of the 16 activation and quantization units are correspondingly connected to 16 single-port register files; the outputs of the 16 single-port register files are simultaneously connected to the serial output unit, and the output of the serial output unit is connected to the first output port.

[0044] The first input port inputs pooling data and pooling data valid signals, and the second input port inputs write enable, read enable, weight address and write data; the first output port outputs convolution data and convolution data valid signals, and the second output port outputs read data and read data valid signals.

[0045] Example 3

[0046] This embodiment provides a method for implementing common convolution calculation based on data streams. The method is implemented based on the common convolution calculation hardware architecture based on data streams provided in the first embodiment. The method specifically includes:

[0047] The ordinary convolution calculation architecture performs ordinary convolution calculation on the feature map of 4input_channel*1920*1080*8bit with an input channel of 4, an output channel of 16, a convolution kernel size of 3×3, a stride of 1, and a "zero padding" of 1. The output feature map size is 16output_channel×1920*1080*8bit.

[0048] The input multi-channel data pool_data is 64 bits / clk, supplemented by a pool_data_vld signal. A row of single-channel data consists of 240 sequentially continuous pool_data signals, meaning that the corresponding pool_data_vld signal is high for 240 clks. Since the input feature image consists of four channels, the valid data signal for a single row contains four high signals that last for 240 clks, with each high signal separated by a single clk.

[0049] Since the step of the implemented ordinary convolution calculation is 1, the architecture will judge whether the input data is the first row of data. If it is the first row of data, it will be broadcast to 16 single-output channel convolution calculation units and input into the row data cache unit 2 at the same time. At this time, the row data input into the single output channel convolution calculation unit will be input into the second three-stage pipeline multiplication and accumulation unit PE2 in the single single output channel convolution calculation unit and the convolution kernel {W3, W2, W1} in the second row for pipeline multiplication and accumulation calculation. After the calculation is completed, the second three-stage pipeline multiplication and accumulation inside the single output channel convolution calculation unit will be input into the second three-stage pipeline multiplication and accumulation unit PE2 in the single output channel convolution calculation unit. Unit PE2 will generate a data ready signal Rdy2. After Rdy2 is ready, the single single-output channel convolution calculation unit will read the first row of data in the row data cache unit 2, and perform pipeline multiplication and accumulation calculation on it with the first three-stage pipeline multiplication and accumulation unit PE1 in the single single-output channel convolution calculation unit and the first row of convolution kernel {W3, W2, W1}. After the calculation is completed, the first three-stage pipeline multiplication and accumulation unit PE1 inside the single output channel convolution calculation unit will generate a data ready signal Rdy1. After Rdy1 is ready, the single output channel convolution calculation unit waits for the arrival of the next row of data;

[0050] When the second row of data is input into the architecture, it will be broadcast to the 16 single-output channel convolution calculation units and input to the row data cache unit 1 at the same time. At this time, the row data input to the single output channel convolution calculation unit will be input to the third three-stage pipeline multiplication and accumulation unit PE3 in the single single output channel convolution calculation unit and perform pipeline multiplication and accumulation calculation with the convolution kernel {W3, W2, W1} in the third row. After the calculation is completed, the third three-stage pipeline multiplication and accumulation unit PE3 inside the single output channel convolution calculation unit will generate a data ready signal Rdy3. After Rdy3 is ready, The full-precision calculation results in the internal cache of the second three-stage pipeline multiplication and accumulation unit PE2 and the third three-stage pipeline multiplication and accumulation unit PE3 are read out one by one and accumulated in a point-to-point manner. The accumulated result is the full-precision calculation result of the first row of single output channels, which is then input into the subsequent activation and quantization units. At the same time, since the Rdy3 signal is ready, the single output channel convolution calculation unit reads the data in the row data cache unit 1. It should be noted that the data in the row data cache unit 1 is transferred to the row data cache unit 2 at the same time as it is transferred to the single output channel convolution calculation unit. The data in the row data cache unit 1 is input into the second three-stage pipeline multiplication and accumulation unit PE2 in the single single output channel convolution calculation unit and performs pipeline multiplication and accumulation calculation with the convolution kernel {W3, W2, W1} in the second row. After the calculation is completed, the second three-stage pipeline multiplication and accumulation unit PE2 in the single output channel convolution calculation unit generates a data ready signal Rdy2. After Rdy2 is ready, the single output channel convolution calculation unit waits for the arrival of the next row of data.

[0051] When the third row of data and all the remaining 1077 rows of data are input into the architecture, they will be broadcast to the 16 single-output channel convolution calculation units and input to the row data cache unit 1 at the same time. At this time, the row data input to the single output channel convolution calculation unit will be input to the third three-stage pipeline multiplication and accumulation unit PE3 in the single single output channel convolution calculation unit and perform pipeline multiplication and accumulation calculation with the convolution kernel {W3, W2, W1} of the third row. After the calculation is completed, the third three-stage pipeline multiplication and accumulation unit PE3 inside the single output channel convolution calculation unit will generate a data ready signal Rdy3. After Rdy3 is ready, the full-precision calculation results of the internal cache of the first three-stage pipeline multiplication and accumulation unit PE1, the second three-stage pipeline multiplication and accumulation unit PE2 and the third three-stage pipeline multiplication and accumulation unit PE3 will be read out one by one, and point-addition accumulation will be performed. The accumulated result That is the full-precision calculation result of the second row of single output channels, and then the result is input into the subsequent activation and quantization units. At the same time, since the Rdy3 signal is ready, the single output channel convolution calculation unit will read the data in the row data cache unit 2, and input the data into the first three-stage pipeline multiplication and accumulation unit PE1 in the single single output channel convolution calculation unit and perform pipeline multiplication and accumulation calculation with the convolution kernel {W3, W2, W1} in the first row. After the calculation is completed, the first three-stage pipeline multiplication and accumulation unit PE1 inside the single output channel convolution calculation unit will generate a data ready signal Rdy1. After Rdy1 is ready, the single output channel convolution calculation unit will read the data in the row data cache unit 1. It should be noted that the data in the row data cache unit 1 is transmitted to the single output channel convolution calculation unit at the same time as it is transmitted to the row data cache unit 2. The data in the row data cache unit 1 is input into the second three-stage pipeline multiplication and accumulation unit PE2 in the single single-output channel convolution calculation unit for pipeline multiplication and accumulation calculation with {W3, W2, W1} of the second row of the convolution kernel. After the calculation is completed, the first three-stage pipeline multiplication and accumulation unit PE1 inside the single output channel convolution calculation unit will generate a data ready signal Rdy2. After Rdy2 is ready, the single output channel convolution calculation unit waits for the arrival of the next row of data.

[0052] According to the above operations, the architecture can obtain the single output channel calculation results of all rows by repeating this process. However, since the input row data will be broadcast to 16 single output channel convolution calculation units at the same time, and each single output channel convolution calculation unit is independent of each other, that is, parallel calculation, when the full-precision convolution calculation results of each output channel of a single row are ready, the architecture will read out the full-precision calculation results of each output channel of a single row in serial order through the serial output unit in the form of output channel by output channel to the activation and quantization unit. After processing by the activation and quantization unit, the full-precision data is quantized into 8-bit data and transmitted to the subsequent network operation layer (such as the Pooling layer).

[0053] Among them, the calculation principle of the three-stage pipeline multiplication and accumulation unit is as follows Figure 3 As shown in the figure, this calculation principle avoids the problem of overlap at the edges of convolution calculations of continuous input data streams.

[0054] The specific method is as follows: for a continuously input 64-bit data stream, a total of 240 64-bit data streams need to be continuously input to form a complete single-line single-input channel data. Figure 3 The registers data_local_p1, data_local_p2, and data_local_p3 in the pipeline are all 64 bits, that is, a three-stage pipeline. When implementing dot multiplication calculation with the help of this three-stage pipeline, the three registers data_local_p1, data_local_p2, and data_local_p3 need to be appropriately converted into multipliers Multiplier_1, Multiplier_2, and Multiplier_3.

[0055] When assigning values ​​to Multiplier_1, Multiplier_2, and Multiplier_3:

[0056] Multiplier_1 is assigned the value {data_local_p1[55:0],data_local_p2[63:56]};

[0057] Multiplier_2 is assigned the value {data_local_p2};

[0058] Multiplier_3 is assigned the value {data_local_p1[7:0],data_local_p2[63:8]};

[0059] Multiplier_1 and Perform a signed dot product ( For The weight of the first column of the corresponding row),

[0060] Multiplier_2 and Perform a signed dot product ( For The second column weight of the corresponding row),

[0061] Multiplier_3 and Perform a signed dot product ( For The third column weight of the corresponding row),

[0062] The products of the three are added in a signed manner to obtain the full-precision result of the continuous input 64-bit stimulus data after single-row convolution calculation.

[0063] That is, assuming the accuracy of the weight is N w bit, then the continuous 8*8bit data input is processed by the 3-stage pipeline multiplication and accumulation unit, and the output of each stage is a continuous bit width of 8*(8+N w )bit intermediate result, the full-precision result of each level of point multiplication is accumulated by point addition to obtain 8*(10+N w )bit full-precision result. In addition, the 3-stage pipeline multiplication and accumulation unit of each single-output channel convolution calculation unit contains not only the 3-stage pipeline multiplication and accumulation unit, but also the cache control unit.

[0064] Its working principle and process are as follows Figure 4 As shown in the figure: when executing the convolution calculation for the feature map of 4input_channel*1920*1080*8bit, the output channel is 16, the convolution kernel size is 3×3, the stride is 1, and the "zero padding" is 1, the data of a single input channel consists of 240 64-bit, and the 240 64-bit are continuously input to the single output convolution calculation unit. When the data stream of the first input channel passes through the three-stage pipeline multiplication and accumulation unit, a continuous bit width of 8*(10+N w )bit full-precision result R1, the cache control unit inside the single-output channel convolution calculation unit will receive and directly store it. This stage is called "stage 1";

[0065] When the data stream of the second input channel passes through the pipeline multiplication and accumulation unit, the continuous bit width is 8*(10+N w ) bit full-precision result R2, while R2 is continuously input, the cache control unit will take R1 and R2 from the cache and perform signed point addition accumulation. The accumulated result R1 + R2 will be stored in the address where R1 was originally stored in the cache. This stage is called "stage 2";

[0066] When the data stream of the third input channel passes through the pipeline multiplication and accumulation unit, the continuous bit width is 8*(10+N w )-bit full-precision result R3, while R3 is continuously input, the cache control unit will take out R1+R2 and R3 from the cache and perform signed dot addition. The accumulated result R1+R2+R3 will be stored in the address where R1+R2 was originally stored in the cache. This stage is called "stage 3";

[0067] When the data stream of the fourth input channel passes through the three-stage pipeline multiplication and accumulation unit, the continuous bit width is 8*(10+N w )-bit full-precision result R4, while R4 is continuously input, the cache control unit will take out R1+R2+R3 from the cache and perform signed point addition on R4. The accumulated result R1+R2+R3+R4 will be stored in the address where R1+R2+R3 was originally stored in the cache. This stage is called "stage 4".

[0068] When PE has gone through these four stages of operation, its internal cache stores the full-precision results of the convolution calculation of the excitation of a single row of multiple input channels and the three columns of weights in the row direction of the convolution kernel corresponding to each input channel. At this time, the cache control unit will generate an Rdy signal, indicating that the calculation of the single-output channel convolution calculation unit has been completed and can be read at any time.

[0069] Example 4

[0070] This embodiment provides a method for implementing ordinary convolution calculation based on data stream, which is implemented based on the ordinary convolution calculation hardware architecture based on data stream provided in Example 2.

[0071] The ordinary convolution calculation architecture performs ordinary convolution calculation on the feature map of 4input_channel*1920*1080*8bit with an input channel of 4, an output channel of 16, a convolution kernel size of 3×3, a stride of 1, and a "zero padding" of 1. The output feature map size is 16output_channel×1920*1080*8bit.

[0072] The input multi-channel data pool_data is 64 bits / clk, supplemented by a pool_data_vld signal. A row of single-channel data consists of 240 sequentially continuous pool_data signals, meaning that the corresponding pool_data_vld signal is high for 240 clks. Since the input feature image consists of four channels, the valid data signal for a single row contains four high signals that last for 240 clks, with each high signal separated by a single clk.

[0073] Since the step of the implemented ordinary convolution calculation is 1, the architecture will judge whether the input data is the first row of data. If it is the first row of data, it will be broadcast to 16 single-output channel convolution calculation units and input into the row data cache unit 2 at the same time. At this time, the row data input into the single output channel convolution calculation unit will be input into the second three-stage pipeline multiplication and accumulation unit PE2 in the single single output channel convolution calculation unit and the convolution kernel {W3, W2, W1} in the second row for pipeline multiplication and accumulation calculation. After the calculation is completed, the second three-stage pipeline multiplication and accumulation inside the single output channel convolution calculation unit will be input into the second three-stage pipeline multiplication and accumulation unit PE2 in the single output channel convolution calculation unit. Unit PE2 will generate a data ready signal Rdy2. After Rdy2 is ready, the single single-output channel convolution calculation unit will read the first row of data in the row data cache unit 2, and perform pipeline multiplication and accumulation calculation on it with the first three-stage pipeline multiplication and accumulation unit PE1 in the single single-output channel convolution calculation unit and the first row of convolution kernel {W3, W2, W1}. After the calculation is completed, the first three-stage pipeline multiplication and accumulation unit PE1 inside the single output channel convolution calculation unit will generate a data ready signal Rdy1. After Rdy1 is ready, the single output channel convolution calculation unit waits for the arrival of the next row of data;

[0074] When the second row of data is input into the architecture, it will be broadcast to the 16 single-output channel convolution calculation units and input to the row data cache unit 1 at the same time. At this time, the row data input to the single output channel convolution calculation unit will be input to the third three-stage pipeline multiplication and accumulation unit PE3 in the single single output channel convolution calculation unit and perform pipeline multiplication and accumulation calculation with the convolution kernel {W3, W2, W1} in the third row. After the calculation is completed, the third three-stage pipeline multiplication and accumulation unit PE3 inside the single output channel convolution calculation unit will generate a data ready signal Rdy3. After Rdy3 is ready, The full-precision calculation results cached in the second three-stage pipeline multiplication and accumulation unit PE2 and the third three-stage pipeline multiplication and accumulation unit PE3 are read out one by one and accumulated in a point-to-point manner. The accumulated result is the full-precision calculation result of the first row of single output channels, which is then input into the subsequent activation and quantization units. At the same time, since the Rdy3 signal is ready, the single-output channel convolution calculation unit reads the data in the row data cache unit 1. It should be noted that the data in the row data cache unit 1 is transferred to the row data cache unit 2 at the same time as it is transferred to the single-output channel convolution calculation unit. The data in the row data cache unit 1 is input into the second three-stage pipeline multiplication and accumulation unit PE2 in the single single-output channel convolution calculation unit and performs pipeline multiplication and accumulation calculation with the convolution kernel {W3, W2, W1} in the second row. After the calculation is completed, the second three-stage pipeline multiplication and accumulation unit PE2 in the single output channel convolution calculation unit generates a data ready signal Rdy2. After Rdy2 is ready, the single output channel convolution calculation unit waits for the arrival of the next row of data.

[0075] When the third row of data and all the remaining 1077 rows of data are input into the architecture, they will be broadcast to the 16 single-output channel convolution calculation units and input to the row data cache unit 1 at the same time. At this time, the row data input to the single output channel convolution calculation unit will be input to the third three-stage pipeline multiplication and accumulation unit PE3 in the single single output channel convolution calculation unit and perform pipeline multiplication and accumulation calculation with the convolution kernel {W3, W2, W1} of the third row. After the calculation is completed, the third three-stage pipeline multiplication and accumulation unit PE3 inside the single output channel convolution calculation unit will generate a data ready signal Rdy3. After Rdy3 is ready, the full-precision calculation results of the internal cache of the first three-stage pipeline multiplication and accumulation unit PE1, the second three-stage pipeline multiplication and accumulation unit PE2 and the third three-stage pipeline multiplication and accumulation unit PE3 will be read out one by one, and point-addition accumulation will be performed. The accumulated result The result is the full-precision calculation result of the second row of single output channels, and then the result is input into the subsequent activation and quantization units. At the same time, since the Rdy3 signal is ready, the single output channel convolution calculation unit will read the data in the row data cache unit 2, and input the data into the first three-stage pipeline multiplication and accumulation unit PE1 in the single single output channel convolution calculation unit and perform pipeline multiplication and accumulation calculation with the convolution kernel {W3, W2, W1} in the first row. After the calculation is completed, the first three-stage pipeline multiplication and accumulation unit PE1 inside the single output channel convolution calculation unit will generate a data ready signal Rdy1. After Rdy1 is ready, the single output channel convolution calculation unit will read the data in the row data cache unit 1. It should be noted that the data in the row data cache unit 1 is transmitted to the single output channel convolution calculation unit at the same time as it is transmitted to the row data cache unit 2. The data in the row data cache unit 1 is input into the second three-stage pipeline multiplication and accumulation unit PE2 in the single single-output channel convolution calculation unit for pipeline multiplication and accumulation calculation with {W3, W2, W1} of the second row of the convolution kernel. After the calculation is completed, the first three-stage pipeline multiplication and accumulation unit PE1 inside the single output channel convolution calculation unit will generate a data ready signal Rdy2. After Rdy2 is ready, the single output channel convolution calculation unit waits for the arrival of the next row of data.

[0076] According to the above operations, by repeating this, the architecture can obtain the single output channel calculation results of all rows. However, since the input row data will be broadcast to 16 single output channel convolution calculation units at the same time, because each single output channel convolution calculation unit is independent of each other, that is, parallel calculation, when the full-precision convolution calculation results of each output channel of a single row are processed by their corresponding activation and quantization units and all input into their corresponding single-port register files, the architecture will read out the data inside the 16 single-port register files in sequence through the serial output unit, that is, transmit the single-row multi-output channel data channel by channel to the subsequent network operation layer (such as the Pooling layer).

[0077] Among them, the calculation principle of the three-stage pipeline multiplication and accumulation unit is as follows Figure 3 As shown in the figure, this calculation principle avoids the problem of overlap at the edges of convolution calculations of continuous input data streams.

[0078] The specific method is as follows: for a continuously input 64-bit data stream, a total of 240 64-bit data streams need to be continuously input to form a complete single-line single-input channel data. Figure 3 The registers data_local_p1, data_local_p2, and data_local_p3 in the pipeline are all 64 bits, that is, a three-stage pipeline. When implementing dot multiplication calculation with the help of this three-stage pipeline, the three registers data_local_p1, data_local_p2, and data_local_p3 need to be appropriately converted into multipliers Multiplier_1, Multiplier_2, and Multiplier_3.

[0079] When assigning values ​​to Multiplier_1, Multiplier_2, and Multiplier_3:

[0080] Multiplier_1 is assigned the value {data_local_p1[55:0],data_local_p2[63:56]};

[0081] Multiplier_2 is assigned the value {data_local_p2};

[0082] Multiplier_3 is assigned the value {data_local_p1[7:0],data_local_p2[63:8]};

[0083] Multiplier_1 and Perform a signed dot product ( For The weight of the first column of the corresponding row),

[0084] Multiplier_2 and Perform a signed dot product ( For The second column weight of the corresponding row),

[0085] Multiplier_3 and Perform a signed dot product ( For The third column weight of the corresponding row),

[0086] The products of the three are added in a signed manner to obtain the full-precision result of the continuous input 64-bit stimulus data after single-row convolution calculation.

[0087] That is, assuming the accuracy of the weight is bit, then the continuous 8*8bit data input is processed by the three-stage pipeline multiplication and accumulation unit, and the output of each stage is a continuous bit width of 8*(10+N w )bit intermediate result, the full-precision result of each level of point multiplication is accumulated by point addition to obtain 8*(10+N w )bit full precision result. In addition, each three-stage pipeline multiplication and accumulation unit contains a cache control unit in addition to the three-stage pipeline multiplication unit and accumulation unit. Its working principle and process are as follows Figure 4 As shown in the figure: when executing the convolution calculation for the feature map of 4input_channel*1920*1080*8bit, the output channel is 16, the convolution kernel size is 3×3, the stride is 1, and the "zero padding" is 1, the data of a single input channel consists of 240 64-bit, and the 240 64-bit are continuously input to the single output convolution calculation unit. When the data stream of input channel 1 passes through the pipeline multiplication and accumulation unit, a continuous bit width of 8*(10+N w )bit full-precision result R1, the cache control unit inside the PE will receive it and directly store it. This stage is called "stage 1";

[0088] When the data stream of input channel 2 passes through the pipeline multiplication and accumulation unit, the continuous bit width is 8*(10+N w ) bit full-precision result R2, while R2 is continuously input, the cache control unit will take R1 and R2 from the cache and perform signed point addition accumulation. The accumulated result R1 + R2 will be stored in the address where R1 was originally stored in the cache. This stage is called "stage 2";

[0089] When the data stream of input channel 3 passes through the pipeline multiplication and accumulation unit, the continuous bit width is 8*(10+N w )-bit full-precision result R3, while R3 is continuously input, the cache control unit will take out R1+R2 and R3 from the cache and perform signed dot addition. The accumulated result R1+R2+R3 will be stored in the address where R1+R2 was originally stored in the cache. This stage is called "stage 3";

[0090] When the data stream of input channel 4 passes through the pipeline multiplication and accumulation unit, the following is obtained: Figure 4 The continuous bit width is 8*(10+Nw )-bit full-precision result R4, while R4 is continuously input, the cache control unit will take out R1+R2+R3 from the cache and perform signed point addition on R4. The accumulated result R1+R2+R3+R4 will be stored in the address where R1+R2+R3 was originally stored in the cache. This stage is called "stage 4".

[0091] When the single-output channel convolution calculation unit has gone through these four stages of operation, its internal cache stores the full-precision results of the convolution calculation of the excitation of a single row of multiple input channels and the three columns of weights in the row direction of the convolution kernel corresponding to each input channel. At this time, the cache control unit will generate an Rdy signal, indicating that the calculation of the single-output channel convolution calculation unit has been completed and can be read at any time.

[0092] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0093] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for implementing a common convolution computing hardware architecture based on row data flow, characterized in that: The architecture includes: a first input port, a second input port, a first output port, a second output port, two row data cache units, a weight control unit, a plurality of single output channel convolution calculation units, a serial output unit, and an activation and quantization unit; The weight control unit includes a RAM unit for storing weight data and a write and read control unit; each of the single-output channel convolution calculation units includes a three-stage pipeline multiplication and accumulation unit, wherein the three-stage pipeline multiplication and accumulation unit includes a three-stage pipeline multiplication unit, an accumulation unit and a cache control unit; the activation and quantization unit includes an adder, a multiplier, a binary symbol number judgment logic unit and a pipeline structure unit; The first input port is connected to two row buffer units, the second input port is connected to the weight control unit, and the first input port is also directly connected to the single-output channel convolution calculation unit; the outputs of the two row buffer units and the weight control unit are simultaneously connected to a plurality of single-output channel convolution calculation units; another output of the weight control unit is connected to the second output port, and the output of the single-output channel convolution calculation unit is connected to the activation and quantization unit through the serial output unit; the output of the activation and quantization unit is connected to the first output port; The first input port inputs pooled data and pooled data valid signals; the second input port inputs write enable, read enable, weight address and write data; the first output port outputs convolution data and convolution data valid signals; the second output port outputs read data and read data valid signals.

2. The method according to claim 1, characterized in that In the three-stage pipeline multiplication and accumulation processing unit: The convolution kernel size is 3×3, the number of input data channels is N1, the number of input data rows is N2, and the first row of input data is input in channel order or in random order, where the interval between every two input channel data inputs is 1 or greater than 1 clk; when the input of the N1 input channel data in the previous row is completed, the data of the N1 input channels in the next row are still input to the single output channel convolution calculation unit in this way.

3. The method according to claim 2, characterized in that In the three-stage pipeline multiplication and accumulation unit: Each level of the pipeline is configured with weights W1, W2 and W3 respectively, and the input weight data W row It is composed of three weights W1, W2 and W3 of a single row of convolution kernels, where W1 is a low N w Bit weight, W2 is the middle N w Bit weight, W3 is high N w bit weight; The three-level registers are R1, R2 and R3. The excitation input In1 of the first-level pipeline multiplication unit is assigned to {R1[(M-1)*N-1:0], R2[M*N-1:(M-1)*N]}, and the weight input We1 is assigned to W1; the excitation input In2 of the second-level pipeline multiplication unit is assigned to R2, and the weight input We2 is assigned to W2; the excitation input In3 of the third-level pipeline multiplication unit is assigned to {R2[N-1:0], R3[M*N-1:N]}, and the weight input We3 is assigned to W3; the sliding of the row-wise data window is realized through registers R1, R2 and R3; where M is the granularity of the input data and N is the number of bits of the input data; The accumulation unit performs point-addition accumulation on the three data processed by the three-stage pipeline multiplication units and stores the accumulation result in the cache control unit. After the three three-stage pipeline multiplication accumulation units in the single single-output channel convolution calculation unit have completed the calculation, the data stored in the cache control unit are sequentially taken out and accumulated in point-addition to obtain the convolution calculation result of the single single-output channel convolution calculation unit; In the cache control unit: When the data of the first input channel enters the three-stage pipeline multiplication and accumulation processing unit and obtains the calculation result Afterwards, Store in the cache control unit; When the data of the second input channel enters the three-stage pipeline multiplication and accumulation processing unit and obtains the calculation result After that, you need to Take it out from the cache and make and After adding, Store in the cache control unit; When the data of the Pth input channel enters the pipeline multiplication and accumulation processing unit and the calculation result rP is obtained, the Stored in the cache control unit, where P is the total number of input channels of input data.

4. The method according to claim 3, characterized in that The first input port adopts a configurable bit width design of M*N bit / clk, where M is the granularity of input data.

5. The method according to claim 4, characterized in that The weight control unit uses a dual-mode data access mechanism to simultaneously support off-chip reading and writing of the RAM interface and achieve timing synchronization alignment of weight data and channel data streams through internal control logic, ensuring that the convolution kernel weight is synchronized with the input data during the continuous input of a single input channel data; The weight control unit uses the RAM interface format to store the single weight precision N w The bit is written from the outside of the chip to the RAM IP core or RF IP core of the unit, and the data valid signal is supported to cooperate with the control signal of the data output of the row data cache unit to read the weight data in the RAM IP core or RF IP core to one or more single-output channel convolution calculation units, and supports the read enable signal in the RAM interface format to read the weight data in the RAM IP core or RF IP core to the outside of the chip, where N w is a positive integer; The convolution kernel size is 3, the number of input channels is A, and the number of output channels is B, where A and B are both positive integers, then the scale of the weight parameter is 3×3×A×B.

6. The method according to claim 5, characterized in that The zero padding and step parameters of the described architecture are fixed to 1.

7. The method according to claim 6, characterized in that The serial output unit manages the data transmission timing through a unified control interface; and pre-fetches data from the row data cache unit to perform real-time pipeline calculations through row data multiplexing.

8. The method according to claim 7, characterized in that In the activation and quantization unit, the quantization unit realizes the conversion of the full-precision result to the target bit width through shifting and data interception technology; The activation function unit is embedded in the quantization processing unit, which first activates and then quantizes to achieve data transmission with the same precision between different operation layers.

Images