A method for transmitting data streams of on-chip network operations based on channel-by-channel and row-by-row basis

Abstract

This invention discloses a method for transmitting data streams for on-chip network operations on a channel-by-channel and row-by-row basis, pertaining to the fields of integrated circuit design and artificial intelligence hardware acceleration. This method enables channel-by-channel transmission, caching, and computation of single rows of raw image data or feature map data inputted by an image sensor array. By time-division multiplexing the hardware computing units corresponding to different computational types within the network, it enables on-chip computation of traditional convolutional neural networks using channel-by-channel and row-by-row input data. This data stream transmission method provides an efficient, real-time solution for performing convolutional neural network computations based on raw data generated row by row by an image sensor.

Description

Technical Field

[0001] The present invention relates to a method for transmitting network-on-chip computing data streams on a channel-by-channel and row-by-row basis, and belongs to the field of integrated circuit design and artificial intelligence hardware acceleration. Background Art

[0002] With the rapid development of artificial intelligence (AI), convolutional neural networks (CNNs) are increasingly being used in fields such as image processing and computer vision. Traditional CNN algorithms typically require complete image data to be fed into a computing unit for processing. These algorithms typically use off-chip memory for data caching and repeatedly transfer data to complete convolution operations. This approach not only increases system power consumption but also reduces real-time performance due to data transmission delays, making it difficult to adapt to the urgent real-time image processing needs of areas such as autonomous driving and intelligent security.

[0003] In recent years, with advances in image sensor technology, on-chip intelligent processing architectures have become a research hotspot. Prior art attempts have been made to integrate computing units and image sensors into the same chip. Chinese Patent Publication No. CN113946538A proposes a convolutional layer fusion storage device and method based on a row cache mechanism. This method improves computational efficiency by partitioning large images into blocks using a layer fusion storage strategy for computation and using a row cache mechanism to reduce the repeated handling of overlapping image blocks. However, this method is limited by the data-dependent nature of traditional CNN algorithms and still requires the complete caching of multiple rows of image data before initiating convolutional computations. This processing approach significantly increases hardware resource consumption, especially in high-resolution image scenarios, where the conflict between row cache requirements and chip area and power consumption becomes increasingly prominent. Furthermore, the allocation of hardware resources for different computational layers (such as convolution, pooling, activation functions, and fully connected layers) in the network model has not been effectively addressed, resulting in poor resource utilization.

[0004] The current technical bottleneck is also reflected in the optimization of data paths. When processing row-by-row input data, traditional solutions usually need to build complex row and column control logic to manage the flow of data, which not only increases the complexity of circuit design, but also leads to increased critical path delays. Especially when processing multi-input channel feature maps, existing architectures often need to configure multiple computing units in parallel to meet throughput requirements. This extensive resource allocation method is in direct conflict with chip area constraints. Although the amount of calculation can be reduced by compressing data accuracy or simplifying the network structure, these methods will sacrifice model accuracy and it is difficult to meet the stringent requirements of algorithm performance in practical applications. How to achieve efficient utilization of hardware resources without losing computing accuracy has become a key problem restricting the development of on-chip intelligent processing technology.

[0005] In terms of real-time performance, existing solutions often use a post-processing model, waiting for a complete frame of data to be input before initiating the computational process. While this approach simplifies the control logic, it introduces an inherent latency of approximately 33ms (at 30fps), making it difficult to meet the demands of applications such as industrial inspection and robotic vision, which require extremely high real-time feedback. Furthermore, traditional data flow scheduling methods require multiple data transfers when processing cross-layer computations, resulting in a decrease in the effective computational time. As the depth of the network model increases, this cumulative latency significantly reduces the overall system response speed.

[0006] In addition, Chinese patent publication number CN118196437A discloses an image FHOG feature extraction device, which outputs data in a channel-by-channel, row-by-row, and column-by-column manner. Although this improves the utilization of the output bandwidth to a certain extent, this data transmission method requires traversing and reading the data rows in each channel and then switching the columns, which increases the complexity and time of data reading. In addition, the channel-by-channel, row-by-row, and column-by-column access method will lead to frequent row switching, making the data access mode mismatched with the storage mode, and increasing the time overhead of data access.

[0007] Therefore, developing an on-chip neural network computing data stream transmission method that can achieve deep pipelining of data input and calculation processes and minimize data transfer delays has become an important breakthrough in improving edge intelligent processing performance. Summary of the Invention

[0008] The object of the present invention is to provide a method for transmitting data streams of network-on-chip operations on a channel-by-channel and row-by-row basis. The method transmits data by first transmitting channel-by-channel and then row-by-row. That is, the data of each channel in the first row is transmitted first. After the data of each channel in the first row is transmitted, the data of each channel in the second row is transmitted; until all data transmission is completed.

[0009] The specific process of this method is:

[0010] For multi-channel original image data or feature map data, the single-line single-channel data transmitted by the data stream is , the number of channels is , each channel has If the data is transmitted in rows, the data stream transmission method is channel by channel and then row by row. 、 、 Until , that is, after all the data of multiple channels in the first row are transmitted in sequence, the data of multiple channels in the second row are transmitted. 、 、 Until , repeat this way until the transmission is completed Multiple channels of data 、 、 Until , that is, the multi-channel original image data or feature map data is transmitted channel by channel and then row by row.

[0011] Furthermore, for multiple computing layers of the on-chip network, the input data flow of each computing layer is in the form of first channel-by-channel and then row-by-row; the output data flow is also in the form of first channel-by-channel and then row-by-row.

[0012] Furthermore, for the activation function operation layer of the on-chip network, the ReLU activation function is generally used. That is, the input data is judged as positive or negative. When it is greater than 0, the original data is retained and output to the next operation layer (generally the Pooling layer); when it is less than or equal to 0, the original data is reset to zero and output to the next operation layer. The hardware architecture corresponding to the ReLU activation function operation is a first-level trigger. That is, when the input data comes in channel by channel and row by row, the result of the ReLU operation is obtained after passing through the judgment path and trigger. Therefore, for the on-chip activation function operation layer, the input data stream becomes the output data stream after passing through the first-level trigger. Therefore, the input data stream and the output data stream are transmitted first channel by channel and then row by row.

[0013] Furthermore, for the pooling operation layer of the on-chip network, its corresponding processing unit performs operations such as odd-row caching and even-row calculation on the input data stream. That is, the data stream input channel by channel of the odd-row passes through the processing unit and first enters the cache unit inside the calculation unit. The depth of the cache unit is the number of channels of the single-row input data. When the data stream input channel by channel of the even-row passes through the processing unit, the odd-row data of the corresponding channel is synchronously taken out from the cache unit, and the odd-row data and the even-row data of the same channel are average pooled or maximum pooled. The result of the calculation is used as the output of the processing unit. That is, the output data stream of the processing unit is also channel-by-channel and row-by-row.

[0014] Furthermore, for the fully connected operation layer of the on-chip network, its corresponding processing unit performs pipeline multiplication and accumulation operations on the input data stream, that is, the channel-by-channel and row-by-row data stream input by the upper-level processing unit directly enters a multi-level pipeline multiplication and accumulation tree structure whose number is equal to the number of fully connected calculation output units for parallel calculation, so that the intermediate full-precision calculation results can be accumulated and temporarily stored. When the data input of all input channels of all rows is completed, the final fully connected calculation result is obtained through time-division multiplexing of the multi-level pipeline multiplication and accumulation tree structure whose number is equal to the number of fully connected calculation output units. In addition, it should be mentioned that when the data stream input by multiple input channels enters the multi-level pipeline multiplication and accumulation tree structure, the weights corresponding to the data of a single input channel are also loaded synchronously in terms of timing, that is, the time when the weight arrives at the multi-level pipeline multiplication and accumulation tree structure and the time when the data of the single input channel corresponding to the weight arrives at the multi-level pipeline multiplication and accumulation tree structure are aligned in terms of timing.

[0015] Furthermore, for the convolutional operation layer of the on-chip network, its corresponding processing unit performs multiplication and addition calculations, caching, output, and overwriting operations on multiple rows of channel-by-channel input data streams. Specifically, for the input channel-by-channel and row-by-row data stream, the processing unit corresponding to the convolutional layer will parallelly input it into P (P is a positive integer) single-output channel ordinary convolution calculation units;

[0016] In a single single-output channel convolution calculation unit, when the first row of channel-by-channel input data flows in, it is convolved with the weight of the corresponding row to obtain the first row of intermediate results, which are temporarily stored in the first register R1, and the first cache unit RAM1 (hereinafter referred to as RAM1) is used to cache the first row of intermediate results temporarily stored in the first register R1;

[0017] When the second row of channel-by-channel input data stream comes in, it is pre-stored in the second row data buffer unit Buffer2 (hereinafter referred to as Buffer2), and at the same time, it is convolved with the weight of the corresponding row to obtain the second row of intermediate results, which are temporarily stored in the second register R2, and the second buffer unit RAM2 (hereinafter referred to as RAM2) is used to cache the second row of intermediate results temporarily stored in the second register R2;

[0018] When the third row of channel-by-channel input data flows in, it is pre-stored in the first row of data buffer unit Buffer1 (hereinafter referred to as Buffer1). At the same time, the multi-input channel data of the third row is convolved with the weights of the corresponding row to obtain the third row of intermediate results, which are temporarily stored in the third register R3.

[0019] Then, the first row of intermediate results and the second row of intermediate results are taken out from the first register R1 and the second register R2, and the first row of intermediate results, the second row of intermediate results and the third row of intermediate results are accumulated to obtain the final result Rout_Single of the single-output channel convolution calculation unit; when the final results Rout_Single of the P single-output channel convolution calculation units are all generated, they will be output channel by channel;

[0020] After the final result Rout_Single of the convolution calculation of a single row is output channel by channel, before waiting for the data input channel by channel in the fourth row to come in, the multi-input channel data of the second row in Buffer2 is taken out, and the new intermediate result obtained by convolving it with the weight of the first row is also temporarily stored in the first register R1, and the new intermediate result obtained by convolution in the first register R1 is cached in RAM1; the third row of data in Buffer1 is taken out, and the multi-input channel data of the original second row in Buffer2 is overwritten. After the data in Buffer1 is taken out, it is convolved with the weight of the second row to obtain a new intermediate result and temporarily stored in the second register R2, and the new intermediate result obtained by convolution in the second register R2 is temporarily stored. The intermediate results are cached in RAM2; when the first register R1 and the second register R2 are ready, wait for the data of the fourth row of multiple input channels to come in, repeat the above operation of "when the data of the third row of multiple input channels comes in", and obtain a new intermediate result and temporarily store it in the third register R3; accumulate the first row of intermediate results, the second row of intermediate results and the third row of intermediate results temporarily stored in the first register R1, the second register R2 and the third register R3 to obtain the final result Rout_Single of the single output channel convolution calculation unit; as described above, when the final results Rout_Single of P single output channel convolution calculation units are generated, the final results Rout_Single of multiple single output channel convolution calculation units are output channel by channel.

[0021] Since the P single-output channel ordinary convolution calculation units are independent of each other and calculated in parallel, when the final results Rout_Single of the P single-output channel convolution calculation units are generated, the processing unit corresponding to the convolution calculation will output the final results Rout_Single generated by the P single-output channel convolution calculation units channel by channel, while waiting for the arrival of the fourth row of channel-by-channel input data.

[0022] The beneficial effects of the present invention are as follows: the present invention provides a method for transmitting network-on-chip computing data streams based on a channel-by-channel and row-by-row basis. This method transmits data by first transmitting channel-by-channel and then row-by-row, thereby avoiding channel data congestion and improving transmission efficiency. Moreover, under this transmission mode, data has been preliminarily classified before entering storage or subsequent stages, thereby improving the efficiency of data processing and storage. It also improves the flexibility and scalability of transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 This is a schematic diagram of multi-channel original image data or feature map data suitable for a channel-by-channel and then row-by-row data stream transmission method based on an on-chip network operation data stream transmission method provided in the first embodiment of the present invention;

[0025] Figure 2 Schematic diagram of a method for transmitting data streams of network-on-chip operations on a channel-by-channel and row-by-row basis in a fully connected layer according to a second embodiment of the present invention;

[0026] Figure 3 Schematic diagram of a method for transmitting data streams of network-on-chip operations on a channel-by-channel and row-by-row basis in a pooling layer according to a third embodiment of the present invention;

[0027] Figure 4 This is a schematic diagram of implementing data transmission in a convolutional layer based on a channel-by-channel and row-by-row on-chip network operation data stream transmission method provided in the fourth embodiment of the present invention. DETAILED DESCRIPTION

[0028] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0029] Example 1

[0030] This embodiment provides a method for transmitting data streams of network-on-chip operations based on a channel-by-channel and row-by-row basis. Figure 1 , the method comprising:

[0031] For multi-channel original image data or feature map data, the single-line single-channel data transmitted by the data stream is , the number of channels is , each channel has If the data is transmitted in rows, the data stream transmission method is channel by channel and then row by row. 、 、 Until , that is, after all the data of multiple channels in the first row are transmitted in sequence, the data of multiple channels in the second row are transmitted. 、 、 Until , repeat this way until the transmission is completed Multiple channels of data 、 、 Until , which means that the multi-channel original image data or feature map data is transmitted channel by channel and then row by row.

[0032] For the multiple computing layers corresponding to the on-chip network, the input data flow of each computing layer is in the form of channel-by-channel and then row-by-row; the output data flow is also in the form of channel-by-channel and then row-by-row.

[0033] The activation function operation layer of the on-chip network generally uses the ReLU activation function. This function checks the positive or negative value of the input data. When the value is greater than 0, the original data is retained and output to the next operation layer (usually the Pooling layer); when the value is less than or equal to 0, the original data is reset to zero and output to the next operation layer. The hardware architecture corresponding to the ReLU activation function operation is a one-level trigger. That is, when the input data comes in channel by channel and row by row, the result of the ReLU operation is obtained after passing through the judgment path and trigger. Therefore, for the on-chip activation function operation layer, the input data stream becomes the output data stream after passing through the first-level trigger. Therefore, the input and output data streams are transmitted first channel by channel and then row by row.

[0034] For the average pooling or maximum pooling operation layer of the on-chip network, its corresponding processing unit performs operations such as odd-row caching and even-row calculation on the input data stream. That is, the data stream of odd-row input channel by channel passes through the processing unit and first enters the cache unit inside the calculation unit. The depth of the cache unit is the number of channels of the single-row input data. When the data stream of even-row input channel by channel passes through the processing unit, the odd-row data of the corresponding channel is synchronously taken out of the cache unit, and the odd-row data and even-row data of the same channel are average pooled or maximum pooled. The calculation result is used as the output of the processing unit. That is, the output data stream of the processing unit is also channel-by-channel and row-by-row.

[0035] For the fully connected layer operations of the on-chip network, its corresponding processing unit performs pipeline multiplication and accumulation operations on the input data stream. That is, the channel-by-channel and row-by-row data stream input by the upper-level processing unit directly enters a multi-level pipeline multiplication and accumulation tree structure with a number equal to the number of fully connected calculation output units for parallel calculation, so that the intermediate full-precision calculation results can be accumulated and temporarily stored. When the data input of all input channels of all rows is completed, the final fully connected calculation result is obtained through time-division multiplexing of the multi-level pipeline multiplication and accumulation tree structure with a number equal to the number of fully connected calculation output units. In addition, it should be mentioned that when the data stream input by multiple input channels enters the multi-level pipeline multiplication and accumulation tree structure, the weights corresponding to the data of a single input channel are also loaded synchronously in terms of timing. That is, the time when the weight arrives at the multi-level pipeline multiplication and accumulation tree structure and the time when the data of the single input channel corresponding to the weight arrives at the multi-level pipeline multiplication and accumulation tree structure are aligned in terms of timing.

[0036] For the convolutional operation layer of the on-chip network, its corresponding processing unit performs multiplication and addition calculations, caching, output, and overwriting operations on multiple rows of channel-by-channel input data streams. Specifically, for the input channel-by-channel and row-by-row data stream, the processing unit corresponding to the convolution calculation will parallelly input it to P (P is a positive integer) single-output channel ordinary convolution calculation units.

[0037] In a single single-output channel ordinary convolution calculation unit, when the first row of channel-by-channel input data flows in, it is convolved with the weight of the corresponding row to obtain the first intermediate result register R1, and the first cache unit RAM1 (hereinafter referred to as RAM1) is used to cache the first convolution intermediate result register R1 corresponding to the first row;

[0038] When the second row of channel-by-channel input data stream comes in, it is pre-stored in the second row data buffer unit Buffer2 (hereinafter referred to as Buffer2), and at the same time, it is convolved with the weight of the corresponding row to obtain the second row of intermediate results, which are temporarily stored in the second register R2, and the second buffer unit RAM2 (hereinafter referred to as RAM2) is used to cache the second row of intermediate results temporarily stored in the second register R2;

[0039] When the third row of channel-by-channel input data flows in, it is pre-stored in the first row data cache unit Buffer1 (hereinafter referred to as Buffer1), and at the same time, the data of the multiple input channels of the third row are convolved with the weights of the corresponding row to obtain the third row of intermediate results, which are temporarily stored in the third register R3. Subsequently, the data in the first register R1 and the second register R2 are taken out from their respective cache units, and the first row intermediate results, the second row intermediate results, and the third row intermediate results temporarily stored in the first register R1, the second register R2, and the third register R3 are accumulated to obtain the final result Rout_Single of the single output channel convolution calculation unit; when the final results Rout_Single of the P single output channel convolution calculation units are all generated, they will be output channel by channel;

[0040] Since the P single-output channel ordinary convolution calculation units are independent of each other and calculated in parallel, when the final results Rout_Single of the P single-output channel convolution calculation units are generated, the processing unit corresponding to the convolution calculation will output the final results Rout_Single generated by the P single-output channel convolution calculation units channel by channel, while waiting for the arrival of the fourth row of channel-by-channel input data.

[0041] Before the fourth row of channel-by-channel input data arrives, the multi-input channel data of the second row in Buffer1 is taken out and convolved with the weights of the first row to obtain a new intermediate result r1. The new intermediate result r1 obtained by convolution is cached using the first cache unit RAM1. The third row of data in Buffer2 is taken out, and the multi-input channel data originally in the second row of Buffer1 is overwritten. After the data in Buffer2 is taken out, it is convolved with the weights of the second row to obtain a new intermediate result r2. The new intermediate result r2 obtained by convolution is cached using the cache unit RAM2. When r1 and r2 are both ready, the processing unit corresponding to the convolution calculation waits for the data of the fourth row of multi-input channels to come in, repeats the above "operation when the data of the third row of multi-input channels comes in", obtains the intermediate result r3, and then outputs the final result Rout_Single of the multiple single-output channel convolution calculation units channel by channel.

[0042] Therefore, after the first register R1 and the second register R2 corresponding to the current two rows of data are ready, the processing unit corresponding to the convolution calculation waits for the arrival of the next row of multi-input channel data, repeats the above "operation when the third row of multi-input channel data comes in", and obtains the intermediate result r3, and then outputs the final result Rout_Single of multiple single-output channel convolution calculation units channel by channel.

[0043] Example 2

[0044] This embodiment provides a method for implementing on-chip network operation data stream transmission based on channel-by-channel and row-by-row in a fully connected operation layer. Figure 2 , the method comprising:

[0045] The upper left pixel array generates raw data row by row. After a single row of data is operated by the activation function, it is output through a single-stage pipeline without involving storage operations.

[0046] After a single row of data is pooled, the odd-numbered rows of data are temporarily stored first; for the even-numbered rows of data, they are pooled with the temporarily stored odd-numbered rows of data to generate a single row of pooled data and input it into the next convolution operation layer. The convolution operation layer needs to temporarily store two rows of data and complete the multiplication and accumulation calculation of each row with the corresponding weight. When waiting for the third row of data to arrive, after completing the multiplication and accumulation calculation with the corresponding weight, the results of the multiplication and accumulation calculation of the two rows with their respective weights are accumulated to obtain a new single row of data and input it into the next activation function operation layer.

[0047] That is, the data flow is circulated in the order of "convolution operation--activation function--pooling operation--convolution operation" until the last layer of pooling operation, and the single row of data generated by the pooling operation is input into the fully connected computing unit block by block. Figure 2 As shown, the bit width of a single input row is M*n*K bits, the bit width of a single data point is n bits, and the input data flow is M*n bits / clk, meaning that K clks are required to transmit a single row of data. The number M of characteristic data points inputted by a single clk corresponds to the number of excitation inputs to the multiplication-accumulation tree structure in a single fully-connected computation unit. Here, input data of granularity M is broadcasted in parallel to N multiplication-accumulation tree structures. Because a single multiplication-accumulation tree consists of multiple pipeline stages, a time-division multiplexing strategy is effectively employed for the multiplication-accumulation tree structure. After traversing all rows and channels of data of granularity M and inputting them into N pipelined multiplication-accumulation tree structures, the final register of these N parallel pipelined multiplication-accumulation trees contains the fully-connected computation result {Result_1, Result_2, ..., Result_N}.

[0048] Example 3

[0049] This embodiment provides a method for implementing on-chip network operation data stream transmission based on channel-by-channel and row-by-row in the pooling operation layer. The method is implemented through the pooling layer microarchitecture, such as Figure 3 As shown, the method includes:

[0050] The granularity of the input data is 8*8 bit / clk = 64 bit / clk, meaning eight 8-bit stimuli form a group of inputs. datain is the channel-by-channel and row-by-row input of the previous-level arithmetic unit. When the data of the first channel of an odd row is input into the microarchitecture, the two consecutive 64-bit inputs, forming a 128-bit data, are stored in the storage unit dprf_32x128_01. The next two consecutive 64-bit inputs, forming a 128-bit data, are then stored in the storage unit dprf_32x128_02. This process repeats until the last two consecutive 64-bit inputs of the first input channel of the odd row form a 128-bit data and are stored in dprf_32x128_n, completing the temporary storage of all data for the first input channel of the odd row. After the microarchitecture completes the temporary storage of all input channel data for the odd row in the same manner, it waits for the channel-by-channel input of the even row data.

[0051] The depth of each storage unit is the number of input channels for a single row of data. When the data of the first channel of an even row is input into the microarchitecture, the two 64-bit data of the first continuous input are combined into a 128-bit data. At the same time, the 128-bit data corresponding to address 0 is retrieved from dprf_32x128_01, which temporarily stores the multi-input channel data of the odd row. The two 128-bit data are input into the average pooling / maximum pooling calculation unit (the average pooling or maximum pooling operation is implemented using the combinational logic of the digital circuit), and a 64-bit pooling calculation result is obtained as the output of the calculation unit.

[0052] When the two consecutive 64-bit data are combined into a 128-bit data, the 128-bit data corresponding to address 0 needs to be taken out from dprf_32x128_02, which temporarily stores the odd-numbered rows of multi-input channel data. The two 128-bit data are input to the average pooling / maximum pooling calculation unit (the average pooling or maximum pooling operation is implemented using the combinational logic of the digital circuit), and a 64-bit pooling calculation result is obtained as the output of the calculation unit;

[0053] This process is repeated until the last two consecutive 64 bits of the first input channel of the even row are pieced together to form a 128-bit data. At the same time, the 128-bit data corresponding to address 0 is taken out from the dprf_32x128_n that temporarily stores the multi-input channel data of the odd row, and the two 128-bit data are input into the average pooling / maximum pooling calculation unit (the average pooling or maximum pooling operation is implemented using the combinational logic of the digital circuit), and a 64-bit pooling calculation result is obtained as the output of the calculation unit. At this time, the pooling calculation of the first input channel data of the two rows of data has been completed, and the calculation result has been output to the next operation layer; when the second input channel of the even row and the data of all the remaining input channels are input into the microarchitecture, the operations performed are the same as those performed when the first input channel data of the even row is input into the microarchitecture. The only difference is that the address corresponding to the data taken out from the storage unit dprf increases with the increase of the input channel of the even row. Furthermore, when new odd-numbered rows of data are input into this microarchitecture, all data within dp_rf is overwritten. New even-numbered rows of data are input into this microarchitecture, and the same operations are performed on the previous even-numbered rows of data. Therefore, in summary, the input data of this pooling unit microarchitecture is channel-by-channel and row-by-row, and the output data is also channel-by-channel and row-by-row.

[0054] Example 4

[0055] This embodiment provides a method for implementing on-chip network operation data stream transmission based on channel-by-channel and row-by-row in the convolution operation layer. The method is implemented based on the common convolution computing hardware architecture. The architecture is as follows: Figure 4 As shown, it includes a first input port, a second input port, a first output port, a second output port, two row data cache units, a weight control unit, 16 single-output channel convolution calculation units, 16 activation and quantization units, 16 single-port register files and a serial output unit; each single-output channel convolution calculation unit includes three three-stage pipeline multiplication and accumulation units, and the three-stage pipeline multiplication and accumulation unit includes a three-stage pipeline multiplication unit, an accumulation unit and a cache control unit; the activation and quantization unit includes an adder, a multiplier, a binary symbol number judgment logic unit corresponding to the adder and the multiplier, and a multi-stage pipeline structure unit with a shift function.

[0056] The method includes:

[0057] This ordinary convolution calculation architecture performs ordinary convolution calculation on the feature map of 4input_channel×1920*1080*8bit with an input channel of 4, an output channel of 16, a convolution kernel size of 3×3, a stride of 1, and a "zero padding" of 1. The output feature map size is 16output_channel×1920*1080*8bit.

[0058] The input multi-channel data pool_data is 64 bits / clk, supplemented by a pool_data_vld signal. A row of single-channel data consists of 240 sequentially continuous pool_data signals, meaning that the corresponding pool_data_vld signal is high for 240 clks. Since the input feature image consists of four channels, the valid data signal for a single row contains four high signals that last for 240 clks, with each high signal separated by a single clk.

[0059] Since the step of the implemented ordinary convolution calculation is 1, the architecture will judge whether the input data is the first row of data. If it is the first row of data, it will be broadcast to 16 single-output channel convolution calculation units and input into the row data cache unit 2 at the same time. At this time, the row data input into the single output channel convolution calculation unit will be input into the second three-stage pipeline multiplication and accumulation unit PE2 in the single single output channel convolution calculation unit and the convolution kernel {W3, W2, W1} in the second row for pipeline multiplication and accumulation calculation. After the calculation is completed, the second three-stage pipeline multiplication and accumulation inside the single output channel convolution calculation unit will be input into the second three-stage pipeline multiplication and accumulation unit PE2 in the single output channel convolution calculation unit. Unit PE2 will generate a data ready signal Rdy2. After Rdy2 is ready, the single single-output channel convolution calculation unit will read the first row of data in the row data cache unit 2, and perform pipeline multiplication and accumulation calculation on it with the first three-stage pipeline multiplication and accumulation unit PE1 in the single single-output channel convolution calculation unit and the first row of convolution kernel {W3, W2, W1}. After the calculation is completed, the first three-stage pipeline multiplication and accumulation unit PE1 inside the single output channel convolution calculation unit will generate a data ready signal Rdy1. After Rdy1 is ready, the single output channel convolution calculation unit waits for the arrival of the next row of data;

[0060] When the second row of data is input into the architecture, it will be broadcast to the 16 single-output channel convolution calculation units and input to the row data cache unit 1 at the same time. At this time, the row data input to the single output channel convolution calculation unit will be input to the third three-stage pipeline multiplication and accumulation unit PE3 in the single single output channel convolution calculation unit and perform pipeline multiplication and accumulation calculation with the convolution kernel {W3, W2, W1} in the third row. After the calculation is completed, the third three-stage pipeline multiplication and accumulation unit PE3 inside the single output channel convolution calculation unit will generate a data ready signal Rdy3. After Rdy3 is ready, the first The full-precision calculation results of the internal cache of the second three-stage pipeline multiplication and accumulation unit PE2 and the third three-stage pipeline multiplication and accumulation unit PE3 are read out one by one and accumulated in a point-by-point manner. The accumulated result is the full-precision calculation result of the first row of single output channels, and then the result is input to the subsequent activation and quantization units. At the same time, since the Rdy3 signal is ready, the single output channel convolution calculation unit will read the data in the row data cache unit 1. It should be noted that the data in the row data cache unit 1 is transmitted to the single output channel convolution calculation unit at the same time as it is transmitted to the row data cache unit 2. The data in the row data cache unit 1 is input into the second three-stage pipeline multiplication and accumulation unit PE2 in the single single-output channel convolution calculation unit for pipeline multiplication and accumulation calculation with the convolution kernel {W3, W2, W1} in the second row. After the calculation is completed, the second three-stage pipeline multiplication and accumulation unit PE2 inside the single output channel convolution calculation unit will generate a data ready signal Rdy2. After Rdy2 is ready, the single output channel convolution calculation unit waits for the arrival of the next row of data;

[0061] When the third row of data and all the remaining 1077 rows of data are input into the architecture, they will be broadcast to the 16 single-output channel convolution calculation units and input to the row data cache unit 1 at the same time. At this time, the row data input to the single output channel convolution calculation unit will be input to the third three-stage pipeline multiplication and accumulation unit PE3 in the single single output channel convolution calculation unit and perform pipeline multiplication and accumulation calculation with the convolution kernel {W3, W2, W1} of the third row. After the calculation is completed, the third three-stage pipeline multiplication and accumulation unit PE3 inside the single output channel convolution calculation unit will generate a data ready signal Rdy3. After Rdy3 is ready, the full-precision calculation results of the internal cache of the first three-stage pipeline multiplication and accumulation unit PE1, the second three-stage pipeline multiplication and accumulation unit PE2 and the third three-stage pipeline multiplication and accumulation unit PE3 will be read out one by one, and point-addition accumulation will be performed. The accumulated result That is the full-precision calculation result of the second row of single output channels, and then the result is input into the subsequent activation and quantization units. At the same time, since the Rdy3 signal is ready, the single output channel convolution calculation unit will read the data in the row data cache unit 2, and input the data into the first three-stage pipeline multiplication and accumulation unit PE1 in the single single output channel convolution calculation unit and perform pipeline multiplication and accumulation calculation with the convolution kernel {W3, W2, W1} in the first row. After the calculation is completed, the first three-stage pipeline multiplication and accumulation unit PE1 inside the single output channel convolution calculation unit will generate a data ready signal Rdy1. After Rdy1 is ready, the single output channel convolution calculation unit will read the data in the row data cache unit 1. It should be noted that the data in the row data cache unit 1 is transmitted to the single output channel convolution calculation unit at the same time as it is transmitted to the row data cache unit 2. The data in the row data cache unit 1 is input into the second three-stage pipeline multiplication and accumulation unit PE2 in the single single-output channel convolution calculation unit for pipeline multiplication and accumulation calculation with {W3, W2, W1} of the second row of the convolution kernel. After the calculation is completed, the first three-stage pipeline multiplication and accumulation unit PE1 inside the single output channel convolution calculation unit will generate a data ready signal Rdy2. After Rdy2 is ready, the single output channel convolution calculation unit waits for the arrival of the next row of data.

[0062] According to the above operations, by repeating this, the architecture can obtain the single output channel calculation results of all rows. However, since the input row data will be broadcast to 16 single output channel convolution calculation units at the same time, and each single output channel convolution calculation unit is independent of each other, that is, parallel calculation, when the full-precision convolution calculation results of each output channel of a single row are processed by their corresponding activation and quantization units and all input into their corresponding single-port register files, the architecture will read out the data in the 16 single-port register files in sequence, that is, transmit the multi-output channel data of a single row channel by channel to the subsequent network operation layer (such as the Pooling layer).

[0063] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0064] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for transmitting data streams of network-on-chip computing operations on a channel-by-channel and row-by-row basis, characterized in that: The method includes: for the convolution operation layer and the pooling operation layer of the on-chip network, the input data stream is generated by the upper-level network operation layer, and the data stream input method is first channel by channel and then row by row; the output data stream is generated by the current-level network operation layer, and the data stream output method is also first channel by channel and then row by row; For the activation function operation layer of the on-chip network, the input data stream is only pipelined, that is, the data stream input channel by channel and row by row is directly output after pipeline processing by the activation function operation layer, that is, the output data stream is output in the same way as the input data stream; For the fully connected operation layer of the on-chip network, its input data stream performs pipeline multiplication and accumulation operations, that is, the channel-by-channel and row-by-row data stream input by the upper processing unit directly enters the multi-level pipeline multiplication and accumulation tree structure whose number is equal to the number of fully connected calculation output units for parallel calculation, and the intermediate full-precision calculation results are accumulated and temporarily stored. When the data input of all input channels of all rows is completed, the data is time-division multiplexed through the multi-level pipeline multiplication and accumulation tree structure whose number is equal to the number of fully connected calculation output units to obtain the final fully connected calculation result; when the data stream input by multiple input channels enters the multi-level pipeline multiplication and accumulation tree structure, the weight corresponding to the data of a single input channel is also loaded synchronously in terms of timing, that is, the time when the weight arrives at the multi-level pipeline multiplication and accumulation tree structure and the time when the data of the single input channel corresponding to the weight arrives at the multi-level pipeline multiplication and accumulation tree structure are aligned in terms of timing; In the convolution operation layer: the input channel-by-channel and row-by-row data stream is input in parallel to P single-output channel convolution calculation units; In a single single-output channel convolution calculation unit, when the first row of channel-by-channel input data flows in, it is convolved with the weight of the corresponding row to obtain the first row of intermediate results, which are temporarily stored in the first register R1, and the first cache unit is used to cache the first row of intermediate results temporarily stored in the first register R1; When the second row of channel-by-channel input data stream comes in, it is temporarily stored in the second row data cache unit in advance, and at the same time, it is convolved with the weight of the corresponding row to obtain the second row of intermediate results, which are temporarily stored in the second register R2, and the second cache unit is used to cache the second row of intermediate results temporarily stored in the second register R2; When the third row of channel-by-channel input data flows in, it is pre-stored in the first row of data cache units, and at the same time, the data of the multiple input channels of the third row are convolved with the weights of the corresponding row to obtain the third row of intermediate results, which are temporarily stored in the third register R3. Subsequently, the data in the first register R1 and the second register R2 are taken out from their respective cache units, and the first row intermediate results, the second row intermediate results, and the third row intermediate results temporarily stored in the first register R1, the second register R2, and the third register R3 are accumulated to obtain the final result Rout_Single of the single output channel convolution calculation unit; when the final results Rout_Single of the P single output channel convolution calculation units are all generated, they will be output channel by channel; After the final result Rout_Single of the convolution calculation of a single row is output channel by channel, before waiting for the data input channel by channel of the fourth row to come in, the data of the multi-input channels of the second row in the second row data cache unit is taken out, and the new intermediate result obtained by convolution calculation with the weight of the first row is also temporarily stored in the first register R1, and the new intermediate result obtained by convolution temporarily stored in the first register R1 is cached in the first cache unit; the third row of data in the first row data cache unit is taken out, and the multi-input channel data of the second row originally in the second row data cache unit is overwritten. After the data in the first row data cache unit is taken out, the new intermediate result obtained by convolution calculation with the weight of the second row is temporarily stored in the second register R2, and the new intermediate result temporarily stored in the second register R 2 is cached in the second cache unit; when the first register R1 and the second register R2 are ready, wait for the data of the fourth row of multiple input channels to come in, repeat the above operation, obtain a new intermediate result and temporarily store it in the third register R3, repeat the above operation, accumulate the first row of intermediate results, the second row of intermediate results and the third row of intermediate results temporarily stored in the first register R1, the second register R2 and the third register R3 to obtain the final result Rout_Single of the single output channel convolution calculation unit; as described above, when the final results Rout_Single of P single output channel convolution calculation units are generated, the final results Rout_Single of multiple single output channel convolution calculation units are output channel by channel.

2. The method according to claim 1, characterized in that The transmission mode of channel-by-channel and then row-by-row is as follows: for multi-channel original image data or feature map data, the data stream transmits a single row and a single channel of data as DATA, the number of channels is M, and each channel has N rows of data, then the data is transmitted channel by channel first. 、 、 Until , that is, after all the data of multiple channels in the first row are transmitted in sequence, the data of multiple channels in the second row are transmitted. 、 、 Until , until the data of multiple channels in row N are transmitted 、 、 Until , until the multi-channel original image data or feature map data is transmitted channel by channel and then row by row.

3. The method according to claim 1, characterized in that In the pooling operation layer: performing odd-numbered row cache and even-numbered row calculation operations on the input data stream; When the odd-numbered lines of input data streams pass through the processing unit, they first enter the cache unit inside the computing unit. The depth of the cache unit is the number of channels of the single-line input data. When the data stream input channel by channel of even rows passes through the processing unit, the odd row data of the corresponding channel is synchronously taken out from the cache unit, and the odd row data and the even row data of the same channel are average pooled or maximum pooled. The calculation result is used as the output of the processing unit, that is, the output data stream of the processing unit is also channel by channel and row by row.

Images