On-chip network operation data stream transmission method based on channel-by-channel line-by-line

Abstract

The invention discloses an on-chip network operation data stream transmission method based on channel-by-channel line-by-line, and belongs to the field of integrated circuit design and artificial intelligence hardware acceleration. According to the method, channel-by-channel transmission, caching and calculation of single-line data can be carried out according to original image data or feature graph data input by an image sensor array channel by channel and line by line, and by means of time division multiplexing of hardware calculation units corresponding to different types of calculation in a network, the calculation efficiency is improved. And the operation of the on-chip traditional convolutional neural network for inputting data channel by channel and line by line is realized. According to the data stream transmission method, an efficient and real-time solution is provided for performing convolutional neural network calculation based on the original data generated by the image sensor line by line.

Description

Technical Field

[0001] The present invention relates to a method for transmitting on-chip network operation data streams channel by channel and row by row, and belongs to the fields of integrated circuit design and artificial intelligence hardware acceleration. Background Art

[0002] With the rapid development of artificial intelligence technology, convolutional neural networks (CNNs) are increasingly widely used in fields such as image processing and computer vision. Traditional CNN algorithms usually need to input complete image data into a computing unit for processing. Usually, off-chip memory is used for data caching, and convolution operations are completed by repeatedly transferring data. This method not only increases the system power consumption, but also reduces the real-time performance due to data transmission delay, and it is difficult to meet the urgent needs for real-time image processing in fields such as autonomous driving and intelligent security.

[0003] In recent years, with the progress of image sensor technology, on-chip intelligent processing architectures have gradually become a research hotspot. In the prior art, there have been attempts to integrate a computing unit and an image sensor on the same chip. In a Chinese patent with the publication number CN113946538A, a convolutional layer fusion storage device and method based on a row buffer mechanism are proposed. This method calculates large images in blocks using a layer fusion storage strategy and uses a row buffer mechanism to reduce the repeated transfer of overlapping image blocks, improving the operation efficiency. However, this method is limited by the data dependence characteristics of traditional CNN algorithms and still needs to completely cache multiple rows of image data to start convolutional calculations. This processing method leads to a significant increase in hardware resource consumption. Especially in high-resolution image scenarios, the contradiction between the row buffer requirements and the chip area and power consumption becomes more prominent. In addition, the problem of hardware resource allocation for different computing layers (such as convolution, pooling, activation function, fully connected) in the network model has not been effectively solved, resulting in low resource utilization.

[0004] The current technical bottleneck is also reflected in the optimization of the data path. When traditional solutions process row-by-row input data, they usually need to build complex row-column control logic to manage the data flow. This not only increases the circuit design complexity but also increases the critical path delay. Especially when processing multi-input channel feature maps, existing architectures often need to configure multiple computing units in parallel to meet the throughput requirements. This extensive resource allocation method directly conflicts with the chip area constraint. Although methods such as compressing data precision or simplifying the network structure can be used to reduce the computational amount, these methods will sacrifice the model accuracy and it is difficult to meet the stringent requirements for algorithm performance in practical applications. How to efficiently utilize hardware resources without sacrificing computational accuracy has become a key problem restricting the development of on-chip intelligent processing technology.

[0005] In terms of real-time performance, most existing technical solutions adopt a post-processing mode, that is, waiting for the input of complete frame data before starting the calculation process. Although this method simplifies the control logic, it causes an inherent delay of about 33 ms (calculated at 30 fps), making it difficult to meet application scenarios with extremely high real-time feedback requirements such as industrial inspection and robot vision. In addition, traditional data flow scheduling methods require multiple data transfers when processing cross-layer calculations, resulting in a decrease in the proportion of effective calculation time. As the depth of the network model increases, this cumulative delay effect will significantly reduce the overall response speed of the system.

[0006] In addition, Chinese Patent Publication No. CN118196437A discloses an image FHOG feature extraction device. By outputting data in a per-channel, per-row, and per-column manner, although it improves the utilization rate of the output bandwidth to a certain extent, this data transmission method requires traversing and reading the data row by row within each channel and then switching columns, increasing the complexity and time of data reading. Moreover, the per-channel, per-row, and per-column access method will cause frequent row switching, resulting in a mismatch between the data access mode and the storage mode, increasing the time overhead of data access.

[0007] Therefore, developing an on-chip neural network operation data flow transmission method that can achieve deep pipelining of data input and calculation processes and maximize the hiding of data transfer delays has become an important breakthrough point for improving the intelligent processing performance at the edge. Summary of the Invention

[0008] The purpose of the present invention is to provide an on-chip network operation data flow transmission method based on per-channel and per-row. This method transmits data in a per-channel first and then per-row manner, that is, first transmits the data of each channel in the first row. After the data of each channel in the first row is transmitted, then transmits the data of each channel in the second row; until all data is transmitted.

[0009] The specific process of this method is as follows: For multi-channel raw image data or feature map data, the single-row and single-channel data transmitted by the data flow is , the number of channels is , and each channel has rows of data. Then the data flow transmission method of per-channel first and then per-row is reflected as first transmitting per-channel , , until , that is, after all the data of multiple channels in the first row is transmitted in sequence, then transmits the data of multiple channels in the second row , , until , and so on, until the Data of multiple channels of a row 、 、 until That is, the transmission of the original image data or feature map data of multiple channels is completed in the order of channel by channel first and then row by row.

[0010] Furthermore, for multiple operation layers of the network-on-chip, the form of the input data stream of each operation layer is channel by channel first and then row by row; the form of the output data stream is also channel by channel first and then row by row.

[0011] Furthermore, for the activation function operation layer of the network-on-chip, the ReLU activation function is generally used, that is, the positive and negative of the input data are judged. When it is greater than 0, the original data is retained and output to the next operation layer (generally the Pooling layer); when it is less than or equal to 0, the original data is set to zero and output to the next operation layer. The hardware architecture corresponding to the ReLU activation function operation is a first-level trigger, that is, when the input data comes in channel by channel and row by row, after passing through the judgment path and the trigger, the result after the ReLU operation can be obtained. Therefore, for the activation function operation layer on the chip, the input data stream becomes the output data stream after passing through the first-level trigger. Therefore, the transmission methods of the input data stream and the output data stream are both channel by channel first and then row by row.

[0012] Furthermore, for the pooling operation layer of the network-on-chip, its corresponding processing unit performs operations such as odd-row caching and even-row calculation on the input data stream, that is, the data stream of odd rows input channel by channel passes through the processing unit and first enters the cache unit inside the calculation unit. The depth of the cache unit is the number of channels of the single-row input data. When the data stream of even rows input channel by channel passes through the processing unit, the odd-row data of the corresponding channel is synchronously retrieved from the cache unit, and the odd-row data and even-row data of the same channel are subjected to average pooling or maximum pooling operations. The calculation result is used as the output of the processing unit, that is, the output data stream mode of the processing unit is also channel by channel and row by row.

[0013] Furthermore, for the fully connected operation layer of the network-on-chip, the corresponding processing unit performs pipelined multiply-accumulate operations on the input data stream. That is, the per-channel and per-row data stream input by the upper-level processing unit directly enters a multi-stage pipelined multiply-accumulate tree structure with the number equal to the number of fully connected calculation output units for parallel calculation, and then the intermediate full-precision calculation results can be obtained for accumulation and temporary storage. When the data input of all input channels of all rows is completed, through time-division multiplexing of the multi-stage pipelined multiply-accumulate tree structure with the number equal to the number of fully connected calculation output units, the final calculation results of the fully connected layer are obtained. In addition, it should be mentioned that when the data stream input by multiple input channels enters the multi-stage pipelined multiply-accumulate tree structure, the weights corresponding to the data of a single input channel are also loaded synchronously in time sequence, that is, the time when the weights reach the multi-stage pipelined multiply-accumulate tree structure is aligned with the time when the data of the single input channel corresponding to the weights reach the multi-stage pipelined multiply-accumulate tree structure in time sequence.

[0014] Furthermore, for the convolutional operation layer of the network-on-chip, the corresponding processing unit performs operations such as multiply-add calculation, caching, output, and overwriting on the data stream input row by row and channel by channel. Specifically, for the per-channel and per-row data stream input, the processing unit corresponding to the convolutional layer will parallelly input it into P (P is a positive integer) single-output-channel ordinary convolutional calculation units; In a single single-output-channel convolutional calculation unit, when the per-channel and per-row data stream of the first row comes in, it is convolved with the weights of the corresponding row to obtain the intermediate result of the first row, which is temporarily stored in the first register R1, and the first buffer unit RAM1 (hereinafter referred to as RAM1) caches the intermediate result of the first row temporarily stored in the first register R1; When the per-channel and per-row data stream of the second row comes in, it is first temporarily stored in the second-row data buffer unit Buffer2 (hereinafter referred to as Buffer2), and at the same time, it is convolved with the weights of the corresponding row to obtain the intermediate result of the second row, which is temporarily stored in the second register R2, and the second buffer unit RAM2 (hereinafter referred to as RAM2) caches the intermediate result of the second row temporarily stored in the second register R2; When the per-channel and per-row data stream of the third row comes in, it is first temporarily stored in the first-row data buffer unit Buffer1 (hereinafter referred to as Buffer1), and at the same time, the multi-input-channel data of the third row is convolved with the weights of the corresponding row to obtain the intermediate result of the third row, which is temporarily stored in the third register R3; Subsequently, the intermediate result of the first row and the intermediate result of the second row are fetched from the first register R1 and the second register R2, and the intermediate result of the first row, the intermediate result of the second row, and the intermediate result of the third row are accumulated to obtain the final result Rout_Single of the single-output-channel convolution calculation unit; when the final results Rout_Single of P single-output-channel convolution calculation units are all generated, they will be output channel by channel. After the final result Rout_Single of the single-row convolution calculation is output channel by channel, before waiting for the data of the fourth row to be input channel by channel, the multi-input-channel data of the second row in Buffer2 is fetched, and the new intermediate result obtained by convolving it with the weights of the first row is also temporarily stored in the first register R1, and the new intermediate result obtained by convolution and temporarily stored in the first register R1 is cached in RAM1; the data of the third row in Buffer1 is fetched, and at the same time, the original multi-input-channel data of the second row in Buffer2 is overwritten. After the data in Buffer1 is fetched, the new intermediate result obtained by convolving it with the weights of the second row is temporarily stored in the second register R2, and the new intermediate result obtained by convolution and temporarily stored in the second register R2 is cached in RAM2; when both the first register R1 and the second register R2 are ready, wait for the data of the fourth row of multi-input channels to come in, and repeat the operation when "the data of the third row of multi-input channels comes in" to obtain the new intermediate result and temporarily store it in the third register R3; the intermediate result of the first row, the intermediate result of the second row, and the intermediate result of the third row temporarily stored in the first register R1, the second register R2, and the third register R3 are accumulated to obtain the final result Rout_Single of the single-output-channel convolution calculation unit; as described above, when the final results Rout_Single of P single-output-channel convolution calculation units are all generated, the final results Rout_Single of multiple single-output-channel convolution calculation units are output channel by channel.

[0015] Since the P single-output-channel ordinary convolution calculation units are independent of each other and perform parallel calculations, when the final results Rout_Single of the P single-output-channel convolution calculation units are all generated, the processing unit corresponding to the convolution calculation will output the final results Rout_Single generated by the P single-output-channel convolution calculation units channel by channel, while waiting for the data of the fourth row to be input channel by channel.

[0016] The beneficial effects of the present invention are as follows: The present invention provides a method for transmitting an on-chip network operation data stream based on channel-by-channel and row-by-row. This method transmits data in a way that first goes channel-by-channel and then row-by-row, which can avoid channel data congestion and improve transmission efficiency. Moreover, under this transmission method, the data has been preliminarily classified when entering storage or subsequent stages, improving the efficiency of data processing and storage. In addition, it also improves the flexibility and scalability of transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0018] Figure 1 It is a schematic diagram of multi-channel raw image data or feature map data suitable for the data stream transmission method of first channel-by-channel and then row-by-row in the method for transmitting an on-chip network operation data stream provided in Embodiment 1 of the present invention; Figure 2 It is a schematic diagram of implementing data transmission in the fully connected layer in the method for transmitting an on-chip network operation data stream based on channel-by-channel and row-by-row provided in Embodiment 2 of the present invention; Figure 3 It is a schematic diagram of implementing data transmission in the pooling layer in the method for transmitting an on-chip network operation data stream based on channel-by-channel and row-by-row provided in Embodiment 3 of the present invention; Figure 4 It is a schematic diagram of implementing data transmission in the convolutional layer in the method for transmitting an on-chip network operation data stream based on channel-by-channel and row-by-row provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will further describe the embodiments of the present invention in detail with reference to the drawings.

[0020] Embodiment 1 This embodiment provides a method for transmitting an on-chip network operation data stream based on channel-by-channel and row-by-row. Refer to Figure 1 , this method includes: For multi-channel raw image data or feature map data, the single-channel and single-row data transmitted by the data stream is , the number of channels is , and each channel has rows of data. Then the data stream transmission method of first channel-by-channel and then row-by-row is reflected as transmitting channel-by-channel first , , until , that is, after the data of multiple channels in the first row are all transmitted in sequence, then the data of multiple channels in the second row are transmitted , , until , and so on, until the data of multiple channels in the th row are transmitted , , until , which indicates that the original image data or feature map data of multiple channels are transmitted in the order of channel by channel first and then row by row.

[0021] For the multiple operation layers corresponding to the network-on-chip, the form of the input data stream of each operation layer is channel by channel first and then row by row; the form of the output data stream is also channel by channel first and then row by row.

[0022] For the activation function operation layer of the network-on-chip, the ReLU activation function is generally used, that is, the positive and negative of the input data are judged. When it is greater than 0, the original data is reserved and output to the next operation layer (generally the Pooling layer); when it is less than or equal to 0, the original data is set to zero and output to the next operation layer. The hardware architecture corresponding to the ReLU activation function operation is a first-level trigger, that is, when the input data comes in channel by channel and row by row, after passing through the judgment path and the trigger, the result after the ReLU operation can be obtained. Therefore, for the on-chip activation function operation layer, the input data stream becomes the output data stream after passing through the first-level trigger. Therefore, the transmission methods of the input data stream and the output data stream are both channel by channel first and then row by row.

[0023] For the average pooling or maximum pooling operation layer of the network-on-chip, the corresponding processing unit performs operations such as odd-row caching and even-row calculation on the input data stream, that is, the data stream of odd rows input channel by channel passes through the processing unit and first enters the cache unit inside the calculation unit. The depth of the cache unit is the number of channels of the single-row input data. When the data stream of even rows input channel by channel passes through the processing unit, the odd-row data of the corresponding channel is synchronously fetched from the cache unit, and the odd-row data and even-row data of the same channel are subjected to average pooling or maximum pooling operations. The calculation result is used as the output of the processing unit, that is, the output data stream mode of the processing unit is also channel by channel and row by row.

[0024] For the fully-connected layer operation of the network-on-chip, the corresponding processing unit performs pipelined multiply-accumulate operations on the input data stream. That is, the per-channel and per-row data stream input by the upper-level processing unit directly enters a multi-stage pipelined multiply-accumulate tree structure with the number equal to the number of fully-connected calculation output units for parallel calculation, and then the intermediate full-precision calculation results can be obtained for accumulation and temporary storage. When the data input of all input channels of all rows is completed, through the time-division multiplexing of the multi-stage pipelined multiply-accumulate tree structure with the number equal to the number of fully-connected calculation output units, the final calculation result of the fully-connected layer is obtained. In addition, it should be mentioned that when the data stream with multiple input channels enters the multi-stage pipelined multiply-accumulate tree structure, the weights corresponding to the data of a single input channel are also loaded synchronously in time sequence, that is, the time when the weights reach the multi-stage pipelined multiply-accumulate tree structure is aligned with the time when the data of the single input channel corresponding to the weights reach the multi-stage pipelined multiply-accumulate tree structure in time sequence.

[0025] For the convolutional operation layer of the network-on-chip, the corresponding processing unit performs operations such as multiply-add calculation, caching, output, and overwrite on the multi-row per-channel input data stream. Specifically, for the per-channel and per-row input data stream, the processing unit corresponding to the convolutional calculation will parallelly input it into P (P is a positive integer) single-output-channel ordinary convolutional calculation units.

[0026] In a single single-output-channel ordinary convolutional calculation unit, when the per-channel and per-row input data stream of the first row comes in, it is convolved with the weights of the corresponding row to obtain the intermediate result in the first register R1, and the first cache unit RAM1 (hereinafter referred to as RAM1) is used to cache the intermediate convolutional result in the first register R1 corresponding to the first row; When the per-channel and per-row input data stream of the second row comes in, it is temporarily stored in the second-row data buffer unit Buffer2 (hereinafter referred to as Buffer2) in advance. At the same time, it is convolved with the weights of the corresponding row to obtain the intermediate result of the second row, which is temporarily stored in the second register R2, and the second cache unit RAM2 (hereinafter referred to as RAM2) is used to cache the intermediate result of the second row stored in the second register R2; When the data stream input channel by channel in the third row comes in, it is temporarily stored in the first-row data buffer unit Buffer1 (hereinafter referred to as Buffer1). At the same time, the data of multiple input channels in the third row is convolved with the weights of the corresponding row to obtain the intermediate result of the third row, which is temporarily stored in the third register R3. Subsequently, the data in the first register R1 and the second register R2 is taken out from their respective buffer units, and the intermediate results of the first row, the second row, and the third row temporarily stored in the first register R1, the second register R2, and the third register R3 are accumulated to obtain the final result Rout_Single of the single-output channel convolution calculation unit. When the final results Rout_Single of P single-output channel convolution calculation units are all generated, they are output channel by channel. Since the P single-output channel ordinary convolution calculation units are independent of each other and perform parallel calculations, when the final results Rout_Single of the P single-output channel convolution calculation units are all generated, the processing unit corresponding to the convolution calculation will output the final results Rout_Single generated by the P single-output channel convolution calculation units channel by channel, and at the same time wait for the data input channel by channel in the fourth row to arrive.

[0027] Before the data input channel by channel in the fourth row arrives, the data of multiple input channels in the second row in Buffer1 is taken out, and it is convolved with the weights of the first row to obtain a new intermediate result r1, and the new intermediate result r1 obtained by convolution is cached using the first buffer unit RAM1. The data in the third row in Buffer2 is taken out, and at the same time, it overwrites the data of multiple input channels in the original second row in Buffer1. After the data in Buffer2 is taken out, it is convolved with the weights of the second row to obtain a new intermediate result r2, and the new intermediate result r2 obtained by convolution is cached using the buffer unit RAM2. When both r1 and r2 are ready, the processing unit corresponding to the convolution calculation waits for the data of multiple input channels in the fourth row to come in, and repeats the above "operations when the data of multiple input channels in the third row comes in" to obtain the intermediate result r3, and then outputs the final results Rout_Single of multiple single-output channel convolution calculation units channel by channel.

[0028] Therefore, when the first register R1 and the second register R2 corresponding to the data of the current two rows are both ready, the processing unit corresponding to the convolution calculation waits for the data of multiple input channels in the next row to arrive, repeats the above "operations when the data of multiple input channels in the third row comes in" to obtain the intermediate result r3, and then outputs the final results Rout_Single of multiple single-output channel convolution calculation units channel by channel.

[0029] Embodiment 2 This embodiment provides a method for implementing on-chip network operation data stream transmission based on per-channel and per-row in a fully-connected operation layer. Refer to Figure 2 , the method includes: The top-left pixel array generates raw data row by row. After the single-row data is operated by the activation function, it is output through a single-stage pipeline without involving storage operations. After the single-row data is operated by the pooling function, for odd-row data, it is first temporarily stored; for even-row data, after it comes in, it is pooled with the temporarily stored odd-row data to generate the pooled single-row data and input it to the next convolutional operation layer. The convolutional operation layer needs to temporarily store two rows of data, and respectively complete the multiplication and accumulation calculations of each row with the corresponding weights. When the third row of data arrives, after completing the multiplication and accumulation calculation with its corresponding weight, it is added to the results of the multiplication and accumulation calculations of the above two rows with their respective weights to obtain the new single-row data and input it to the next activation function operation layer.

[0030] That is, the data stream circulates in the order of "convolution operation - activation function - pooling operation - convolution operation" until after the last pooling operation, the single-row data generated by the pooling operation is input to the fully-connected calculation unit block by block. As Figure 2 shown, the bit width of the input single row is M*n*K bit, the bit width of a single data is n bit, and the input data stream is M*n bit / clk, that is, it takes K clks to transmit the single-row data. The number M of feature data input per clk is the number of excitation inputs of the multiplication and accumulation tree structure in a single fully-connected calculation unit. Here, the input data with a granularity of M will be broadcast in parallel to N multiplication and accumulation tree structures. Since a single multiplication and accumulation tree is composed of multiple-stage pipelines, a time-division multiplexing strategy is actually adopted for the multiplication and accumulation tree structure. When traversing all rows and all channels of data with a granularity of M and inputting it to N pipelined multiplication and accumulation tree structures, the last-stage register of the N parallel pipelined multiplication and accumulation trees is the result of the fully-connected calculation {Result_1, Result_2... Result_N}.

[0031] Embodiment III This embodiment provides a method for implementing on-chip network operation data stream transmission based on per-channel and per-row in a pooling operation layer. This method is implemented through the pooling layer microarchitecture. As Figure 3 shown, the method includes: The granularity of the input data is 8 * 8 bit / clk = 64 bit / clk, that is, 8 8-bit excitations are input as a group. Datain is the per-channel and per-row input from the previous-level operation unit. When the data of the first channel in the odd row is input into this microarchitecture, a 128-bit data formed by combining two consecutive 64-bit data is stored in the storage unit dprf_32x128_01. The next two consecutive 64-bit data combined into a 128-bit data are stored in the storage unit dprf_32x128_02, and so on, until a 128-bit data formed by combining the last two consecutive 64-bit data of the first input channel in this odd row is stored in dprf_32x128_n, completing the temporary storage of all data of the first input channel in this odd row. Immediately after this microarchitecture successively completes the temporary storage of all input channel data in this odd row in the above manner, this architecture waits for the per-channel input of even-row data.

[0032] The depth of each storage unit is the number of input channels for a single row of data. When the data of the first channel in the even row is input into this microarchitecture, two consecutive 64-bit data are combined into a 128-bit data for the first time. At the same time, it is necessary to retrieve the 128-bit data corresponding to address 0 from dprf_32x128_01 that temporarily stores multi-channel data of the odd row, and input the two 128-bit data into the average pooling / maximum pooling calculation unit (the average pooling or maximum pooling operation is implemented by the combinational logic of digital circuits), and a 64-bit pooling calculation result is obtained as the output of this calculation unit; When the next two consecutive 64-bit data are combined into a 128-bit data, it is necessary to retrieve the 128-bit data corresponding to address 0 from dprf_32x128_02 that temporarily stores multi-channel data of the odd row, and input the two 128-bit data into the average pooling / maximum pooling calculation unit (the average pooling or maximum pooling operation is implemented by the combinational logic of digital circuits), and a 64-bit pooling calculation result is obtained as the output of this calculation unit; Repeat this process until a 128-bit data is formed by the last two consecutive 64-bit data in the first input channel of the even rows. At the same time, take out the 128-bit data corresponding to the address 0 from dprf_32x128_n that temporarily stores the multi-input channel data of the odd rows, and input the two 128-bit data into the average pooling / maximum pooling calculation unit (the average pooling or maximum pooling operation is implemented by the combinational logic of digital circuits), and obtain a 64-bit pooling calculation result as the output of this calculation unit. At this time, the pooling calculation of the data in the first input channel of the two rows of data has been completed, and the calculation result has been output to the next operation layer; when the data in the second input channel and all the remaining input channels of the even rows are input into this micro-architecture, the operations performed are the same as those performed when the data in the first input channel of the even rows is input into this micro-architecture. The only difference is that the address corresponding to the data taken out from the storage unit dprf increases as the input channels of the even rows increase. In addition, when new odd-row data is input into this micro-architecture, it will overwrite all the data inside dp_rf. When new even-row data is input into this micro-architecture, the operations performed are the same as those of the previous even-row data. Therefore, in summary, the input data method of this pooling operation unit micro-architecture is channel-by-channel and row-by-row, and the output data method is also channel-by-channel and row-by-row.

[0033] Example 4 This embodiment provides a method for implementing an on-chip network operation data stream transmission based on channel-by-channel and row-by-row in a convolution operation layer. This method is implemented based on a general convolution calculation hardware architecture, and this architecture is as Figure 4 shown, including a first input port, a second input port, a first output port, a second output port, two row data buffer units, a weight control unit, 16 single-output channel convolution calculation units, 16 activation and quantization units, 16 single-port register files, and a serial output unit; each single-output channel convolution calculation unit includes three three-stage pipelined multiply-accumulate units, and the three-stage pipelined multiply-accumulate unit includes a three-stage pipelined multiplication unit, an accumulation unit, and a cache control unit; the activation and quantization unit includes an adder, a multiplier, a binary signed number judgment logic unit corresponding to the adder and the multiplier, and a multi-stage pipelined structure unit with a shift function.

[0034] This method includes: This general convolution calculation architecture performs a general convolution calculation with a 4input_channel×1920*1080*8bit feature map, where the input channels are 4, the output channels are 16, the convolution kernel size is 3×3, the stride is 1, and "zero padding" is 1. The size of the output feature map is 16 output_channel×1920*1080*8bit.

[0035] The input multi-input channel data pool_data is 64bit / clk, supplemented by a pool_data_vld signal. One row of single-input channel data has 240 consecutive pool_data in time series, that is, the corresponding pool_data_vld is a high level that lasts for 240 clks. Since the input feature image has 4 channels, that is, the single-row input data valid signal contains 4 high levels that last for 240 clks, and the interval between each high level is 1 clk.

[0036] Since the step of the implemented ordinary convolution calculation is 1, the architecture will judge whether the input data is the first row of data. If it is the first row of data, it will be broadcast to 16 single-output channel convolution calculation units and simultaneously input to the row data buffer unit 2. At this time, the row data input to a single-output channel convolution calculation unit will be input to the second three-stage pipelined multiply-accumulate unit PE2 in a single single-output channel convolution calculation unit and perform pipelined multiply-accumulate calculation with {W3, W2, W1} of the second row of the convolution kernel. After the calculation is completed, the second three-stage pipelined multiply-accumulate unit PE2 inside the single-output channel convolution calculation unit will generate a data ready signal Rdy2. After Rdy2 is ready, the single single-output channel convolution calculation unit will read the first row of data in the row data buffer unit 2 and perform pipelined multiply-accumulate calculation with {W3, W2, W1} of the first row of the convolution kernel by the first three-stage pipelined multiply-accumulate unit PE1 in the single single-output channel convolution calculation unit. After the calculation is completed, the first three-stage pipelined multiply-accumulate unit PE1 inside the single-output channel convolution calculation unit will generate a data ready signal Rdy1. After Rdy1 is ready, the single-output channel convolution calculation unit waits for the arrival of the next row of data; When the second row of data is input into the architecture, while being broadcast to 16 single-output channel convolutional calculation units, it is also input into the row data buffer unit 1. At this time, the row data input into a single-output channel convolutional calculation unit will be input into the third three-stage pipelined multiply-accumulate unit PE3 in a single single-output channel convolutional calculation unit and perform pipelined multiply-accumulate calculation with {W3, W2, W1} of the third row of the convolutional kernel. After the calculation is completed, the third three-stage pipelined multiply-accumulate unit PE3 inside the single-output channel convolutional calculation unit will generate a data ready signal Rdy3. After Rdy3 is ready, the full-precision calculation results cached inside the second three-stage pipelined multiply-accumulate unit PE2 and the third three-stage pipelined multiply-accumulate unit PE3 will be read out one by one and perform dot-addition accumulation. The accumulated result is the full-precision calculation result of the first row of the single-output channel. Subsequently, this result is input into the subsequent activation and quantization unit. At the same time, because the Rdy3 signal is ready, the single-output channel convolutional calculation unit will read the data in the row data buffer unit 1. It should be noted that the data in the row data buffer unit 1 will be transmitted to the row data buffer unit 2 while being transmitted to the single-output channel convolutional calculation unit. The data in the row data buffer unit 1 is input into the second three-stage pipelined multiply-accumulate unit PE2 in a single single-output channel convolutional calculation unit and perform pipelined multiply-accumulate calculation with {W3, W2, W1} of the second row of the convolutional kernel. After the calculation is completed, the second three-stage pipelined multiply-accumulate unit PE2 inside the single-output channel convolutional calculation unit will generate a data ready signal Rdy2. After Rdy2 is ready, the single-output channel convolutional calculation unit waits for the arrival of the next row of data; When the third row of data and all the remaining 1077 rows of data are input into this architecture, while being broadcast to 16 single-output-channel convolutional computing units, they will also be input into the row data buffer unit 1. At this time, the row data input to a single-output-channel convolutional computing unit will be input to the third three-stage pipelined multiply-accumulate unit PE3 in a single single-output-channel convolutional computing unit to perform pipelined multiply-accumulate calculation with {W3, W2, W1} of the third row of the convolutional kernel. After the calculation is completed, the third three-stage pipelined multiply-accumulate unit PE3 inside the single-output-channel convolutional computing unit will generate a data ready signal Rdy3. After Rdy3 is ready, the full-precision calculation results cached inside the first three-stage pipelined multiply-accumulate unit PE1, the second three-stage pipelined multiply-accumulate unit PE2, and the third three-stage pipelined multiply-accumulate unit PE3 will be read out one by one and subjected to dot-addition accumulation. The result of the accumulation is the full-precision calculation result of the second row of the single-output channel. Subsequently, this result is input to the subsequent activation and quantization unit. At the same time, because the Rdy3 signal is ready, the single-output-channel convolutional computing unit will read the data in the row data buffer unit 2, input this data to the first three-stage pipelined multiply-accumulate unit PE1 in a single single-output-channel convolutional computing unit to perform pipelined multiply-accumulate calculation with {W3, W2, W1} of the first row of the convolutional kernel. After the calculation is completed, the first three-stage pipelined multiply-accumulate unit PE1 inside the single-output-channel convolutional computing unit will generate a data ready signal Rdy1. After Rdy1 is ready, the single-output-channel convolutional computing unit will read the data in the row data buffer unit 1. It should be noted that the data in the row data buffer unit 1 will be transmitted to the row data buffer unit 2 while being transmitted to the single-output-channel convolutional computing unit. The data in the row data buffer unit 1 is input to the second three-stage pipelined multiply-accumulate unit PE2 in a single single-output-channel convolutional computing unit to perform pipelined multiply-accumulate calculation with {W3, W2, W1} of the second row of the convolutional kernel. After the calculation is completed, the first three-stage pipelined multiply-accumulate unit PE1 inside the single-output-channel convolutional computing unit will generate a data ready signal Rdy2. After Rdy2 is ready, the single-output-channel convolutional computing unit waits for the arrival of the next row of data.

[0037] According to the above operations, repeating this process, this architecture can obtain the single-output-channel calculation results of all rows. However, because the input row data will be broadcast to 16 single-output-channel convolutional computing units simultaneously, and since each single-output-channel convolutional computing unit is independent of each other, that is, parallel computing, when the full-precision convolutional calculation results of each output channel of a single row pass through the respective activation and quantization units and are all input into their respective single-port register files, this architecture will sequentially read out the data inside the 16 single-port register files, that is, transmit the multi-output-channel data of a single row channel by channel to the subsequent network operation layer (such as the Pooling layer).

[0038] Some steps in the embodiments of the present invention can be implemented by software, and the corresponding software program can be stored in a readable storage medium, such as an optical disc or a hard disk, etc.

[0039] The above are only the preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for transmitting data streams of network-on-chip computing operations on a channel-by-channel and row-by-row basis, characterized in that: The method comprises: for the convolution operation layer and the pooling operation layer of the on-chip network, the input data stream is generated by the upper-level network operation layer, and the data stream input mode is first channel by channel and then row by row; the output data stream is generated by the current-level network operation layer, and the data stream output mode is also first channel by channel and then row by row; For the activation function operation layer of the on-chip network, the input data stream is only subjected to pipeline processing, that is, the data stream input channel by channel and row by row is directly output after pipeline processing by the activation function operation layer, that is, the output data stream is the same as the input data stream; For the fully connected operation layer of the on-chip network, its input data stream performs pipeline multiplication and accumulation operations, that is, the channel-by-channel and row-by-row data stream input by the upper processing unit directly enters the multi-level pipeline multiplication and accumulation tree structure whose number is equal to the number of fully connected calculation output units for parallel calculation, and the intermediate full-precision calculation results are accumulated and temporarily stored. When the data input of all input channels of all rows is completed, the final calculation result of the full connection is obtained through time division multiplexing of the multi-level pipeline multiplication and accumulation tree structure whose number is equal to the number of fully connected calculation output units; when the data stream input by multiple input channels enters the multi-level pipeline multiplication and accumulation tree structure, the weight corresponding to the data of a single input channel is also loaded synchronously in timing, that is, the time when the weight arrives at the multi-level pipeline multiplication and accumulation tree structure and the time when the data of the single input channel corresponding to the weight arrives at the multi-level pipeline multiplication and accumulation tree structure are aligned in timing.

2. The method according to claim 1, characterized in that The transmission mode of first channel by channel and then line by line is as follows: for multi-channel original image data or feature map data, the single-line single-channel data transmitted by the data stream is , the number of channels is , each channel has If the row data is transmitted, it is transmitted channel by channel first. , , Until That is, after all the data of multiple channels in the first row are transmitted in sequence, the data of multiple channels in the second row are transmitted. , , Until , until the transmission is completed Multiple channels of data , , Until , until the multi-channel original image data or feature map data is transmitted channel by channel and then row by row.

3. The method according to claim 1, characterized in that In the pooling operation layer: the operations of caching odd rows and calculating even rows are performed on the input data stream; When the data stream of odd-numbered lines input channel by channel passes through the processing unit, it first enters the cache unit inside the calculation unit. The depth of the cache unit is the number of channels of the single-line input data; When the data stream of even-numbered rows input channel by channel passes through the processing unit, the odd-numbered rows of data of the corresponding channel are synchronously taken out from the cache unit, and the odd-numbered row data and the even-numbered row data of the same channel are average pooled or maximum pooled. The calculated result is used as the output of the processing unit, that is, the output data stream of the processing unit is also channel by channel and row by row.

4. The method according to claim 1, characterized in that In the convolution operation layer: the input channel-by-channel and row-by-row data streams are input in parallel to P single-output channel convolution calculation units; In a single single-output channel convolution calculation unit, when the first row of channel-by-channel input data flows in, it is convolved with the weight of the corresponding row to obtain the first row of intermediate results, which are temporarily stored in the first register R1, and the first cache unit is used to cache the first row of intermediate results temporarily stored in the first register R1; When the second row of channel-by-channel input data stream comes in, it is pre-stored in the second row data cache unit, and at the same time, the second row of intermediate results are obtained by convolution calculation with the weight of the corresponding row and temporarily stored in the second register R2, and the second cache unit is used to cache the second row of intermediate results temporarily stored in the second register R2; When the data stream input channel by channel of the third row comes in, it is temporarily stored in the data cache unit of the first row in advance, and the data of the multi-input channels of the third row are convolved with the weights of the corresponding row to obtain the intermediate result of the third row, which is temporarily stored in the third register R3, and then the data in the first register R1 and the second register R2 are taken out from their respective cache units, and the intermediate results of the first row, the second row and the third row temporarily stored in the first register R1, the second register R2 and the third register R3 are accumulated to obtain the final result Rout_Single of the single output channel convolution calculation unit; when the final results Rout_Single of P single output channel convolution calculation units are generated, they will be output channel by channel; After the final result Rout_Single of the convolution calculation of a single row is output channel by channel, before waiting for the data input channel by channel of the fourth row to come in, the data of the multi-input channels of the second row in the data cache unit of the second row are taken out, and the new intermediate result obtained by convolution calculation with the weight of the first row is also temporarily stored in the first register R1, and the new intermediate result obtained by convolution temporarily stored in the first register R1 is cached in the first cache unit; The third row of data in the first row of data cache units is taken out, and the multi-input channel data of the original second row in the second row of data cache units is overwritten at the same time. After the data in the first row of data cache units is taken out, it is convolved with the weight of the second row to obtain a new intermediate result and temporarily stored in the second register R2, and the new intermediate result obtained by the convolution temporarily stored in the second register R2 is cached in the second cache unit; when the first register R1 and the second register R2 are ready, wait for the data of the fourth row of multi-input channels to come in, repeat the above operation, obtain a new intermediate result and temporarily store it in the third register R3, repeat the above operation, accumulate the first row intermediate result, the second row intermediate result and the third row intermediate result temporarily stored in the first register R1, the second register R2 and the third register R3, and obtain the final result Rout_Single of the single output channel convolution calculation unit; as described above, when the final results Rout_Single of P single output channel convolution calculation units are generated, the final results Rout_Single of multiple single output channel convolution calculation units are output channel by channel.

Images