PE unit and systolic array of reconfigurable convolutional neural network
By designing the PE unit and pulsating array of reconfigurable convolutional neural networks, the problem of inefficiency in multi-precision neural network calculations is solved, efficient processing and data transmission of multi-bit wide computing is realized, and the speed and efficiency of convolutional computing are improved. .
Patent Information
- Application Number
- CN202420377077.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Utility models(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-28
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2034-02-28
AI Technical Summary
The existing multi-precision neural network computing hardware cannot effectively compatible with weighted data of different precisions, resulting in inefficient computing efficiency and data errors, and lack of acceleration structures suitable for multi-precision to reduce data relocation.
A PE unit of reconfigurable convolutional neural network was designed. Combined with a pulsating array, it can process 2bit, 4bit, and 6bit low bit width weights, and realize horizontal transmission and weight row of image data through a pulsating array composed of nine PE units. Inter-cycle, the calculation results are vertically transmitted, and multiple bit width operations are completed to improve the convolution operation speed.
It realizes efficient calculations under the multi-bit wide computing conditions, reduces data handling volume, improves computing efficiency, and replaces the addition tree structure with a new pulsating array structure, shortens the data communication distance and saves IO bandwidth.
Smart Images

Figure CN221977397U_ABST
Abstract
Description
Technical Field
[0001] The utility model relates to the field of convolutional neural networks, in particular to a PE unit and a systolic array of a reconfigurable convolutional neural network. Background Art
[0002] At present, convolutional neural networks are developing rapidly, and have achieved performance close to or even surpassing that of humans in the fields of computer vision, speech recognition, and natural language processing. However, as the network recognition rate increases, it brings about the emergence of deeper and more complex network structures and larger parameter calculations. This has led to a surge in the amount of calculation and operation time of neural networks. In order to deal with this problem, some people have proposed to reduce the computational complexity by optimizing the layer-by-layer accuracy of neural networks, and low-bit multi-precision quantization parameters have emerged. After experimental verification, the difference in weight bit widths for different networks and different layers does not lead to a decrease in network accuracy. Therefore, lower data precision quantization can be used to represent the weight parameters of different network layers, thereby reducing the computational complexity and parameter storage, and achieving the effect of speeding up the neural network.
[0003] However, the calculation of multi-precision neural networks requires customized hardware circuits to support it, because single-precision processors waste a lot of resources when calculating mixed precision, while multi-precision neural network processors support multi-precision at the circuit level, and the calculation can achieve higher utilization than single-precision processors. However, multi-precision calculations in existing technologies cannot balance the requirements of low computing cost consumption and higher speed. Most of them still need to reduce the data bit width to a lower bit under low precision, which can easily lead to performance loss.
[0004] At the same time, faced with a large amount of data in the neural network, how to reuse the scheduling data to reduce the amount of data relocation is also a problem. Appropriate acceleration calculation results are needed to effectively organize the basic multiplication units. The current mainstream single-precision acceleration structures mainly include row buffers, two-dimensional arrays, pulsating arrays, etc., all of which use data reuse to reduce transportation. However, the acceleration structure for multi-precision is not complete. The complex scheduling form and the need to be compatible with different precision weights make the traditional acceleration architecture not universal on it, which will cause inefficiency and even data errors.
[0005] Therefore, there are many areas that need to be improved in the existing multi-precision network computing hardware. What is needed now is to design a hardware circuit that is compatible with multi-precision and achieves reasonable data scheduling. Utility Model Content
[0006] The main purpose of the utility model is to overcome the above-mentioned defects in the prior art and propose a PE unit of a reconfigurable convolutional neural network, which can complete 2-bit, 4-bit, and 6-bit low-bit width weight convolutional neural networks; at the same time, a new systolic array is designed to combine nine PE units into a group, image data is transmitted horizontally, weights are circulated between rows, and calculation results are transmitted vertically, which can realize four 8bitx6bit operations, six 8bitx4bit operations or twelve 8bitx2bit operations in one cycle, greatly improving the speed of convolution operations.
[0007] The utility model adopts the following technical solutions:
[0008] A PE unit of a reconfigurable convolutional neural network comprises six multipliers, five shift registers, two first adders and one second adder; the six multipliers are arranged in parallel, one input end of each multiplier inputs image data, and the other input end inputs weight data; the input ends of the five shift registers are connected to the output ends of five of the multipliers; the two input ends of one of the first adders are respectively connected to the output ends of one group of two shift registers, and the two input ends of another of the first adders are respectively connected to the output end of another shift register and the output end of the multiplier not connected to the shift register; the two input ends of the second adder are respectively connected to the output ends of another group of two shift registers, and the output ends of the two first adders are connected to the output end of the second adder.
[0009] The pins after the PE unit is packaged include a clk terminal, a rst terminal, a data terminal, a weight terminal, a choose terminal, a result terminal and a data_out terminal, wherein the clk terminal is used to input a clock signal, the rst terminal is used to input a reset signal, the data terminal is used to input image data, the weight terminal is used to input weight data, the choose terminal is used to input weight bit width selection data, the result terminal is used to output image data, and the data_out terminal is used to output calculation results.
[0010] The weight bit width selection data includes 00 representing 2 bits, 01 representing 4 bits and 10 representing 6 bits; the calculation result is 8 bits wide, when the weight bit width is 2 bits, seven data are output; when the weight bit width is 4 bits, three data are output; when the weight bit width is 6 bits, two data are output.
[0011] A multi-width bit reconfigurable convolution systolic array, characterized by comprising a register and a plurality of PE units of a reconfigurable convolutional neural network.
[0012] It includes a first row of units, a second row of units and a third row of units arranged vertically, wherein the first row of units, the second row of units and the third row of units are respectively provided with three PE units arranged from left to right and connected, the calculation result of each PE unit of the first row of units is cascaded to the corresponding PE unit of the second row of units, the calculation result of each PE unit of the second row of units is cascaded to the corresponding PE unit of the third row of units, and the calculation result of each PE unit of the third row of units is transmitted to the register for data accumulation and then output.
[0013] The image data flows into the PE unit on the right side of each row of units and propagates laterally, the weight data flows into the three PE units of each row of units, the image data and the weight data flow into the second row of units one clock cycle slower than into the first row of units, and the image data and the weight data flow into the third row of units one clock cycle slower than into the second row of units.
[0014] The weight data is set for inter-row circulation. For the nine weight data of 3x3 convolution, the first row of units is circulated in the order of [w1, w2, w3], w1, w2, w3 respectively represent the weights of the three PE units of the first row of units, the second row of units is circulated in the order of [w4, w5, w6] and is one clock cycle slower than the first row of units, w4, w5, w6 respectively represent the weights of the three PE units of the second row of units, the third row of units is circulated in the order of [w7, w8, w9] and is one clock cycle slower than the second row of units, w7, w8, w9 respectively represent the weights of the three PE units of the third row of units.
[0015] It can be seen from the above description of the utility model that compared with the prior art, the utility model has the following beneficial effects:
[0016] The utility model designs a reconfigurable convolutional neural network PE unit that can complete 2, 4, and 6-bit low-bit width weights and fixed-bit width image data, and adopts a pulsating array form to transmit data. It can be used in the hardware acceleration hardware system of a 3x3 neural network to improve the computing speed while satisfying the condition of multi-bit width operations.
[0017] The circuit structure of the utility model is reconfigurable, and the operation mode can be determined according to the input bit width. It is compatible with the three common low bit widths of 2bit, 4bit and 6bit in convolution, and is combined with the output channel parallel in the convolution acceleration operation. It can output multi-channel data results in one operation, which greatly reduces the amount of data transport and improves the operation efficiency.
[0018] The utility model adopts a novel systolic array structure to replace the addition tree structure, and combines the sliding window with the systolic array. The data communication distance between the processing units is short, so that the design of the data flow and the control flow and the synchronous control save the IO bandwidth. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 This is the composition diagram of the PE unit of the utility model;
[0020] Figure 2 This is the packaging diagram of the PE unit of the utility model;
[0021] Figure 3 This is a composition diagram of the pulsating array of the utility model.
[0022] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. DETAILED DESCRIPTION
[0023] The present invention is further described below through specific implementation methods.
[0024] In the present utility model, the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. In the description, the directions or positional relationships indicated by "upper", "lower", "left", "right", "front", and "back" are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present utility model, and are not intended to indicate or imply that the device referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the scope of protection of the present utility model. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to the specific circumstances.
[0025] In addition, in the description of this application, unless otherwise specified, "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.
[0026] See also Figure 1, a PE unit of a reconfigurable convolutional neural network, which includes six multipliers 10, five shift registers 20, two first adders 31 and a second adder 32. The six multipliers 10 are arranged in parallel, and one input end of each multiplier 10 inputs image data, and the other input end inputs weight data. The input ends of the five shift registers 20 are connected to the output ends of five of the multipliers 10. As shown in the figure, the output end of the multiplier 10 on the far right is not connected to the shift register 20. The two input ends of one of the first adders 31 are respectively connected to the output ends of two shift registers 20 of one group, which is the two shift registers 20 on the far left; the two input ends of another first adder 31 are respectively connected to the output end of another shift register 20 and the output end of the adjacent multiplier 10 that is not connected to the shift register 20. The first adder 31 is a three-input adder, and the third input ends of the two first adders 31 are connected to or. Two input ends of the second adder 32 are respectively connected to output ends of two shift registers 20 of another group. The two shift registers 20 of another group are two adjacent shift registers 20 in the middle. The output ends of the two first adders 31 are connected to the output ends of the second adder 32.
[0027] Among them, or is a data selection function, which selects whether the last data connected to the first adder 31 is 0 or the data after normal shifting. The data after shifting refers to the data output from the output end of the shift register 20 located in the middle. When the weight is 4 bits, only two results need to be added, so or selects an addend input to the first adder 31 as 0; when the weight is 6 bits, three results need to be added, and the last addend of the first adder 31 is the data after shifting of one of the shift registers 20 located in the middle.
[0028] In the utility model, the multi-bit weight multiplier structure is the same image data and different weight data, similar to the output channel parallelism in the convolutional neural network. When inputting data, the corresponding position weight data is loaded into the PE unit according to the channel parallel strategy, and the image data is input at the same time.
[0029] When the parameter is set to 8bitx6bit, six 8bitx2bit multiplication operations are completed in the first clock cycle, the shift operation is completed in the second clock cycle, and the addition operation of two second adders 32 is completed in the third clock cycle.
[0030] When the parameter is set to 8bitx4bit, the first clock cycle completes 6 8bitx2bit multiplication operations, the second clock cycle completes the shift operation, and the third clock cycle completes the addition operation of two first adders 31 and one second adder 32, and one input terminal of one of the first adders 31 is 0.
[0031] When the parameter is set to 8bitx2bit, six 8bitx2bit multiplication operations are completed in the first clock cycle.
[0032] It can be seen that when the weight bit width is 6 bits and 4 bits, three clock cycles are required to output a batch of calculation results. When the weight is 2 bits, a batch of calculation results can be output in one clock cycle.
[0033] The PE unit package pins of the utility model are as follows Figure 2 As shown, the specific meanings of the input and output ports are shown in Table 1.
[0034] Table 1
[0035]
[0036]
[0037] The pins after the PE unit is packaged in the utility model include a clk terminal, an rst terminal, a data terminal, a weight terminal, a choose terminal, a result terminal and a data_out terminal, wherein the clk terminal is used to input a clock signal, the rst terminal is used to input a reset signal, the data terminal is used to input image data, the weight terminal is used to input weight data, the choose terminal is used to input weight bit width selection data, the result terminal is used to output image data, and the data_out terminal is used to output calculation results.
[0038] The weight bit width selection data includes 00 representing 2 bits, 01 representing 4 bits, and 10 representing 6 bits; the calculation result is 8 bits of bit width. When the weight bit width is 2 bits, seven data are output; when the weight bit width is 4 bits, three data are output; when the weight bit width is 6 bits, two data are output.
[0039] Based on this, the utility model also proposes a multi-width bit reconfigurable convolution systolic array, including a register 40 and the above-mentioned reconfigurable convolutional neural network PE unit to form a first row of units, a second row of units and a third row of units arranged vertically, the first row of units, the second row of units and the third row of units are respectively provided with three PE units arranged from left to right and connected, the calculation result of each PE unit of the first row of units is cascaded to the corresponding PE unit of the second row of units, the calculation result of each PE unit of the second row of units is cascaded to the corresponding PE unit of the third row of units, and the calculation result of each PE unit of the third row of units is transmitted to the register 40 for data accumulation and then output; image data flows into the PE unit on the right side of each row and propagates horizontally, weight data flows into the three PE units in each row, image data and weight data flow into the second row of units one clock cycle slower than flowing into the first row of units, and image data and weight data flow into the third row of units one clock cycle slower than flowing into the second row of units.
[0040] The utility model adopts nine PE units to form a systolic array. Image data flows into the systolic array from the PE units on the right in row order. At the same time, three PE units in each row input the same weight data. For the nine weight data of 3x3 convolution, the weight data is set in an inter-row loop. For the nine weight data of 3x3 convolution, the first row of units cycles in the order of [w1, w2, w3], the second row of units cycles in the order of [w4, w5, w6] and is one clock cycle slower than the first row of units, and the third row of units cycles in the order of [w7, w8, w9] and is one clock cycle slower than the second row of units. w1, w2, w3 represent the weights of the three PE units of the first row of units, w4, w5, w6 represent the weights of the three PE units of the second row of units, and w7, w8, w9 represent the weights of the three PE units of the third row of units. The result of the multiplication of the two is passed down and added to the multiplication result of the next PE unit. A weight data and an image data are multiplied, and the result is the intermediate result. Finally, all the intermediate results are added together to obtain a 3x3 convolution result.
[0041] The utility model adopts such a setting that three PE units of each row unit in the vertical direction can complete a 3x3 convolution operation within three cycles. In the pipeline case, three vertical directions can complete three 3x3 convolution operations within three cycles.
[0042] The operation process of the systolic multi-bit width reconfigurable convolution systolic array of the utility model can be divided into three steps:
[0043] (1) Data loading phase: In the first three cycles, image data flows into the rightmost PE units of the three row units in turn, and the weight data circulates into the systolic array in a broadcast form. The systolic array starts working, and the internal PE units of the array start 2-bit multiplication, shift calculation, and accumulation calculation in turn.
[0044] (2) Circular operation stage: Starting from the fourth cycle, the data of the first row unit, the second row unit, and the third row unit complete the first calculation in turn, and the results are cascaded and transmitted downward, that is, vertically transmitted, and accumulated to the bottom register 40. Register 40 accumulates and outputs every time it is full of three output data, and then continues to wait for three new data.
[0045] (3) Data outflow stage: In this stage, when a line or image is changed, the image data of the three row units stops flowing in one by one, and the weight data is cleared from top to bottom until the last output data at the bottom of the pulsation array is accumulated. At this time, the pulsation array has been cleared, indicating that a new line or a new image data can be imported.
[0046] The circuit structure designed by the utility model is reconfigurable, and the operation mode can be determined according to the input bit width. It is compatible with the three common low bit widths of 2bit, 4bit and 6bit in convolution, and is combined with the output channel parallel in the convolution acceleration operation. It can output multi-channel data results in one operation, which greatly reduces the amount of data transport and improves the operation efficiency.
[0047] The utility model proposes a novel systolic array structure to replace the addition tree structure, and combines the sliding window with the systolic array. The data communication distance between the processing units is short, so that the design of the data flow and the control flow and the synchronous control save the IO bandwidth.
[0048] The above is only a specific implementation method of the utility model, but the design concept of the utility model is not limited to this. Any non-substantial changes to the utility model using this concept shall be deemed as an infringement of the protection scope of the utility model.
Claims
1. A PE unit of a reconfigurable convolutional neural network, characterized in that: It comprises six multipliers, five shift registers, two first adders and one second adder; the six multipliers are arranged in parallel, one input end of each multiplier inputs image data, and the other input end inputs weight data; the input ends of the five shift registers are connected to the output ends of five of the multipliers; the two input ends of one of the first adders are respectively connected to the output ends of one group of two shift registers, and the two input ends of another of the first adders are respectively connected to the output end of another shift register and the output end of the multiplier not connected to the shift register; the two input ends of the second adder are respectively connected to the output ends of another group of two shift registers, and the output ends of the two first adders are connected to the output end of the second adder.
2. The PE unit of a reconfigurable convolutional neural network according to claim 1, characterized in that: The pins after the PE unit is packaged include a clk terminal, a rst terminal, a data terminal, a weight terminal, a choose terminal, a result terminal and a data_out terminal, wherein the clk terminal is used to input a clock signal, the rst terminal is used to input a reset signal, the data terminal is used to input image data, the weight terminal is used to input weight data, the choose terminal is used to input weight bit width selection data, the result terminal is used to output image data, and the data_out terminal is used to output calculation results.
3. A PE unit of a reconfigurable convolutional neural network as claimed in claim 2, characterized in that: The weight bit width selection data includes 00 representing 2 bits, 01 representing 4 bits and 10 representing 6 bits; the calculation result is 8 bits wide, when the weight bit width is 2 bits, seven data are output; when the weight bit width is 4 bits, three data are output; when the weight bit width is 6 bits, two data are output.
4. A multi-width reconfigurable convolutional systolic array, characterized in that: A PE unit of a reconfigurable convolutional neural network comprising a register and several of claims 1 to 3.
5. A multi-width reconfigurable convolutional systolic array as claimed in claim 4, characterized in that: It includes a first row of units, a second row of units and a third row of units arranged vertically, wherein the first row of units, the second row of units and the third row of units are respectively provided with three PE units arranged from left to right and connected, the calculation result of each PE unit of the first row of units is cascaded to the corresponding PE unit of the second row of units, the calculation result of each PE unit of the second row of units is cascaded to the corresponding PE unit of the third row of units, and the calculation result of each PE unit of the third row of units is transmitted to the register for data accumulation and then output.
6. A multi-width reconfigurable convolutional systolic array as claimed in claim 5, characterized in that: The image data flows into the PE unit on the right side of each row of units and propagates laterally, the weight data flows into the three PE units of each row of units, the image data and the weight data flow into the second row of units one clock cycle slower than into the first row of units, and the image data and the weight data flow into the third row of units one clock cycle slower than into the second row of units.
7. The multi-width reconfigurable convolutional systolic array of claim 5, wherein: The weight data is set for inter-row circulation. For the nine weight data of 3x3 convolution, the first row of units is circulated in the order of [w1, w2, w3], w1, w2, w3 respectively represent the weights of the three PE units of the first row of units, the second row of units is circulated in the order of [w4, w5, w6] and is one clock cycle slower than the first row of units, w4, w5, w6 respectively represent the weights of the three PE units of the second row of units, the third row of units is circulated in the order of [w7, w8, w9] and is one clock cycle slower than the second row of units, w7, w8, w9 respectively represent the weights of the three PE units of the third row of units.
