Linear vector decomposition AI processing architecture and use method thereof

Abstract

The invention discloses a linear vector decomposition AI processing architecture which comprises a vector convolution unit, a vector full connection unit and a vector register accumulation unit, no data cache storage unit exists between the units, no input data cache structure exists in front of the vector convolution unit, and convolution calculation is a local convolution window. The vector register accumulation unit is composed of a state selection unit, a register addition unit and a selection output unit, the state selection unit is used for specifying storage, accumulation and output states, configuring an effective data address by specifying the position of input data, and when the position of the input data is matched with the effective address, outputting the input data; and the vector register accumulation unit stores, adds or outputs data according to the input state at the moment, and selects the output unit to output the calculation result of each position, otherwise, the output is 0. According to the invention, the cache occupation time of the linear layer and the connection of the linear layer can be reduced, the output integrity can be realized, and the real-time performance of data transmission and calculation can be ensured at the same time.

Description

Technical Field

[0001] The present invention belongs to the technical field of neural network processing unit architecture, and specifically relates to a linear vector decomposition AI processing architecture and a method of using the same. Background Art

[0002] For the calculation of artificial intelligence neural network, most of them are parallelized by NPU, DSP, GPU, FPGA and other units. The traditional structure caches the input in rows, and uses complete convolution window units to calculate the output when performing linear calculation tasks such as convolution and full connection. After the output result of this layer is complete, the calculation of the next layer is performed. This computing architecture consumes storage units because the input data is huge and needs to be cached. In addition, the post-processing of the neural network is completed only after the pre-processing of the neural network is completed, which causes the neural network calculation from input completion to the entire neural network processing output, the corresponding output time is long, and the real-time performance is poor due to poor pipeline parallelism. Under this complete output calculation processing method, in parallel processors based on CPU as the control core, the real-time performance is poor due to the lack of pipeline parallelism; and in processors based on FPGA, due to limited computing resources and limited storage units, the cache and data parallelism between neural network layers are poor, resulting in the real-time performance of neural network calculations not being able to meet the requirements well.

[0003] In addition, neural network structures emerge in an endless stream. ASICs represented by NPUs have a fixed structure and are inefficient in executing algorithmic tasks that are not suitable for their own processor structures. Therefore, they cannot meet application scenarios with diverse algorithms. Summary of the invention

[0004] The purpose of the present invention is to provide a linear vector decomposition AI processing architecture and its use method, which can reduce the cache occupancy time of linear layers and linear layer connections, achieve output integrity, and ensure the real-time performance of data transmission and calculation.

[0005] To achieve the above-mentioned purpose, the present invention provides a linear vector decomposition AI processing architecture, including a vector convolution unit, a vector fully connected unit, and a vector register accumulation unit. There is no input data cache structure in front of the vector convolution unit, and the convolution calculation is a local convolution window. There is no data cache storage unit between the vector convolution unit, the vector fully connected unit, and the vector register accumulation unit. The vector register accumulation unit is composed of a state selection unit, a register addition unit, and a selection output unit. The state selection unit is used to specify storage, accumulation, and output states, and configures a valid data address by specifying the position of input data. When the position of the input data matches the valid address, the vector register accumulation unit saves, adds, or outputs data according to the input state at this time, and selects the output unit to output the calculation results of each position, otherwise the output is 0.

[0006] As a further solution of the present invention: the input data of the vector convolution unit is a single-row normalized value of the length of the RGB convolution window of a row of image data, the convolution step size of the convolution layer is the same as the convolution window length, and there are 4 batches of vector convolution multiplication-addition arrays in the vector convolution unit to form a convolution vector group. The number of multipliers in each batch of vector convolution multiplication-addition arrays is the same as the single-row quantity of the input RGB convolution window length. After each batch of vector convolution multiplication-addition arrays performs parameter multiplication operations, addition tree operations are performed and then output as the vector multiplication-addition operation results of this batch.

[0007] As a further solution of the present invention: the number of vector fully-connected multiplication-addition groups in the vector fully-connected unit is three times the output channels of the convolutional layer, the input data amount of each vector fully-connected multiplication-addition group matches the output data of the previously connected convolutional vector group, and each vector fully-connected multiplication-addition group has 4 data inputs.

[0008] As a further solution of the present invention: the vector storage accumulation unit has a total of 24 batches of state selection units, storage addition units and selection output units; driven by the clock and the valid mark position, the input data needs to specify the position of the input data, and the valid position is specified in advance to indicate the valid position. When the position of the input data is the same as the valid position, the data is saved, added or output, and the output unit is selected to output the calculation results of each position, otherwise the output is 0; the effective addresses are different, and the calculation results of different output units are output by single data through addition, and the parallel data is converted into serial output in the form of single data through 0 value and or logic.

[0009] The method for using the above-mentioned linear vector decomposition AI processing architecture includes the following steps:

[0010] S31, connecting the data input and parameter input interfaces of the vector convolution unit to the input image matrix generator, the input image matrix generator is used to simulate the data generation of the output image; the data output of the convolution layer reordering parameter output device is connected to the parameter input interface of the vector convolution unit, the matrix control unit controls the data multiplication and addition order of the input image matrix generator and the convolution layer reordering parameter output device to be consistent; driven by the clock and reset signals, the output data is the data and valid flag bus;

[0011] S32, connecting the parameter input end of the vector fully connected unit and the linear layer reordering parameter output device, connecting the data input of the vector fully connected unit and the data output of the vector convolution unit, controlling the synchronization of the data through the valid data flag of the vector convolution unit, and the output of the vector fully connected unit is the valid bit and the calculated data;

[0012] S33. Connect the data input end of the vector storage accumulation unit to the vector fully connected unit of the previous stage, connect the control line to the accumulation processing control unit, enable the accumulation processing control unit through open-loop control by the effective signal of the previous stage network, and the accumulation processing control unit realizes the output superposition of the data storage position to the correct position; the output of the vector storage accumulation unit is the valid data flag and data output.

[0013] Compared with the prior art, the present invention has the following beneficial effects:

[0014] The present invention adopts local row vector input calculation, which effectively eliminates the resource consumption of a large amount of data cache required by traditional computing architecture before neural network calculation. The data is convolved and fully connected in the form of vectors, and the bandwidth between network layers is matched, which ensures the real-time performance of the calculation, greatly reduces the consumption of storage units between network layers, and can reduce the cache occupancy time of linear layers and linear layer connections. After the linear layer calculation is completed, the scattered vector form is accumulated into a complete output structure through the vector storage accumulation unit, which ensures the integrity and correctness of the calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a schematic diagram of the linear vector decomposition AI processing architecture of the present invention;

[0016] Figure 2 for Figure 1 A magnified schematic diagram of the vector convolution unit;

[0017] Figure 3 for Figure 1 A magnified schematic diagram of the fully connected unit in the mid-vector;

[0018] Figure 4 for Figure 1 A magnified schematic diagram of the vector storage accumulation unit;

[0019] Figure 5 It is a vector convolution multiply-add array diagram;

[0020] Figure 6 is a vector fully connected multiply-add group;

[0021] Figure 7 It is the basic unit structure diagram in the vector storage accumulation unit;

[0022] Figure 8 It is the structure diagram of the neural network model;

[0023] Fig. 9 It is the timing diagram of input data;

[0024] Fig.10 Enter the timing diagram for the convolution parameters;

[0025] Fig.11 Output timing diagram of the vector register accumulation unit output layer. DETAILED DESCRIPTION

[0026] The present invention will be further described below by way of examples.

[0027] like Figures 1 to 4 As shown, a linear vector decomposition AI processing architecture includes a vector convolution unit, a vector fully connected unit, and a vector register accumulation unit. There is no input data cache structure in front of the vector convolution unit, and the convolution calculation is a local convolution window. There is no data cache storage unit between the vector convolution unit, the vector fully connected unit, and the vector register accumulation unit. The vector register accumulation unit is composed of a state selection unit, a register addition unit, and a selection output unit. The state selection unit is used to specify the storage, accumulation, and output states. By specifying the position of the input data, the effective data address is configured. When the position of the input data matches the effective address, the vector register accumulation unit saves, adds, or outputs the data according to the input state at this time, and selects the output unit to output the calculation results of each position, otherwise the output is 0. If the effective addresses are different, the calculation results of different output units can be output by single data through addition.

[0028] Furthermore, the input data of the vector convolution unit is a single-row normalized value of the length of the RGB convolution window of a row of image data. The convolution step length of the convolution layer is the same as the convolution window length. There are 4 batches of vector convolution multiplication and addition arrays in the vector convolution unit to form a convolution vector group. The vector convolution multiplication and addition array is as follows: Figure 5 As shown, Figure 5 The module name is conv2d; i_clk is the input driving clock, one_line_mul implements the multiplication function, i_din_b, i_din_g, i_din_r in the input represent the three RGB components of the input normalized image vector, i_weight_b, i_weight_g, i_weight_r represent the RGB corresponding weight parameters of the convolution layer in the neural network respectively; mult_out_b, mult_out_g, mult_out_r represent the multiplication output results corresponding to the RGB channels respectively; zonghe_test is the addition unit, which adds mult_out_b, mult_out_g, and mult_out_r to obtain the output add_out; the number of multipliers in each batch of vector convolution multiplication and addition arrays is the same as the single row of the input RGB convolution window length. After each batch of vector convolution multiplication and addition arrays performs parameter multiplication operations, the addition tree operation is superimposed and then output as the vector multiplication and addition operation results of this batch.

[0029] Furthermore, the number of vector fully connected multiplication and addition groups in the vector fully connected unit is three times the number of output channels of the convolutional layer. Figure 6 As shown, Figure 6 The module name is linear_iclk_1_in_weight, which consists of a multiplication unit linear_mul_1clk_one_p_weight and an addition unit linear_add_1clk_one_p_weight. i_linear_weight and i_relu_date represent the weight parameter and the input data respectively, and o_linear_add represents the result after addition. i_clk and o_valid represent the driving clock and data valid signal flags respectively. The input data amount of each vector fully connected multiplication-addition group matches the output data of the previously connected convolution vector group, and each vector fully connected multiplication-addition group has 4 data inputs.

[0030] Furthermore, the vector register accumulation unit has 24 batches of state selection units, register addition units and selection output units; Figure 7 The following is the basic unit structure diagram of the vector storage accumulation unit. Figure 7 The name of the structural unit is expressed as sum_x_one_data_unit_parameterized4; i_clk represents the driving clock signal, i_conv_nobias represents the input data signal, i_din_cnt represents the data position or position number corresponding to the input data signal, i_state represents the state flag signal of whether the input data is written or accumulated, valid represents the valid signal flag of the input signal, o_conv_nobias_sum represents the final calculation result output after accumulation, and o_valid represents the flag bit of the output final accumulated number complete. In the detailed original, REG, ADD, MUX, ROM, EQ, and AND represent triggers, adders, multiplexers, read-only registers, data comparators, and AND gates, respectively; driven by the clock and the valid flag position, the input data needs to specify the position of the input data, and specify the valid position in advance to indicate the valid position. When the position of the input data is the same as the valid position, the data is saved, added, or output, and the output units are selected to output the calculation results of their respective positions, otherwise the output is 0; the effective addresses are different, and the calculation results of different output units are output by single data through addition, and the parallel data is converted into serial output in the form of single data through 0 value and or logic.

[0031] The method for using the above-mentioned linear vector decomposition AI processing architecture includes the following steps:

[0032] S31, connecting the data input and parameter input interfaces of the vector convolution unit to the input image matrix generator, the input image matrix generator is used to simulate the data generation of the output image; the actual working circuit uses a camera sensor instead; the data output of the convolution layer sequencing parameter output device is connected to the parameter input interface of the vector convolution unit, the matrix control unit controls the data multiplication and addition sequence of the input image matrix generator and the convolution layer sequencing parameter output device to be consistent; driven by the clock and reset signals, the output data is the data and valid flag bus;

[0033] S32, connecting the parameter input end of the vector fully connected unit and the linear layer reordering parameter output device, connecting the data input of the vector fully connected unit and the data output of the vector convolution unit, controlling the synchronization of the data through the valid data flag of the vector convolution unit, and the output of the vector fully connected unit is the valid bit and the calculated data;

[0034] S33. Connect the data input end of the vector storage accumulation unit to the vector fully connected unit of the previous stage, connect the control line to the accumulation processing control unit, enable the accumulation processing control unit through open-loop control by the effective signal of the previous stage network, and the accumulation processing control unit realizes the output superposition of the data storage position to the correct position; the output of the vector storage accumulation unit is the valid data flag and data output.

[0035] Specifically, take the image information of the RGB three channels of a single row vector of the convolution window size and input it to 4 batches of multiplication arrays. The data input of these 4 batches of multiplication arrays is the same, and the parameters are the parameters of different channels in the neural network. The output single row of convolution vectors is directly connected to the vector fully connected unit as data input. There are 24 groups of vector fully connected multiplication and addition groups in the vector fully connected unit. The input data of each group is the same, but the calculation parameters are different from each other as the parameters of the neural network full connection. The output of the fully connected layer is 24 data parallel outputs. The data is input to the vector storage accumulation unit, and then respectively input to 24 batches of state selection units, storage addition units and selection output units for corresponding superposition and output operations.

[0036] The vector convolution unit assumes that the input RGB data are:

[0037]

[0038] Assume that the input network parameter of a batch of vector convolution multiplication and addition arrays is W cn , where n represents the number of the batch of vector convolution multiplication and addition arrays, and there are four batches in total, so n can be 1, 2, 3, or 4. cn for:

[0039]

[0040] Suppose the output of the batch of vector convolution multiplication and addition arrays after the addition tree is yc n ,but:

[0041]

[0042] The driving clock of the vector convolution unit is 1clk. The calculation starts when the data is valid. The valid data output of the entire calculation unit is yc, which is expressed as:

[0043]

[0044] When the driving clock of the vector fully connected unit is valid, assuming that the input weight is W l , within an effective clock, the vector parameters are set to l1, l2, …, l 24 ,l n Expressed as:

[0045] l n =[l n-1 ,l n-2 ,l n-3 ,l n-4 ];

[0046] Therefore, the input weight of the vector fully connected unit is W l It can be expressed as:

[0047]

[0048] According to the fully connected calculation rule, that is, the matrix multiplication rule, let the output data of the vector fully connected unit be yl, then yl is:

[0049] yl=W l ·yc;

[0050]

[0051] The initial value and reset value of the vector storage accumulation unit are both 0. When the driving clock is valid, the input state control is set to m. When m is 0, it means the input value is stored, 1 means the original state superimposed input but the result is incomplete, and 2 means the added input vector is complete. The output value of the last clock storage accumulation unit is yo. At this time, the storage accumulation unit state is s, and the output at this moment is yo', then:

[0052]

[0053] The neural network model structure implemented in the present invention is as follows: Figure 8As shown in the figure, the algorithm model is designed with the entire image as RGB three-channel data, the data tensor size is 3*9*9, the convolution step size of the convolution layer is the same as the convolution window size, the convolution parameter tensor size is 8*3*3, and the data input after the convolution operation is 8*3*3. After dimensionality reduction and splicing, the model dimension is 9*8, the fully connected data dimension is 24*8, and after the fully connected layer, it is 9*24. In the entire neural network, the linear layer composition does not contain bias terms.

[0054] Embodiment 1:

[0055] The width of the vector convolution unit is 4, and the width of the convolution layer is 8. When the input data is valid, the two clock data need to be kept unchanged. The same data is processed with two clocks using different parameters. In the data tensor after splicing and dimension reduction, two consecutive clocks represent a batch output of a row of data. That is, the input RGB input data is replaced every two clock cycles, and the RGB pixels of three consecutive positions of a row of RGB data are transmitted each time. In each clock cycle, the corresponding position parameters of the adjacent 4 width layers of the convolution layer RGB parameters that match the input data are transmitted in parallel, such as Fig. 9 , Fig.10 shown.

[0056] Embodiment 2:

[0057] The result of the vector convolution unit output is used as the data input of the vector fully connected unit. The input of the multiplication and addition unit of the vector fully connected unit is the result of the vector convolution unit output, which is the same as each other. The input parameter of each effective clock of this architecture is the fully connected parameter weight of each position. The cycle is performed once every 2 clock cycles. The output result of the vector fully connected unit is output to the vector register accumulation unit. When the image input data is the first row corresponding to the convolution window, the control state of the vector register accumulation unit is 0. When the image input data is the last row corresponding to the convolution window, the control state of the vector register accumulation unit is 2, and the other control states are 1. The output of the data is calculated when the control state is 2. The accumulation calculation is completed when the control state is 2, and the completion data flag is valid. Fig.11 As shown. Take the data output of rows 1 to 4, and the superposition result of two consecutive valid data in each row is a valid data after convolution and full connection. Every two clocks, a calculation result of the neural network of the 4 rows is output in parallel. After accumulating the results of the vector register accumulation unit, it is completely consistent with the calculation result executed by the software.

Claims

1. A linear vector decomposition AI processing architecture, comprising a vector convolution unit, a vector fully connected unit, and a vector register accumulation unit, characterized in that: There is no input data cache structure in front of the vector convolution unit, and the convolution calculation is a local convolution window. There is no data cache storage unit between the vector convolution unit, the vector fully connected unit, and the vector register accumulation unit. The vector register accumulation unit is composed of a state selection unit, a register addition unit, and a selection output unit. The state selection unit is used to specify the storage, accumulation, and output states. By specifying the location of the input data, the effective data address is configured. When the location of the input data matches the effective address, the vector register accumulation unit saves, adds, or outputs the data according to the input state at this time, and selects the output unit to output the calculation results of their respective positions, otherwise the output is 0.

2. A linear vector decomposition AI processing architecture according to claim 1, characterized in that: The input data of the vector convolution unit is a single-row normalized value of the length of the RGB convolution window of a row of image data. The convolution step of the convolution layer is the same as the convolution window length. The vector convolution unit is composed of 4 batches of vector convolution multiplication-addition arrays. The number of multipliers in each batch of vector convolution multiplication-addition arrays is the same as the single-row quantity of the input RGB convolution window length. After each batch of vector convolution multiplication-addition arrays performs parameter multiplication operations, the addition tree operation is superimposed and then output as the vector multiplication-addition operation results of this batch.

3. A linear vector decomposition AI processing architecture according to claim 2, characterized in that: The number of vector fully connected multiplication-addition groups in the vector fully connected unit is three times the number of output channels of the convolutional layer. The amount of input data of each vector fully connected multiplication-addition group matches the output data of the previously connected vector convolution multiplication-addition array, and each vector fully connected multiplication-addition group has 4 data inputs.

4. A linear vector decomposition AI processing architecture according to claim 3, characterized in that: The vector register accumulation unit has 24 batches of state selection units, register addition units and selection output units. It is driven by the clock and the valid flag position. The input data needs to specify the position of the input data, and specify the valid position in advance to indicate the valid position. When the position of the input data is the same as the valid position, the data is saved, added or output, and the output unit is selected to output the calculation results of each position, otherwise the output is 0. The effective addresses are different, and the calculation results of different output units are output as a single data through addition, and the parallel data is converted into a serial output in the form of a single data through 0 value and or logic.

5. A method for using a linear vector decomposition AI processing architecture according to any one of claims 1 to 4, characterized in that: The following steps are involved: S31, connecting the data input and parameter input interfaces of the vector convolution unit to the input image matrix generator, the input image matrix generator is used to simulate the data generation of the output image; the data output of the convolution layer reordering parameter output device is connected to the parameter input interface of the vector convolution unit, the matrix control unit controls the data multiplication and addition order of the input image matrix generator and the convolution layer reordering parameter output device to be consistent; driven by the clock and reset signals, the output data is the data and valid flag bus; S32, connecting the parameter input end of the vector fully connected unit and the linear layer reordering parameter output device, connecting the data input of the vector fully connected unit and the data output of the vector convolution unit, controlling the synchronization of the data through the valid data flag of the vector convolution unit, and the output of the vector fully connected unit is the valid bit and the calculated data; S33. Connect the data input end of the vector storage accumulation unit to the vector fully connected unit of the previous stage, connect the control line to the accumulation processing control unit, enable the accumulation processing control unit through open-loop control by the effective signal of the previous stage network, and the accumulation processing control unit realizes the output superposition of the data storage position to the correct position; the output of the vector storage accumulation unit is the valid data flag and data output.

Images