Matmul layer data processing method based on NNA2.0

Abstract

The invention provides a matmul layer data processing method based on NNA2.0. The method comprises the following steps: S1, weight rearrangement; s2, data rearrangement is carried out; s3, performing convolution operation summation by using NNA; and S4, parallel computing. Matmul calculation logic is converted into convolution, NNA is used for acceleration, and calculation efficiency is improved. The method is simple, the waiting time is reduced, and the calculation efficiency of the NNA is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a matmul layer data processing method based on NNA2.0. Background Art

[0002] In the prior art, the NNA technology is implemented to accelerate the convolution multiplication and addition calculation. The hardware accelerator on the CPU's SIMD pipeline is controlled by special CPU / SIMD instructions and runs on one thread to solve most of the convolution multiplication and addition, and supports quantization of the final result. Matrix dot multiplication Matmul is the sum of row and column dot multiplication to get the output point. MatMul implementation logic: rslt0 = w00*f00+w01*f10+......+w0n*fn0; (a total of n pixels are multiplied and summed).

[0003] However, the main defects of the existing technology are: the data is discontinuous, which makes parallel calculation impossible, has many steps and is slow.

[0004] In addition, common terms in the prior art include:

[0005] 1. feature_shape: the shape of the input data, abstractly understood as multi-dimensional data, from left to right, from high dimension to low dimension (NDHWC) N: the number of FeatureMaps processed at a time, set to 1; D: the number of 32 groups of input channels, H: the height of the input data, W: the width of the input data, C: 32.

[0006] 2. MAC: NNA-based convolution, accumulation, and calculation

[0007] 3. IC, OC: input channel and output channel, generally used for channel description of convolution kernel.

[0008] 4. FRAM: NNA is used to store input data space. Summary of the invention

[0009] In order to solve the above problems, the purpose of this application is to convert the Matmul calculation logic into convolution, use NNA for acceleration, and improve calculation efficiency.

[0010] Specifically, the present invention provides a matmul layer data processing method based on NNA2.0, the method comprising the following steps:

[0011] S1: Weight rearrangement, the 0th input source is understood as the weight saved by the convolution kernel, the outer loop is N, and 9 weights can be calculated in N cycles; the inner loop is M, M can be any value, 9 weights are calculated each time, and if the number is less than 9, the remaining positions can be filled with 0 according to the MASK mask of NNA;

[0012] Assume it is a row with 9 weights;

[0013] W00 W01 W02 W03 W04 W05 W06 W07 W08

[0014] Rearrange to a 3x 3 shape to facilitate NNA convolution calculation:

[0015] W00 W01 W02 W03 W04 W05 W06 W07 W08

[0016] Each input channel retains a valid data. The convolution logic is to multiply and accumulate all channels. Matmul only multiplies the current weight by the feature, and the influence of other channels needs to be removed.

[0017] Ic0 Ic1 Ic2 Ic3 Ic4 Ic5 Ic6 ... Ic31 Oc W00 0 0 0 0 0 0 ... 0 Oc1 0 W00 0 0 0 0 0 ... 0 ... Oc31 0 0 0 0 0 0 0 ... W00

[0018] Keep one weight channel valid and set the rest of the weight channels to 0, so that the accumulation in the channel direction after multiplication and summation will not cause abnormal results;

[0019] S2: Data rearrangement. The first input source feature1 is understood as the input data in the convolution process. The number of rows can be any value. It is assumed to be 9 rows and 32 channels. The 9 rows correspond to the 9 weights of the channels in step S1. The channels are multiples of 32 for one cycle. If they are not multiples of 32, they can be calculated separately:

[0020] F00 F01 F02 F03 F04 F05 F06 ... F0,31 F10 F11 F12 F13 F14 F15 F16 ... F1,31 F20 F21 F22 F23 F24 F25 F26 ... F2,31 ... F8,0 F8,1 F8,2 F8,3 F8,4 F8,5 F8,6 ... F8,31

[0021] Load 32 channels chn in sequence and store them into NNA storage input data space FRAM, treating each 32 chn as a pixel.

[0022] Then rearrange the 9 pixels in the form of a convolution kernel:

[0023] Assuming 8-bit data:

[0024] LA(o,VR0,0,in_ptr,0); row 0 data;

[0025] LA(o,VR0,1,in_ptr+ichn,0); data of the first row;

[0026] LA(o,VR0,0,in_ptr+2*ichn,0); data of the second row;

[0027] Read F00 F10 F20 data from the input address;

[0028] NNDWF0(VR0)

[0029] NNDWF0(VR1)

[0030] .......Loop to read the following channels and calculate 8 times in parallel;

[0031] Store the data of VR0 and VR1 in FRAM, configure the NNA register in advance, and jump to the next row after reading a fixed pixel. Assuming it is set to 8, NNDWF0 (VR) will automatically jump to the next row FRAM address 8 times:

[0032] LA(o,VR0,0,in_ptr+3*ichn,0); data in the third row;

[0033] LA(o,VR0,1,in_ptr+4*ichn,0); data in the 4th row;

[0034] LA(o,VR0,0,in_ptr+5*ichn,0); data of the 5th row;

[0035] NNDWF0(VR0)

[0036] NNDWF0(VR1)

[0037] Read F30 F40 F50 data from the input address;

[0038] LA(o,VR0,0,in_ptr+6*ichn,0); data of the 6th row;

[0039] LA(o,VR0,1,in_ptr+7*ichn,0); data in row 7;

[0040] LA(o,VR0,0,in_ptr+8*ichn,0); data in row 8;

[0041] NNDWF0(VR0)

[0042] NNDWF0(VR1)

[0043] Read F60 F70 F80 data from the input address;

[0044] In this way, the corresponding weights are multiplied by the corresponding data to obtain the calculation logic of matmul:

[0045] F00 F10 F20 F30 F40 F50 F60 F70 F80 ;

[0047] S3: Use NNA to perform convolution operations and sum: Due to the hardware settings, the stride of the convolution kernel is set to 2:

[0048]

[0049] F00 to F80 in the upper left box of the above table represent channels 0 to 31;

[0050] F00 to F80 in the middle upper frame of the above table represent channels 32 to 63;

[0051] When NNA is calculating, the data of the upper left corner box and the upper middle box area will be multiplied by the weight at the same time, realizing the idea of ​​weight reuse and reducing the operation of repeated weight reading:

[0052] Use NNDWF0 to store F00~F80, 0~63 channel data into FRAM;

[0053] Use NNDWW to store weight data in FRAM;

[0054] Use the NNMACG instruction to multiply the weight data and the input data, and use the same WRAM weight data for channels 0 to 31 and 32 to 63.

[0055] S4: Parallel calculation: Write the first convolution calculation, i.e., the data of channels 0 to 63, into FRAM0 in advance, and then perform NNA convolution operation MAC00. Use NNMACG (MAC00, OPT_NNMAC_WRAING, 1, OPT_NNMAC_FP_NUM0) to start calculating MAC00. During the MAC process, write the data of channels 64 to 127 into FRAM1, and store the data of FRAM0 into the temporarily allocated space tmp, and then MAC01. Alternately, MAC00 takes time and can be performed in parallel with simd instructions. The operation is as follows:

[0056] The calculation time of MAC00 is used for reading data of MAC01, and the calculation time of MAC01 is used for reading data of MAC00. The two spaces are used for ping-pong calculation.

[0057] F00 to F80 in the lower left box of the above table represent channels 64 to 95;

[0058] F00 to F80 in the lower middle box in the above table represent channels 96 to 127;

[0059] The 64 channels that need to be calculated next time are stored in the NNA FRAM space in advance, so that the time for NNA to perform convolution operation is used to read the data for the next convolution operation, thus improving the calculation efficiency of NNA;

[0060] After all channels are calculated, the data is stored in a tmp, the weight is updated, and the above steps are repeated, which is expressed as:

[0061] LA(o,VR0,0,in_ptr+ichn,0);

[0062] LA(o,VR0,1,in_ptr+1*ichn,0);

[0063] LA(o,VR0,0,in_ptr+2*ichn,0);

[0064] NNDWF0(VR0)

[0065] NNDWF0(VR1)

[0066] Store the remaining row data in FRAM, update the FRAM data, then perform MAC calculation, and then add the result to the tmp data until all weight calculations are completed, expressed as: NNDRDOCB (VR0, OPT_NNDRDOCB_PIXEL0, NNDRDOCB_LO); store the NNA calculation result in the VR0 register

[0067] LA(o,VR1,0,tmp_ptr,0); read tmp space data

[0068] ADD(w,VR2,VR0,VR1); tmp data plus the result of this calculation

[0069] SA(o,VR2,0,tmp_ptr,0); Update the data in tmp space. So far, the result of multiplying and accumulating all weights and channels is obtained.

[0070] If the number in step S2 is not a multiple of 32, it can be calculated separately using C++ code.

[0071] The step size stride=2 is based on the NNA hardware function. The output shape is 1x4, generating 4 points at a time, 2 of which are valid pxiel data generation; if it is NNA2.0, the output shape is 1X8, generating 8 output points at a time, 4 of which are valid data.

[0072] The matmul implementation logic is the lowest two dimensions, namely w: width and c: channel, multiplying and accumulating to get the final result.

[0073] Therefore, the advantages of the present application are: the method is simple, the waiting time is reduced, and the calculation efficiency of the NNA is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0075] Figure 1 It is a flow chart of the present application method. DETAILED DESCRIPTION

[0076] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.

[0077] The embodiment of the present application relates to a matmul layer data processing method based on NNA2.0, and the Matmul implementation logic is: the lowest two dimensions (w: width, c: channel) are multiplied and added to obtain the final result.

[0078] Ping-pong idea: FRAM is divided into two areas FRAM0 and FRAM1. NNA needs a certain amount of time to perform convolution operation. This time is used to read the data that needs convolution operation next time. The two operations are performed alternately to reduce waiting time and improve the computing efficiency of NNA.

[0079] example:

[0080] Feature0[3,2]*feature1[2,4]=out[3,4], the 0th feature0 is regarded as the weight data of the convolution, and the 1st feature1 is regarded as the input data of the convolution.

[0081]

[0082] The output result points are obtained by respectively adding the following weights W and input data F.

[0083] R00=w00*F00+w01*F10

[0084] R01=w00*F01+w01*F11

[0085] R02=w00*F02+w01*F12

[0086] R03=w00*F03+w01*F13 .....

[0088] R20=w20*F00+w21*F10

[0089] R21=w20*F01+w21*F11

[0090] R22=w20*F02+w21*F12

[0091] R23=w20*F03+w21*F13

[0092] Weight sharing, multiplication and summation are performed with all input sources.

[0093] like Figure 1 As shown, the overall calculation logic is divided into the following steps:

[0094] S1: Weight rearrangement (the 0th input source is understood as the weight saved by the convolution kernel). Taking one row as an example, there are 9 weights.

[0095] W00 W01 W02 W03 W04 W05 W06 W07 W08

[0096] Rearrange to a 3x3 shape to facilitate NNA convolution calculation:

[0097]

[0098]

[0099] Each input channel retains a valid data. The convolution logic is to multiply and accumulate all channels. Matmul only multiplies the current weight by the feature, and the influence of other channels needs to be removed.

[0100] Ic0 Ic1 Ic2 Ic3 Ic4 Ic5 Ic6 ... Ic31 Oc W00 0 0 0 0 0 0 ... 0 Oc1 0 W00 0 0 0 0 0 ... 0 ... Oc31 0 0 0 0 0 0 0 ... W00

[0101] Keep one weight channel valid and set the rest of the weight channels to 0, so as to ensure that the accumulation in the channel direction after multiplication and summation will not cause abnormal results.

[0102] S2: Data rearrangement (the first input source feature1 is understood as the input data in the convolution process), taking 9 rows and 32 channels as an example; the 9 rows correspond to the 9 weights of the channels in step S1, and the channels are multiples of 32 for one cycle. If it is not a multiple of 32, it can be calculated separately using C++ code:

[0103] F00 F01 F02 F03 F04 F05 F06 ... F0,31 F10 F11 F12 F13 F14 F15 F16 ... F1,31 F20 F21 F22 F23 F24 F25 F26 ... F2,31 ... F8,0 F8,1 F8,2 F8,3 F8,4 F8,5 F8,6 ... F8,31

[0104] Load 32 chn in sequence and store them into FRAM (NNA storage input data space), treating each 32 chn as a pixel.

[0105] Then rearrange the 9 pixels in the form of a convolution kernel:

[0106] Assuming 8-bit data:

[0107] LA(o,VR0,0,in_ptr,0); row 0 data;

[0108] LA(o,VR0,1,in_ptr+ichn,0); data of the first row;

[0109] LA(o,VR0,0,in_ptr+2*ichn,0); data of the second row;

[0110] Read F00 F10 F20 data from the input address;

[0111] NNDWF0(VR0)

[0112] NNDWF0(VR1)

[0113] .......Loop to read the following channels and calculate 8 times in parallel;

[0114] Store the data of VR0 and VR1 in FRAM, configure the NNA register in advance, and jump to the next row after reading a fixed pixel. Assuming it is set to 8, NNDWF0 (VR) will automatically jump to the next row FRAM address 8 times:

[0115] LA(o,VR0,0,in_ptr+3*ichn,0); data in the third row;

[0116] LA(o,VR0,1,in_ptr+4*ichn,0); data in the 4th row;

[0117] LA(o,VR0,0,in_ptr+5*ichn,0); data of the 5th row;

[0118] NNDWF0(VR0)

[0119] NNDWF0(VR1)

[0120] Read F30 F40 F50 data from the input address;

[0121] LA(o,VR0,0,in_ptr+6*ichn,0); data of the 6th row;

[0122] LA(o,VR0,1,in_ptr+7*ichn,0); data in row 7;

[0123] LA(o,VR0,0,in_ptr+8*ichn,0); data in row 8;

[0124] NNDWF0(VR0)

[0125] NNDWF0(VR1)

[0126] Read F60 F70 F80 data from the input address;

[0127] In this way, the corresponding weights are multiplied by the corresponding data to obtain the calculation logic of matmul.

[0128] F00 F10 F20 F30 F40 F50 F60 F70 F80

[0129] S3: Use NNA to perform convolution operations and sum. There are differences in the use of instructions between NNA2.0 and NNA1.0, but the overall idea remains the same: Due to hardware settings, the stride of the convolution kernel is set to 2.

[0130]

[0131] F00 to F80 in the upper left box of the above table represent channels 0 to 31;

[0132] F00 to F80 in the middle upper frame of the above table represent channels 32 to 63;

[0133] When NNA is calculating, the data of the upper left corner box and the upper middle box area will be multiplied by the weight at the same time, realizing the idea of ​​weight reuse and reducing the operation of repeated weight reading:

[0134] Use NNDWF0 to store F00~F80, 0~63 channel data into FRAM;

[0135] Use NNDWW to store weight data in FRAM;

[0136] Use the NNMACG instruction to multiply the weight data and the input data, and use the same WRAM weight data for channels 0 to 31 and 32 to 63.

[0137] Step 4: Parallel calculation; write the data of the first convolution calculation (channels 0 to 63) into FRAM0 in advance, then perform MAC00 (NNA convolution operation), write the data of FRAM1 (channels 64 to 127) during the MAC process, and store the data of FRAM0 into tmp, then MAC01, alternately; MAC00 takes time and can be parallel with simd instructions, the operation is as follows: MAC00 calculation time is used for MAC01 data reading, MAC01 calculation time is used for MAC00 data reading, two blocks of space ping-pong calculation;

[0138] F00 to F80 in the lower left box of the above table represent channels 64 to 95;

[0139] F00 to F80 in the lower middle box in the above table represent channels 96 to 127;

[0140] The 64 channels that need to be calculated next time are stored in the NNA FRAM space in advance, so that the time for NNA to perform convolution operation is used to read the data for the next convolution operation, thus improving the calculation efficiency of NNA;

[0141] After all channels are calculated, the data is stored in a tmp (temporarily opened space), the weight is updated, and the above steps are repeated, which is expressed as:

[0142] LA(o,VR0,0,in_ptr+ichn,0);

[0143] LA(o,VR0,1,in_ptr+1*ichn,0);

[0144] LA(o,VR0,0,in_ptr+2*ichn,0);

[0145] NNDWF0(VR0)

[0146] NNDWF0(VR1)

[0147] Store the remaining row data in FRAM, update the FRAM data, then perform MAC calculation, and then add the result to the tmp data until all weight calculations are completed, expressed as: NNDRDOCB (VR0, OPT_NNDRDOCB_PIXEL0, NNDRDOCB_LO); store the NNA calculation result in the VR0 register

[0148] LA(o,VR1,0,tmp_ptr,0); read tmp space data

[0149] ADD(w,VR2,VR0,VR1); tmp data plus the result of this calculation

[0150] SA(o,VR2,0,tmp_ptr,0); Update the data in tmp space. So far, the result of multiplying and accumulating all weights and channels is obtained.

[0151] stride=2, the output shape is 1x4 (according to the NNA hardware function, the NNA2.0 output shape is 1X8, generating 8 output points at a time, 4 of which are valid data), generating 4 points at a time, 2 of which are valid pxiel data generation; if the NNA2.0 output shape is 1X8, 8 output points are generated at a time, 4 of which are valid data.

[0152] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A matmul layer data processing method based on NNA2.0, characterized in that: The method comprises the following steps: S1: Weight rearrangement, the 0th input source is understood as the weight saved by the convolution kernel, the outer loop is N, and 9 weights can be calculated in N cycles; the inner loop is M, M can be any value, 9 weights are calculated each time, and if the number is less than 9, the remaining positions can be filled with 0 according to the MASK mask of NNA; Assume it is a row with 9 weights; Rearrange to a 3x3 shape to facilitate NNA convolution calculation: Each input channel retains a valid data. The convolution logic is to multiply and accumulate all channels. Matmul only multiplies the current weight by the feature, and the influence of other channels needs to be removed. Keep one weight channel valid and set the rest of the weight channels to 0, so that the accumulation in the channel direction after multiplication and summation will not cause abnormal results; S2: Data rearrangement. The first input source feature1 is understood as the input data in the convolution process. The number of rows can be any value. It is assumed to be 9 rows and 32 channels. The 9 rows correspond to the 9 weights of the channels in step S1. The channels are multiples of 32 for one cycle. If they are not multiples of 32, they can be calculated separately: Load 32 channels chn in sequence and store them into NNA storage input data space FRAM, treating each 32 chn as a pixel. Then rearrange the 9 pixels in the form of a convolution kernel: Assuming 8-bit data: LA(o,VR0,0,in_ptr,0); row 0 data; LA(o,VR0,1,in_ptr+ichn,0); data of the first row; LA(o,VR0,0,in_ptr+2*ichn,0); data of the second row; Read F00 F10 F20 data from the input address; NNDWF0(VR0) NNDWF0(VR1) .......Loop to read the following channels and calculate 8 times in parallel; Store the data of VR0 and VR1 in FRAM, configure the NNA register in advance, and jump to the next row after reading a fixed pixel. Assuming it is set to 8, NNDWF0 (VR) will automatically jump to the next row FRAM address 8 times: LA(o,VR0,0,in_ptr+3*ichn,0); data in the third row; LA(o,VR0,1,in_ptr+4*ichn,0); data in the 4th row; LA(o,VR0,0,in_ptr+5*ichn,0); data of the 5th row; NNDWF0(VR0) NNDWF0(VR1) Read F30 F40 F50 data from the input address; LA(o,VR0,0,in_ptr+6*ichn,0); data of the 6th row; LA(o,VR0,1,in_ptr+7*ichn,0); data in row 7; LA(o,VR0,0,in_ptr+8*ichn,0); data in row 8; NNDWF0(VR0) NNDWF0(VR1) Read F60 F70 F80 data from the input address; In this way, the corresponding weights are multiplied by the corresponding data to obtain the calculation logic of matmul: ； S3: Use NNA to perform convolution operations and sum: Due to the hardware settings, the stride of the convolution kernel is set to 2: F00 to F80 in the upper left box of the above table represent channels 0 to 31; F00 to F80 in the middle upper frame of the above table represent channels 32 to 63; When NNA is calculating, the data of the upper left corner box and the upper middle box area will be multiplied by the weight at the same time, realizing the idea of ​​weight reuse and reducing the operation of repeated weight reading: Use NNDWF0 to store F00~F80, 0~63 channel data into FRAM; Use NNDWW to store weight data in FRAM; Use the NNMACG instruction to multiply the weight data and the input data, and use the same WRAM weight data for channels 0 to 31 and 32 to 63. S4: Parallel calculation: Write the first convolution calculation, i.e., the data of channels 0 to 63, into FRAM0 in advance, and then perform NNA convolution operation MAC00. Use NNMACG (MAC00, OPT_NNMAC_WRAING, 1, OPT_NNMAC_FP_NUM0) to start calculating MAC00. During the MAC process, write the data of channels 64 to 127 into FRAM1, and store the data of FRAM0 into the temporarily allocated space tmp, and then MAC01. Alternately, MAC00 takes time and can be performed in parallel with simd instructions. The operation is as follows: The calculation time of MAC00 is used for reading data of MAC01, and the calculation time of MAC01 is used for reading data of MAC00. The two spaces are used for ping-pong calculation. F00 to F80 in the lower left box of the above table represent channels 64 to 95; F00 to F80 in the lower middle box in the above table represent channels 96 to 127; The 64 channels that need to be calculated next time are stored in the NNA FRAM space in advance, so that the time for NNA to perform convolution operation is used to read the data for the next convolution operation, thus improving the calculation efficiency of NNA; After all channels are calculated, the data is stored in a tmp, the weight is updated, and the above steps are repeated, which is expressed as: LA(o,VR0,0,in_ptr+ichn,0); LA(o,VR0,1,in_ptr+1*ichn,0); LA(o,VR0,0,in_ptr+2*ichn,0); NNDWF0(VR0) NNDWF0(VR1) Store the remaining row data in FRAM, update the FRAM data, then perform MAC calculation, and then add the result to the data in tmp until all weight calculations are completed, expressed as: NNDRDOCB(VR0,OPT_NNDRDOCB_PIXEL0,NNDRDOCB_LO); Store the NNA calculation result in the VR0 register LA(o,VR1,0,tmp_ptr,0); read tmp space data ADD(w,VR2,VR0,VR1); add the calculation result SA(o,VR2,0,tmp_ptr,0) to the tmp data; update the data in the tmp space. So far, the result of multiplying and accumulating all weights and channels is obtained.

2. The matmul layer data processing method based on NNA2.0 according to claim 1, characterized in that: If the number in step S2 is not a multiple of 32, it can be calculated separately using C++ code.

3. The matmul layer data processing method based on NNA2.0 according to claim 1, characterized in that: The step size stride=2 is based on the NNA hardware function. The output shape is 1x4, generating 4 points at a time, 2 of which are valid pxiel data generation; if it is NNA2.0, the output shape is 1X8, generating 8 output points at a time, 4 of which are valid data.

4. The matmul layer data processing method based on NNA2.0 according to claim 1, characterized in that: The matmul implementation logic is the lowest two dimensions, namely w: width and c: channel, multiplying and accumulating to get the final result.

Images