Method for implementing vectorization of multi-sample multi-channel convolution back propagation

Abstract

The application discloses a kind of multi-sample multi-channel convolution back propagation vectorization implementation methods, steps include: S01.input feature data and input error gradient are stored as two-dimensional matrix;S02.extract submatrix from input error gradient matrix and transmission, and transmit convolution kernel matrix;S03.to submatrix and convolution kernel matrix are executed vectorized matrix multiplication, obtain output error gradient matrix;S04.extract submatrix from input error gradient matrix, execute vectorized matrix summation calculation, obtain the gradient of convolution layer bias parameter;S05.input error gradient matrix is transmitted to scalar memory, and submatrix is extracted from input feature data matrix transmission;S06.to input error gradient matrix and the submatrix of input feature data are executed vectorized matrix multiplication calculation, obtain the gradient of convolution layer weight parameter.The application has the advantages such as simple realization, high computing efficiency, small overhead, low storage bandwidth demand.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vector processors, in particular to a multi-sample multi-channel convolution back propagation vectorization implementation method. BACKGROUND

[0002] Deep learning technology represented by deep convolutional neural networks has been widely applied in various fields such as image recognition and classification, natural language processing, finance, autonomous driving and recommendation algorithms, etc. When designing a convolutional neural network structure, a three-dimensional input data is usually set, i.e. the number of channels, height and width of the picture.

[0003] In order to obtain a convolutional neural network model with excellent performance, the convolutional neural network needs to be trained, i.e. the parameters are initialized, then the gradient of the loss function with respect to the parameters is calculated, and then the value of the parameters is updated according to the gradient. Through continuous iteration, the corresponding parameters that minimize the loss function are calculated, thereby completing the training of the network. The most critical step in the training process is to calculate the gradient of the loss function with respect to the parameters, which can be achieved by using the back propagation algorithm.

[0004] In the prior art, when calculating the gradient of the input feature map by convolution back propagation, 0 elements are usually inserted between the rows and columns of the error gradient data map, and 0 elements are filled around it. Then, the transformation of 180 degrees clockwise rotation is performed on each channel of the convolution kernel, and the new convolution kernel is used to perform convolution operation on the loss data. However, this kind of method has the following problems:

[0005] (1) The storage position of 0 elements is discontinuous, resulting in large operation overhead of 0 element filling;

[0006] (2) The time overhead of matrix rotation and copying error gradient data is large.

[0007] A vector processor is a novel architecture that can have strong computing power while maintaining low power consumption, and is particularly suitable for accelerating large convolutional neural network calculations. Figure 1As shown, the vector processor generally includes a scalar processing unit (SPU) and a vector processing unit (VPU), the SPU is responsible for scalar task calculation and flow control, the VPU is responsible for vector calculation, provides main computing power, includes several vector processing elements (VPE), each VPE includes MAC, ALU and other operation function components. A data transmission and exchange mechanism is provided between the SPU and the VPU, to realize sharing and communication of scalar and vector data. A vector data access unit supports Load / Store of vector data, provides a large-capacity dedicated vector array memory (AM) and off-chip memory shared by scalars and vectors.

[0008] The prior art cannot realize convolution back propagation vectorization based on the vector processor, if the convolution back propagation vectorization is directly performed on the basis of the vector processor in the traditional way, the calculation efficiency is low, the weight data cannot be effectively shared, the storage bandwidth is wasted, the computing performance of the vector processor cannot be fully utilized, and the 0 operation is needed, so that there are too much data migration and a large amount of additional overhead. SUMMARY

[0009] The technical problem to be solved by the present application is that, in view of the technical problems existing in the prior art, the present application provides a multi-sample multi-channel convolution back propagation vectorization implementation method which is simple to implement, has high calculation efficiency, has small overhead, has low storage bandwidth requirement, can fully utilize the computing performance of the vector processor, reduces the storage bandwidth requirement and hardware overhead, improves the calculation efficiency, and efficiently realizes vectorization calculation of multi-sample multi-channel convolution back propagation.

[0010] To solve the above technical problems, the technical solution provided by the present application is as follows:

[0011] A multi-sample multi-channel convolution back propagation vectorization implementation method, comprising the following steps:

[0012] S01. Storing input feature data of forward calculation of a convolutional neural network and input error gradient of back propagation in a two-dimensional matrix in a sample dimension priority manner, to obtain input feature data and input error gradient matrix;

[0013] S02. The vector processor extracts V rows of data from the input error gradient matrix obtained in step S01 to form a first sub-matrix and transmits the first sub-matrix to the vector array memory AM of each core, and transmits the convolution kernel matrix to the scalar memory SM of each core, wherein 0V<=K, K=O*kH*kW, kH is the height of the convolution kernel, kW is the width of the convolution kernel, and O is the number of convolution kernels.

[0014] S03. The vector processor performs vectorized matrix multiplication on the first sub-matrix of the input error gradient matrix and the convolution kernel matrix to calculate an output error gradient matrix and transmits the output error gradient matrix to the off-chip memory of the vector processor; steps S02 and S03 are repeated until the calculation of all output error gradient matrices is completed;

[0015] S04. The vector processor extracts dS groups of data in rows from the input error gradient matrix obtained in step S01 to form a second sub-matrix and transmits the second sub-matrix to the vector array memory AM of each core, where dS = nH*nW, nH is the height of the output feature map, and nW is the width of the output feature map; the vector processor performs vectorized matrix summation calculation on the second sub-matrix to obtain the gradient of the convolution layer bias parameter and transmits the gradient to the off-chip memory of the vector processor; after repeating step S04 until the calculation of all convolution layer bias parameter gradients is completed, the vector processor updates the convolution layer bias parameter according to the learning rate;

[0016] S05. The vector processor transmits the input error gradient matrix obtained in step S01 to the scalar memory SM of each core, and extracts an (N*nnH*nnW)*C order sub-matrix from the input feature data matrix obtained in step S01 and transmits the sub-matrix to the vector array memory AM of each core, where N is the number of training samples, C is the number of channels of the input feature map, nnH and nnW represent the logical height and width of the output feature map after (S-1) 0s are padded between the rows and columns, respectively, and S is the step size of the convolution kernel.

[0017] S06. The vector processor performs vectorized matrix multiplication calculation on the input error gradient matrix and the sub-matrix of the input feature data in step S05 to calculate the gradient of the convolution layer weight parameter and transmits the gradient to the off-chip memory of the vector processor; steps S05 and S06 are repeated until the calculation of all convolution layer weight parameter gradients is completed, and then the vector processor updates the convolution layer weight parameter according to the learning rate.

[0018] Further, when storing the input feature data of the forward calculation of the convolutional neural network and the input error gradient of the back propagation in step S01, one sample data is stored in each column, and the storage order of each column is: first, the image width W direction is given priority, then the image height H direction is given priority, and finally the image channel C direction is given priority; the convolution kernel data is stored as a two-dimensional convolution kernel matrix, where each row of the matrix represents a convolution kernel, and the storage order of each row is first the convolution kernel width kW direction is given priority, then the image convolution kernel kH direction is given priority, and finally the convolution kernel channel O direction is given priority; the convolution kernel data is specifically stored in the off-chip memory of the vector processor as an O*KC order matrix, where KC = kH*kW*C is the number of pixel data of a single convolution kernel.

[0019] Further, in the step S02, the total number of times of extracting V rows of data to form the first sub-matrix is H*W, and the calculation of the entire output error gradient matrix is completed, wherein:

[0020] nH = (H + 2pH - kH) / S + 1, nW = (W + 2pW - kW) / S + 1, nnH = nH + (nH - 1) * (S - 1),

[0021] nnW = nW + (nW - 1) * (S - 1), H' = H + 2pH, W' = W + 2pW

[0022] M = nH * nW * O

[0023] wherein H and W are the image height and image width of the input feature map of the current convolutional layer, pH and pW are the number of 0 elements filled in the high and wide directions, H' and W' are the logical height and logical width of the input feature map after padding with 0, and M represents the number of pixels of a single output feature map.

[0024] Further, in the step S02, the specific steps of extracting V rows of data to form the first sub-matrix are as follows:

[0025] Step S201: let valid = 0, h = 0, w = 0, the value range of h is {0,..., H'-1}, the value range of w is {0,..., W'-1}, and it is determined whether h < pH or h >= (pH + H) or w < pW or w >= (pW + W) is satisfied, if yes, go to step S202, otherwise go to S204;

[0026] Step S202: increase w by 1, and determine whether w > pW - 1, if no, go to step S201, if yes, go to step S203;

[0027] Step S203: increase h by 1, w = 0, and determine whether h > pH - 1, if yes, end, if no, go to step S201;

[0028] Step S204: construct a vector flag with a length of K, let o = 0, m = 0, n = 0, the value range of o is {0,..., O - 1}, the value range of m is {0,..., kH - 1}, and the value range of n is {0,..., kW - 1};

[0029] Step S205: judge whether (h+m>=(kH-1) && (w+n)>=(kW-1) && (h+m)<=(kH+nnH-2) && (w+n)<=(kW+nnW-2)) is true, if not, go to step S206; if yes, judge whether (h+m-kH+1)%S==0 && (w+n-kW+1)%S==0 is true, wherein % is a remainder operation, if not, record the o*kH*kW+m*kW+n element of the vector flag as 0, if yes, define t as follows:

[0030] t=(h+m-kH+1) / s*nW+(w+n-kW+1) / S+o*nH*nW,

[0031] record the o*kH*kW+m*kW+n element of the vector flag as 1, and extract the t row of the input error gradient matrix, valid+1;

[0032] Step S206: n+1, and judge whether n>=kW is satisfied, if not, go to step S205, if yes, go to step S207;

[0033] Step S207: m+1, n=0, and judge whether m>=kH is satisfied, if not, go to step S205, if yes, go to step S208;

[0034] Step S208: o+1, m=0, n=0, judge whether o>=O is satisfied, if not, go to step S205, if yes, go to step S03.

[0035] Further, the specific steps of performing vectorized matrix multiplication on the first sub-matrix of the input error gradient matrix and the convolution kernel matrix by each core of the vector processor in the step S03 include:

[0036] Step S301: the vector processor transmits the extracted first sub-matrix of the input error gradient matrix to the preset input buffer in the vector array memory AM of each core of the vector processor respectively, and the size of the first sub-matrix of the input error gradient matrix transmitted by each core is (valid*kH*kW)*p order, wherein p is the number of vector processing units VPE of each core;

[0037] Step S302: the vector processor transmits the convolution kernel data matrix to the preset convolution kernel data buffer in the scalar memory SM of each core of the vector processor respectively, and the size of the convolution kernel data matrix transmitted by each core is O*KC order, let i=0, j=0, invalid=0;

[0038] Step S303: Let indx = i*O*kH*kW+j, o = indx / (O*kH*kW), u = indx%(kH*kW), c = indx / (kH*kW)-o*O, the scalar processing unit SPU of each core of the vector processor reads the element of the c-th row and the kH*kW-u-1+o*kH*kW-th column of the convolution kernel data buffer into a scalar register in turn, judges whether the input error gradient matrix row corresponding to the read convolution kernel data is 0, if yes, invalid is increased by 1, and the next element is read; if not, the scalar broadcast instruction is broadcast to a vector register;

[0039] Step S304: the vector processing unit VPU of each core of the vector processor reads the input error gradient data of the (j-invalid) -th row from the input error gradient buffer into a vector register in turn, and performs multiply-accumulate calculation on the vector register and the vector register obtained in the step S303, and j is increased by 1;

[0040] Step S305: judges whether j is greater than or equal to O*kH*kW, if not, jumps to step S303, and the reading position in the step S304 moves to the next row, if yes, jumps to step S306;

[0041] Step S306: judges whether i is greater than or equal to C, if not, j = 0, jumps to step S303, and the reading position in the step S304 returns to the start address of the input error gradient data buffer, if yes, the vector processor completes the calculation of the C*N order output error gradient data.

[0042] Further, in step S03, the specific steps of storing the calculation result of the output error gradient matrix in the off-chip memory of the vector processor are as follows:

[0043] The vector processor transmits the obtained output error gradient matrix to the output buffer preset in each core of the vector processor respectively, and each core is responsible for the matrix scale of C*p order, p is the number of vector processing units VPE of each core, and the i-th row in the matrix is transmitted to the (h-pH)*W+(w-pW)+i*H*W row of the output buffer.

[0044] Further, the specific steps of performing vectorized matrix summation calculation on the second sub-matrix in the step S04 include:

[0045] Step S401: the vector processor transmits the second sub-matrix of the input error gradient matrix to the input buffer preset in the vector array memory AM of each core of the vector processor respectively, and the input error gradient matrix scale of each core is (nH*nW)*p order, p is the number of vector processing units VPE of each core;

[0046] Step S402: the vector processing component VPU of each core of the vector processor traverses the second sub-matrix of the input error gradient matrix, and sequentially reads a row from the second sub-matrix of the input error gradient matrix into a vector register, performs vector summation calculation, and writes the result into a scalar register; the scalar processing component SPU of each core uniformly stores the value of the scalar register into a scalar register of one of the cores and performs accumulation calculation; after traversing (nH*nW) rows, the vector processor completes the calculation of the corresponding component in the gradient of the convolution layer bias parameter;

[0047] In the step S04, updating the convolution layer bias parameter according to the learning rate comprises: the vector processor traverses the convolution layer bias parameter and updates it according to the following formula: convolution layer bias parameter-(convolution layer bias parameter gradient*learning rate).

[0048] Further, in the step S05, extracting the (N*nnH*nnW)*C order sub-matrix from the input feature data matrix obtained in the step S01 and transmitting it to the vector array memory AM of each core comprises:

[0049] Step S501: let h=0, w=0, where the value range of h is {0,...,kH-1} and the value range of w is {0,...,kW-1}, and let c=0, m=0, n=0, where the value range of c is {0,...,N-1}, the value range of m is {0,...,nnH-1}, and the value range of n is {0,...,nnW-1}, and create a vector flag with a size of (N*nnH*nnW);

[0050] Step S502: determine whether h+m>pH-1&&h+m<H+pH&&w+n>pW-1 and w+n<(pW+W) are satisfied, if yes, go to S503, otherwise go to S507;

[0051] Step S503: construct a vector temp with a length of C, let i=0, where the value range of i is {0,...,C-1}, and t=(h+m-pH)*W+(w+n-pW);

[0052] Step S504: determine whether i is greater than C-1, if yes, go to 10.6, if not, extract the pixel point in the i*H*W+t row of the c column of the input data matrix to the i position in temp, and go to step 10.5.

[0053] Step S505: i is incremented by 1, and go to step S504;

[0054] Step S506: transmit the temp vector to the sub-matrix;

[0055] Step S507: n is increased by 1, and it is judged whether n is greater than nnW-1, if yes, go to S508, if not, go to step S502;

[0056] Step S508: m is increased by 1, and it is judged whether m is greater than nnH-1, if yes, go to S509, if not, go to step S502.

[0057] Step S509: c is increased by 1, m=0, n=0, and it is judged whether c is greater than N-1, if yes, go to step S511, if not, go to step S502;

[0058] Step S510: w is increased by 1, c=0, m=0, n=0, and it is judged whether w is greater than kW-1, if yes, go to S511, if not, go to step S502;

[0059] Step S511: h is increased by 1, w=0, c=0, m=0, n=0, and it is judged whether h is greater than kH-1, if yes, the current flow is terminated, if not, go to step S502.

[0060] Further, the specific steps of performing vectorized matrix multiplication calculation on the input error gradient matrix and the sub-matrix of the input feature data in step S05 by the vector processor in step S06 include:

[0061] Step S601: the vector processor transmits the input data into the preset input data buffer of the vector array memory AM of the vector processor respectively, and the input data matrix has a scale of (N*nnH*nnW)*C;

[0062] Step S602: the vector processor transmits the input error gradient matrix into the preset data buffer of the scalar memory SM of the vector processor respectively, and the input error gradient matrix has a scale of (O*nnH*nnW)*N, let i=0, j=0; step S603: let x=i*N*nnH*nnW+j, o=x / (nnH*nnW*N), c=x / (nnH*nnW)-o*N, s=x / nnW-o*N*nnH-c*nnH, r=x%nnW, st=o*nnH*nnW+(s / S)*nnW+(r / S), if s%S is not equal to 0 or r%S is not equal to 0, jump to step S605, otherwise let: yc=j / (nnH*nnW), ym=j / nnW-yc

[0063] nnH, yn=j%nnW, if h+ym is greater than pH-1 and h+ym is less than pH+H and w+yn is greater than pW-1 and w+yn is less than pW+W, the vector processing unit VPU of the vector processor reads the jth row data of the input feature data matrix to a vector register;

[0064] Step S604: the scalar processing unit SPU of the vector processor reads the element in the st row and the c column of the input error gradient matrix to a scalar register, and performs multiply-accumulate calculation on the scalar register and the scalar register obtained in step S603;

[0065] Step S605: j is incremented by 1, and it is judged whether j is greater than or equal to N*nnH*nnW, if yes, jump to step S606, if no, jump to step S603.

[0066] Step S606: i is incremented by 1, and it is judged whether i is greater than or equal to O, if yes, end the current process; if no, j=0 jump to step S603.

[0067] Further, the specific steps of transmitting the gradient of the convolution layer weight parameter in step S06 include: the vector processor extracts the element in the m row and the n column of the two-dimensional matrix obtained by the current execution of the vectorized matrix multiplication calculation to the m row and the n*kH*kW+h*kW+w column of the convolution layer weight gradient matrix, wherein the value range of m is {0,…O-1}, and the value range of n is {0,…,C-1}; and the specific steps of updating the convolution layer weight parameter according to the learning rate include: the vector processor transmits the convolution kernel matrix and the convolution layer weight parameter gradient to the preset input data buffer area in the scalar array memory SM of each core of the vector processor respectively, the vector accelerator reads the convolution kernel matrix and the convolution layer weight parameter gradient matrix row by row, and updates according to the following formula: current convolution kernel matrix row-(current convolution layer bias parameter gradient matrix row*learning rate), and the updating of the convolution layer weight parameter is completed after repeating O times.

[0068] Compared with the prior art, the present application has the following advantages:

[0069] 1、The present application makes full use of the characteristics of vectorization and parallelization calculation of the vector processor, and sequentially performs the calculation of the output error gradient matrix and the convolution layer bias parameter gradient and the convolution layer weight parameter gradient based on the vector processor, so as to convert the complex multiple loop convolution back propagation calculation into efficient vectorization and parallelization matrix multiplication calculation, which can fully exert the calculation performance of the vector processor to efficiently realize convolution back propagation vectorization, not only can effectively reduce the storage bandwidth demand and hardware overhead, but also can greatly improve the calculation efficiency.

[0070] 2, The application is based on the process of realizing convolution back propagation calculation of vector processor, through expanding each convolution kernel data into vector data, and performing vector multiplication and accumulation calculation with all input data, the SIMD and inter-core parallelism of vector processor can be fully utilized, the calculation efficiency of convolution neural network is greatly improved, and through storing all input error gradient data of the same sample on a column, the multiplication and accumulation calculation of convolution kernel data and input error gradient data is operated on the same VPE processing unit, thereby avoiding reduction summation between multiple processing units, and improving the overall calculation efficiency of vector processor.

[0071] 3, The embodiment stores input feature data and input error gradient according to sample dimension priority, and transmits the convolution kernel data of each core through scalar memory and broadcasts to vector processing unit for calculation, so that the convolution kernel data can be shared, the transmission amount of calculation data is greatly reduced, the bandwidth requirement of convolution kernel data is significantly reduced, the transmission time of convolution kernel data is reduced, the original image data operation does not need to be copied, the complex multiple loop convolution back propagation calculation operation is converted into H*W times of efficient vectorization and parallelization matrix multiplication calculation, the high efficiency of vector processor for matrix multiplication calculation and the characteristics that vector processor is very suitable for vectorization and parallelization calculation can be fully utilized, and the actual storage of 0 elements is not needed, so that additional memory overhead is not needed, the data migration times and calculation times are reduced

[0072] 4, V=valid row data can be extracted from the M*N order input error gradient matrix by row, to form a V*N order input error gradient matrix submatrix, wherein the 0 elements are realized in the calculation process, the actual storage of 0 elements is not needed, so that additional memory overhead is not needed, and the calculation corresponding to 0 elements is reduced BRIEF DESCRIPTION OF DRAWINGS

[0073] Figure 1 It is a general structure schematic diagram of vector processor.

[0074] Figure 2 It is a detailed flow schematic diagram of the vectorization implementation method of multi-sample multi-channel convolution back propagation of the embodiment.

[0075] Figure 3 It is a principle schematic diagram of reordering input error gradient data in the specific application embodiment of the application.

[0076] Figure 4 It is a detailed flow schematic diagram of the submatrix extraction operation in step S02 in the specific application embodiment of the application.

[0077] Figure 5 It is a detailed flow schematic diagram of performing vectorization calculation in step S03 in the embodiment.

[0078] Figure 6 is the principle diagram of performing vectorized matrix summation calculation on the sub-matrix in the specific application embodiment of the present application.

[0079] Figure 7 is the principle diagram of updating the convolution layer bias parameter in the specific application embodiment of the present application. DETAILED DESCRIPTION

[0080] The present application is further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the present application is not limited thereby.

[0081] As shown in Figure 2 , the steps of the multi-sample multi-channel convolution back propagation vectorization implementation method of the present embodiment include:

[0082] S01. Storing the input feature data of the forward calculation of the convolutional neural network and the input error gradient of the back propagation in a two-dimensional matrix according to the sample dimension priority, obtaining the input feature data and the input error gradient matrix;

[0083] S02. The vector processor extracts V rows of data from the input error gradient matrix obtained in step S01 to form a first sub-matrix and transmits it to the vector array memory AM of each core, and transmits the convolution kernel matrix to the scalar memory SM of each core, wherein 0

[0084] S03. The vector processor core performs vectorized matrix multiplication on the first sub-matrix of the input error gradient matrix and the convolution kernel matrix, calculates the output error gradient matrix and transmits it to the off-chip memory of the vector processor; repeat step S02, step S03 until the calculation of all output error gradient matrices is completed;

[0085] S04. The vector processor extracts dS rows of data from the input error gradient matrix obtained in step S01 to form a second sub-matrix and transmits it to the vector array memory AM of each core, wherein dS = nH*nW, nH is the height of the output feature map, and nW is the width of the output feature map; perform vectorized matrix summation calculation on the second sub-matrix to obtain the gradient of the convolution layer bias parameter and transmit it to the off-chip memory of the vector processor; repeat step S04 until the calculation of all convolution layer bias parameter gradients is completed, and then update the convolution layer bias parameter according to the learning rate;

[0086] S05. The vector processor transmits the input error gradient matrix obtained in step S01 into the scalar memory SM of each core, and extracts the (N*nnH*nnW)*C order sub-matrix from the input feature data matrix obtained in step S01 and transmits it into the vector array memory AM of each core, wherein N is the number of training samples, C is the number of channels of the input feature map, nnH and nnW represent the logical height and width of the output feature map after (S-1) 0s are padded between the rows and columns, respectively, and S is the step size of the convolution kernel;

[0087] S06. The vector processor performs vectorized matrix multiplication calculation on the input error gradient matrix and the sub-matrix of the input feature data in step S05, calculates the gradient of the convolution layer weight parameter and transmits it into the off-chip memory of the vector processor, and repeats step S05 and step S06 until the calculation of all the convolution layer weight parameter gradients is completed, and then updates the convolution layer weight parameter according to the learning rate.

[0088] The convolution back propagation calculation is to update the weight parameter reversely according to the error between the calculation result and the correct result, until the error meets the requirement, and the required weight parameter is obtained. Through the above steps, the complex convolution back propagation calculation with multiple loops is converted into efficient vectorized and parallelized matrix multiplication calculation, the calculation of the output error gradient matrix, the convolution layer bias parameter gradient and the convolution layer weight parameter gradient is completed in turn, the high efficiency of the vector processor for matrix multiplication calculation and the characteristics of the vector processor being very suitable for vectorized and parallelized calculation are fully utilized, the calculation performance of the vector processor is fully utilized, the convolution back propagation vectorization is efficiently implemented, the storage bandwidth requirement and hardware overhead are effectively reduced, and the calculation efficiency is greatly improved.

[0089] In step S01 of the embodiment, when the input feature data of the forward calculation of the convolutional neural network and the input error gradient of the back propagation are stored, one sample data is stored in each column, and the storage order of each column is: first, the image width W direction is given priority, then the image height H direction is given priority, and finally the image channel C direction is given priority; the convolution kernel data is stored as a two-dimensional convolution kernel matrix, wherein each row of the matrix represents a convolution kernel, and the storage order of each row is first the convolution kernel width kW direction is given priority, then the image convolution kernel kH direction is given priority, and finally the convolution kernel channel O direction is given priority; the convolution kernel data is specifically stored in the off-chip memory of the vector processor in the form of an O*KC order matrix, wherein KC=kH*kW*C is the number of pixel data of a single convolution kernel. It is assumed that the input error gradient matrix for the back propagation calculation of the current convolution layer is a two-dimensional matrix of M*N order, wherein N is the number of training samples, nH is the height of the output feature map calculated in the forward direction, nW is the width of the output feature map calculated in the forward direction, and M=O*nH*nW.

[0090] The input error gradient data is reordered in a specific application embodiment as shown in Figure 3 where the element index starts from 0, Figure 3 R, G represent two channels of the input error gradient data respectively. R(x, y) represents the value of the x-th input image at the 0-th channel at the y-th position. G(x, y) represents the value of the x-th input image at the 1-st channel at the y-th position. The original input data is arranged in sequence as follows: all gradient values of the 0-th image at the 0-th channel, all gradient values of the 0-th image at the 1-st channel, all gradient values of the 0-th image at the 2-nd channel, all gradient values of the 1-st image at the 0-th channel, and so on. After the rearrangement, the input error gradient data is converted into a two-dimensional matrix with (O*nH*nW) rows and N columns. Each column of the two-dimensional matrix represents the input gradient values of an image, in sequence, the 0-th image, the 1-st image, and so on, until the N-1-st image. The matrix has a total of (O*nH*nW) rows. Taking the 0-th column as an example, all gradient values at the R channel (0-th channel) are stored in sequence, followed by all gradient values at the G channel (1-st channel).

[0091] The embodiment can realize the sharing of the convolution kernel data, significantly reduce the transmission amount of the calculation data, significantly reduce the bandwidth requirement of the convolution kernel data, reduce the transmission time of the convolution kernel data, and does not need to copy the original image data operation. The complex multi-loop convolution back propagation calculation operation is converted into H*W efficient vectorization and parallelization matrix multiplication calculation, which can fully utilize the high efficiency of the vector processor for matrix multiplication calculation and the characteristics of the vector processor being very suitable for vectorization and parallelization calculation, and does not need the actual storage of 0 elements, thus not needing additional memory overhead, reducing the data migration times and the calculation times.

[0092] In step S02, the total number of times of extracting V rows of data to form the first sub-matrix is H*W, and the complete output error gradient matrix calculation is completed, wherein:

[0093] nH = (H+2pH-kH) / S+1, nW = (W+2pW-kW) / S+1, nnH = nH + (nH-1)*(S-1),

[0094] nnW = nW + (nW-1)*(S-1), H' = H+2pH, W' = W+2pW

[0095] M = nH*nW*O

[0096] wherein H, W are the image height and image width of the input feature map of the current convolution layer, pH, pW are the number of 0 elements filled in the high and wide directions, H', W' are the logical height and logical width of the input feature map after padding with 0, M is the number of pixels of a single output feature map.

[0097] As shown in FIG. 2, the specific steps of extracting V row data to form the first sub-matrix in step S02 of the embodiment are as follows: Figure 4

[0098] Step S201: let valid = 0, h = 0, w = 0, the value range of h is {0,..., H'-1}, the value range of w is {0,..., W'-1}, determine whether h < pH or h >= (pH + H) or w < pW or w >= (pW + W) is satisfied, if yes, go to step S202, otherwise go to step S204;

[0099] Step S202: increase w by 1, and determine whether w > pW-1 is satisfied, if no, go to step S201, if yes, go to step S203;

[0100] Step S203: increase h by 1, w = 0, and determine whether h > pH-1 is satisfied, if yes, end, if no, go to step S201;

[0101] Step S204: construct a vector flag with a length of K, let o = 0, m = 0, n = 0, the value range of o is {0,..., O-1}, the value range of m is {0,..., kH-1}, the value range of n is {0,..., kW-1};

[0102] Step S205: determine whether (h+m >= (kH-1) && (w+n) >= (kW-1) && (h+m) <= (kH+nnH-2) && (w+n) <= (kW+nnW-2)) is true, if no, go to step S206; if yes, continue to determine whether (h+m-kH+1) % S == 0 && (w+n-kW+1) % S == 0 is true, wherein % is the remainder operation, if no, record the (o*kH*kW+m*kW+n) th element of the vector flag as 0, if true, define t as follows:

[0103] t = (h+m-kH+1) / s*nW+(w+n-kW+1) / S+o*nH*nW,

[0104] and record the (o*kH*kW+m*kW+n) th element of the vector flag as 1, and extract the t th row of the input error gradient matrix, valid is increased by 1;

[0105] ​Step S206: n is increased by 1, and it is determined whether n >= k W is satisfied, if not, go to step S205, if yes, go to step S207;

[0106] Step S207: m is increased by 1, n = 0, and it is determined whether m >= k H is satisfied, if not, go to step S205, if yes, go to step S208;

[0107] Step S208: o is increased by 1, m = 0, n = 0, and it is determined whether o >= O is satisfied, if not, go to step S205, if yes, go to step S03.

[0108] Through the above steps, V = valid row data can be extracted from the input error gradient matrix of M*N order by row, to form a sub-matrix of the input error gradient matrix of V*N order, wherein the 0 elements are realized in the calculation process, and actual storage of 0 elements is not required, so that no additional memory overhead is required, and the calculation corresponding to the 0 elements is reduced.

[0109] In step S03 of the embodiment, the convolution kernel data matrix of O*KC = 2*12 order and the input error gradient sub-matrix of V*N = valid*64 order are subjected to vectorized matrix multiplication calculation and parallelized matrix multiplication calculation of each core, as shown in Figure 5 The specific steps of performing vectorized matrix multiplication include:

[0110] Step S301: the vector processor transmits the extracted first sub-matrix of the input error gradient matrix to the preset input buffer in the vector array memory AM of each core of the vector processor, respectively, the size of the first sub-matrix of the input error gradient matrix transmitted into each core is (valid*kH*kW)*p order, and p is the number of vector processing units VPE of each core;

[0111] Step S302: the vector processor transmits the convolution kernel data matrix to the preset convolution kernel data buffer in the scalar memory SM of each core of the vector processor, respectively, the size of the convolution kernel data matrix transmitted into each core is O*KC order, i = 0, j = 0, and invalid = 0;

[0112] Step S303: Let indx = i*O*kH*kW+j, o = indx / (O*kH*kW), u = indx%(kH*kW), c = indx / (kH*kW)-o*O, the scalar processing unit SPU of each core of the vector processor reads the element in the c-th row and the kH*kW-u-1+o*kH*kW-th column of the convolution kernel data buffer into a scalar register in turn, judges whether the input error gradient matrix row corresponding to the read convolution kernel data is 0, if yes, invalid is increased by 1, and the next element is read; if not, the scalar broadcast instruction is broadcast to a vector register;

[0113] Step S304: the vector processing unit VPU of each core of the vector processor reads the input error gradient data in the (j-invalid) -th row of the input error gradient buffer into a vector register in turn, and performs multiplication and accumulation calculation on the vector register and the vector register obtained in the step S303, and j is increased by 1.

[0114] Step S305: judges whether j is greater than or equal to O*kH*kW, if not, jumps to step S303, and the reading position in the step S304 moves to the next row, if yes, jumps to step S306.

[0115] Step S306: judges whether i is greater than or equal to C, if not, j = 0, jumps to step S303, and the reading position in the step S304 returns to the start address of the input error gradient data buffer, if yes, the vector processor completes the calculation of the C*N order output error gradient data.

[0116] The data buffers in the steps S301 and S302 described above in the embodiment can be two, when the calculation is performed on one of the data buffers, data transmission is performed on the other data buffer, so that the data transmission and the calculation time overlap, and the calculation efficiency can be further improved.

[0117] The judgment in the step S303 described above in the embodiment whether the input error gradient matrix row corresponding to the convolution kernel data is a 0 element row is implemented according to the vector flag, and the specific steps are as follows: according to the element value of the vector flag, it is judged that if flag[w] = 0, the input error gradient matrix row corresponding to the convolution kernel data is a 0 element row, if flag[w] = 1, the input error gradient matrix row corresponding to the convolution kernel data is not a 0 element row.

[0118] In step S03 of the embodiment, the specific steps of storing the calculation result of the output error gradient matrix in the off-chip memory of the vector processor are as follows: the vector processor transmits the obtained output error gradient matrix to the output buffer area preset in each core of the vector processor respectively, and each core is responsible for the matrix of C*p order, p is the number of VPEs of each core, and the i-th row in the matrix is transmitted to the (h-pH)*W+(w-pW)+i*H*W row of the output buffer area.

[0119] In step S04 of the embodiment, the vector processor extracts dS=nH*nW rows of data from the input error gradient matrix in step S01 by rows to form a second sub-matrix and transmits the second sub-matrix to the vector array memory AM of each core, and performs vectorized matrix summation calculation on the second sub-matrix to obtain the gradient of the convolution layer bias parameter. The specific steps of performing vectorized matrix summation calculation on the second sub-matrix include:

[0120] Step S401: The vector processor transmits the second sub-matrix of the input error gradient matrix to the input buffer area preset in the vector array memory AM of each core of the vector processor respectively, and each core transmits the input error gradient matrix of (nH*nW)*p order, p is the number of VPEs of each core;

[0121] Step S402: The vector processing unit VPU of each core of the vector processor traverses the second sub-matrix of the input error gradient matrix, reads a row from the second sub-matrix of the input error gradient matrix into a vector register in turn, performs vector summation calculation, and writes the result to a scalar register; the scalar processing unit SPU of each core uniformly stores the value of the scalar register in the scalar register of one of the cores and performs accumulation calculation; after traversing (nH*nW) rows, the vector processor completes the calculation of the corresponding component in the gradient of the convolution layer bias parameter;

[0122] In step S04 of the embodiment, updating the convolution layer bias parameter according to the learning rate includes: the vector processor traverses the convolution layer bias parameter and updates it according to the following formula: convolution layer bias parameter-(convolution layer bias parameter gradient*learning rate). For example, if the learning rate is lr=0.1 and the bias parameter gradient is db, then bias[k]=bias[k]-0.1*db[k].

[0123] The number of times of extracting the sub-matrix of dS rows of data from the input gradient error matrix by rows in the embodiment is O times, where dS=nH*nW and O is the number of channels of the forward calculation output data of the convolution layer.

[0124] In step S05 of the embodiment, the (N*nnH*nnW)*C order sub-matrix (the total number of sub-matrix extraction is kH*kW) is extracted from the input feature data matrix obtained in step S01 and transmitted to the vector array memory AM of each core. The specific steps include:

[0125] Step S501: let h=0, w=0, where the value range of h is {0, …, kH-1} and the value range of w is {0, …, kW-1}, and let c=0, m=0, n=0, where the value range of c is {0, …, N-1}, the value range of m is {0, …, nnH-1}, and the value range of n is {0, …, nnW-1}, and create a vector flag with a size of (N*nnH*nnW);

[0126] Step S502: determine whether h+m>pH-1&&h+m<H+pH&&w+n>pW-1 and w+n<(pW+W) are satisfied, if yes, go to S503, otherwise go to S507;

[0127] Step S503: construct a vector temp with a length of C, let i=0, where the value range of i is {0, …, C-1}, and t=(h+m-pH)*W+(w+n-pW);

[0128] Step S504: determine whether i is greater than C-1, if yes, go to 10.6, if not, extract the pixel point in the cth column of the i*H*W+t row in the input data matrix to the i th position in temp, and go to step 10.5.

[0129] Step S505: i is incremented by 1, and go to step S504;

[0130] Step S506: transmit the temp vector to the sub-matrix;

[0131] Step S507: n is incremented by 1, and determine whether n is greater than nnW-1, if yes, go to S508, if not, go to step S502;

[0132] Step S508: m is incremented by 1, n=0, and determine whether m is greater than nnH-1, if yes, go to S509, if not, go to step S502.

[0133] Step S509: c is incremented by 1, m=0, n=0, and determine whether c is greater than N-1, if yes, go to step S511, if not, go to step S502;

[0134] Step S510: w is incremented by 1, c=0, m=0, n=0, and determine whether w is greater than kW-1, if yes, go to S511, if not, go to step S502;

[0135] Step S511: h is increased by 1, w=0, c=0, m=0, n=0, it is judged whether h is greater than kH-1, if yes, the current flow is terminated, if not, the flow is transferred to step S502.

[0136] In step S06 of the embodiment, the specific steps of the vector processor performing the vectorized matrix multiplication calculation on the input error gradient matrix and the sub-matrix of the input feature data in step S05 include:

[0137] Step S601: the vector processor transmits the input data into the preset input data buffer of the vector array memory AM of the vector processor respectively, and the input data matrix has a size of (N*nnH*nnW)*C;

[0138] Step S602: the vector processor transmits the input error gradient matrix into the preset data buffer of the scalar memory SM of the vector processor respectively, and the input error gradient matrix has a size of (O*nnH*nnW)*N, i=0, j=0; step S603: x=i*N*nnH*nnW+j, o=x / (nnH*nnW*N), c=x / (nnH*nnW)-o*N, s=x / nnW-o*N*nnH-c*nnH, r=x%nnW, st=o*nnH*nnW+(s / S)*nnW+(r / S), if s%S is not equal to 0 or r%S is not equal to 0, jump to step S605, otherwise yc=j / (nnH*nnW), ym=j / nnW-yc

[0139] nnH, yn=j%nnW, if h+ym is greater than pH-1 and h+ym is less than pH+H and w+yn is greater than pW-1 and w+yn is less than pW+W, the vector processing unit VPU of the vector processor reads the jth row data of the input feature data matrix into a vector register;

[0140] Step S604: the scalar processing unit SPU of the vector processor reads the stth row cth column element of the input error gradient matrix into a scalar register, and performs multiplication and accumulation calculation on the scalar register and the scalar register obtained in step S603; step S605: j is increased by 1, it is judged whether j is greater than or equal to N*nnH*nnW, if yes, jump to step S606, if not, jump to step S603;

[0141] Step S606: i is increased by 1, it is judged whether i is greater than or equal to O, if yes, the current flow is terminated; if not, j=0 jump to step S603.

[0142] In step S06 of the embodiment, the specific steps of transmitting the gradient of the convolution layer weight parameter include: the vector processor extracts the element of the mth row and the nth column of the two-dimensional matrix obtained by performing the vectorized matrix multiplication to the mth row and the n*kH*kW+h*kW+w column of the convolution layer weight gradient matrix, where the value range of m is {0,…,O-1}, and the value range of n is {0,…,C-1}; the specific steps of updating the convolution layer weight parameter according to the learning rate include: the vector processor transmits the convolution kernel matrix and the convolution layer weight parameter gradient to the preset input data buffer area in the scalar array memory SM of each core of the vector processor, respectively, the vector accelerator reads the convolution kernel matrix and the convolution layer weight parameter gradient matrix row by row, and updates according to the following formula: current convolution kernel matrix row-(current convolution layer bias parameter gradient matrix row*learning rate), for example, the learning rate is lr=0.1, the weight parameter gradient is denoted as dw, and the convolution kernel matrix is denoted as Weight, then Weight[i][j]=Weight[i][j]–0.1*dw[i][j]. After repeating O times, the update of the convolution layer weight parameter is completed.

[0143] The present application can fully exert the SIMD and inter-core parallelism of the vector processor by expanding each convolution kernel data into vector data and simultaneously performing vector multiplication and accumulation calculation on all input data, greatly improves the calculation efficiency of the convolution neural network, and can avoid reduction summation between multiple processing units and improve the overall calculation efficiency of the vector processor by storing all input error gradient data of the same sample on one column and performing all multiplication and accumulation calculations of the convolution kernel data and the input error gradient data on the same VPE processing unit. The multi-sample multi-channel convolution back propagation vectorization implementation method of the present application not only has high calculation efficiency, but also can improve the calculation accuracy, can balance efficiency and accuracy, can be applied to batch processing of convolution back propagation calculation, and can support convenient and flexible setting of batch size.

[0144] The present application is further described below by taking the implementation of the convolution back propagation vectorization by using the above-mentioned method in the specific application embodiment as an example.

[0145] In the embodiment, the number of cores of the target vector processor is q=4, the number of vector processing units VPE of each core is p=16, the number of training samples is N, wherein N=q*p=64, the input feature map height H and width W of the convolutional neural network of the current convolution layer are both 2, the channel number C is 3, the convolution kernel size KC is 12, kH and kW are both 2, the number of convolution kernels O is 2, K=8, the step size S of convolution calculation is 1. Assuming that the number of 0 elements filled in the high and wide directions is both 1, the original image is expanded to 4*4. It can be known from nH=H+2pH-kH+1 and nW=W+2pW-kW+1 that the output image size is 3*3. The detailed steps of the multi-sample multi-channel convolution back propagation vectorization implementation method in the embodiment are as follows:

[0146] Step 1. The input feature data of the forward calculation of the convolutional neural network and the input error gradient of the back propagation are stored as two-dimensional matrices in the order of sample dimension priority. The input error gradient of the current layer of the convolutional neural network is an M*N order matrix, each column of the M*N order matrix stores the input error gradient data of a single sample, and the storage order of the column is the image width nW direction priority, then the image height nH direction priority, and finally the image channel O direction priority. The elements of the input error gradient matrix are specifically represented as y[i][m], wherein the column coordinate m represents the m+1th sample, and the value range is {0, 1, 2,..., N-1}; the row coordinate i represents the i+1th input feature value of the sample, and the value range is {0, 1, 2,..., O*nH*nW-1}.

[0147] The data of the convolution kernel is stored in the order of the number of convolution kernels priority, specifically including: the data of the convolution kernel is stored in the order of an O*KC=2*12 order matrix in the off-chip memory of the vector processor, wherein O is the number of convolution kernels, KC=kH*kW*C=12 is the number of pixel data of a single convolution kernel, and C is the channel number of the input picture.

[0148] Each row of the above O*K order matrix stores the pixel data of a single convolution kernel, and the storage order of the row is the convolution kernel height kW direction priority, then the convolution kernel height kH direction priority, and finally the channel C direction priority. The elements of the convolution kernel data matrix are specifically represented as w[j][c], wherein the row coordinate j represents the j+1th convolution kernel, and the value range is {0, 1, 2,..., O-1}; the column coordinate c represents the c+1th pixel data value of the convolution kernel, and the value range is {0, 1, 2,..., C*kH*kW-1}. The elements of the bias data column vector are represented as bias[j], wherein the coordinate j corresponds to the j+1th convolution kernel, and the value range is {0, 1, 2,..., O-1}; each convolution kernel corresponds to a bias data value.

[0149] Step 2: The vector processor extracts V rows of data from the input error gradient matrix in step 1 by row to form a first sub-matrix and transmits it to the vector array memory AM of each core, where 0 < V <= K, K = O * kH * kW = 8.

[0150] Specifically, V rows of data are extracted from the M*N=18*64 order input error gradient matrix by row to form a V*N order sub-matrix, which is transmitted to the vector array memory AM of each core. Each core further calculates the C*N=3*64 order output error gradient matrix through vectorized matrix multiplication calculation of scalar and vector cooperation. After extracting the first sub-matrix according to the above steps S201-S208, V=valid rows of data can be extracted from the M*N=18*64 order input error gradient matrix by row to form a V*N=valid*64 order sub-matrix of the input error gradient matrix. The calculation process is realized by filling 0 elements, and the actual storage of 0 elements is not required. The total number of times of extracting V rows of data from the M*N order input error gradient matrix by row to form a V*N order sub-matrix of the input error gradient matrix is H*W=2*2, where: nH=H+2pH-kH+1, nW=W+2pW-kW+1, nnH=nH, nnW=nW, H'=H+2pH, W'=W+2pW.

[0151] Step 3: The vector processor transmits the convolution kernel matrix in step S1 to the scalar memory SM of each core.

[0152] Step 4: The vector processor executes vectorized matrix multiplication on the input error gradient sub-matrix and the convolution kernel matrix in step S02 by each core to calculate the output error gradient matrix.

[0153] Specifically, the O*KC=2*12 order convolution kernel data matrix and the V*N=valid*64 order input error gradient sub-matrix are executed vectorized matrix multiplication calculation and parallelized matrix multiplication calculation of each core, and the specific implementation steps are described above in steps S301-S306.

[0154] Step 5: The calculation result of the output error gradient matrix in step 4 is transmitted to the off-chip memory of the vector processor.

[0155] Step 6: Repeat steps 2 to 5 until all input error gradient matrix calculations are completed.

[0156] Step 7: The vector processor extracts a sub-matrix of dS=3*3=9 rows of data from the input error gradient matrix in step S01 by row and transmits it to the vector array memory AM of each core, and performs vectorized matrix summation calculation on the sub-matrix to obtain the gradient of the convolution layer bias parameter, as Figure 6As shown, the element index all starts from 0, R(x, y) represents the value of the xth input image at the 0th channel at the yth position. G(x, y) represents the value of the xth input image at the 1st channel at the yth position. Where db[0] represents the gradient value of the vector bias at the 0th position, which is the sum of the gradient values of all input samples at the 0th channel. db[1] represents the gradient value of the vector bias at the 1st position, which is the sum of the gradient values of all input samples at the 1st channel. Figure 6 In the above, the value of db[0] is the sum of all element values from the 0th row to the 8th row of the input error gradient matrix, and the value of db[1] is the sum of all element values from the 0th row to the 8th row of the input error gradient matrix.

[0157] The vectorized matrix summation calculation on the sub-matrix is performed as described in steps S401-S402 above, wherein the input error gradient matrix scale of each core is (nH*nW)*p=9*16 order, and the vector processing component VPU of each core of the vector processor traverses the input error gradient matrix nH*nW=9 rows, and the vector processor completes the calculation of a component in the convolution layer bias parameter gradient.

[0158] Step 8. The convolution layer bias parameter gradient calculation result is transmitted to the off-chip memory of the vector processor.

[0159] Step 9. Repeat steps 7 and 8 until the calculation of all convolution layer bias parameter gradients is completed, and update the convolution layer bias parameter according to the learning rate, specifically let the learning rate be lr=0.1, the bias parameter gradient be db, and then bias[k]=bias[k]-0.1*db[k], as shown in Figure 7 .

[0160] Step 10: The vector processor transmits the input error gradient matrix in step 1 to the scalar memory SM of each core, and extracts the (N*nnH*nnW)*C=576*3 order sub-matrix from the input feature data matrix in step 1 to the vector array memory AM of each core, where N is the number of training samples. The sub-matrix is extracted as described in steps S501-S511 above.

[0161] Step 11: The vector processor performs vectorized matrix multiplication calculation on the input error gradient matrix and the input feature data sub-matrix in step 10 for each core, and the specific steps are as described in steps S601-S606 above. The gradient of the convolution layer weight parameter is calculated.

[0162] Step 12: The calculation result of the step 11 convolution layer weight parameter gradient is transmitted to the off-chip memory of the vector processor. The specific vector processor extracts the element of the mth row and nth column of the two-dimensional matrix obtained in step 11 to the mth row and n*kH*kW+h*kW+w column of the convolution layer weight gradient matrix.

[0163] Step 13: Repeat steps 10 to 12 until the update of all convolution layer weight parameter gradients is completed, and update the convolution layer weight parameters according to the learning rate. After repeating 2 times, the update of the convolution layer weight parameters is completed, and the learning rate is lr=0.1, the weight parameter gradient is denoted as dw, and the convolution kernel matrix is denoted as Weight, then Weight[i][j]=Weight[i][j]-0.1*dw[i][j].

[0164] The output error gradient matrix obtained by executing the above steps 2-6 has a size of P*N order matrix, where N is the number of training samples; P=H*W*C is the number of input features of a single sample. The P*N order output error gradient matrix is stored in the off-chip memory of the vector processor in the sample dimension first manner, that is, each column of the P*N order matrix stores the output error gradient of a single sample, and the storage order of the column is the image width W direction first, then the image height H direction first, and finally the channel C direction first. The element of the output error gradient matrix is specifically represented as a[j][m], where the column coordinate m represents the m+1th sample, and the value range is {0, 1, 2,..., N-1}; the row coordinate j represents the j+1th output feature value of the sample, and the value range is {0, 1, 2,..., H*W*C-1}.

[0165] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Although the present application has been disclosed as above with the preferred embodiment, it is not intended to limit the present application. Therefore, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present application, without departing from the technical solution of the present application, shall fall within the scope of protection of the technical solution of the present application.

Claims

1. A multi-sample multi-channel convolution back propagation vectorization implementation method, characterized in that the steps Including: S01. Store the input feature data of the forward calculation of the convolutional neural network and the input error gradient of the backpropagation as a two-dimensional matrix in the order of priority of the sample dimension, to obtain the input feature data and the input error gradient matrix; S02. The vector processor extracts V rows of data from the input error gradient matrix obtained in step S01 to form a first sub-matrix and transfers it to the vector array memory AM of each core, and transfers the convolutional kernel matrix to the scalar memory SM of each core, where 0 < V <= K, K = O * kH * kW, kH is the height of the convolutional kernel, kW is the width of the convolutional kernel, and O is the number of convolutional kernels; S03. Each core of the vector processor performs vectorized matrix multiplication on the first sub-matrix of the input error gradient matrix and the convolutional kernel matrix, calculates the output error gradient matrix and transfers it to the off-chip memory of the vector processor; repeat step S02 and step S03 until all output error gradient matrices are calculated; S04. The vector processor extracts dS rows of data from the input error gradient matrix obtained in step S01 to form a second sub-matrix and transfers it to the vector array memory AM of each core, where dS = nH * nW, nH is the height of the output feature map, and nW is the width of the output feature map; perform vectorized matrix summation calculation on the second sub-matrix to obtain the gradient of the convolutional layer bias parameter and transfer it to the off-chip memory of the vector processor; Repeat step S04 until all gradients of the convolutional layer bias parameter are calculated, and then update the convolutional layer bias parameter according to the learning rate; S05. The vector processor transfers the input error gradient matrix obtained in step S01 to the scalar memory SM of each core, and extracts a (N * nnH * nnW) * C-order sub-matrix from the input feature data matrix obtained in step S01 and transfers it to the vector array memory AM of each core, where N is the number of training samples, C is the number of channels of the input feature map, nnH and nnW respectively represent the logical height and width after padding (S - 1) zeros between the rows and columns of the output feature map, and S is the stride of the convolutional kernel; S06. The vector processor performs vectorized matrix multiplication calculation on the input error gradient matrix and the sub-matrix of the input feature data in step S05, calculates the gradient of the convolutional layer weight parameter and transfers it to the off-chip memory of the vector processor, repeat step S05 and step S06 until all gradients of the convolutional layer weight parameter are calculated, and then update the convolutional layer weight parameter according to the learning rate.

2. The method for implementing multi-sample multi-channel convolution back propagation vectorization according to claim 1, characterized in that: When storing the input feature data of the forward calculation of the convolutional neural network and the input error gradient of the backpropagation in step S01, each column stores a sample data, and the storage order of each column is as follows: first, it is in the order of the image width W direction first, then the image height H direction first, and finally the image channel C direction first; the convolutional kernel data is stored as a two-dimensional convolutional kernel matrix, where each row of the matrix represents a convolutional kernel, and the storage order of each row is first in the order of the convolutional kernel width kW direction first, then the image convolutional kernel kH direction first, and finally the convolutional kernel channel O direction first; the convolutional kernel data is specifically stored continuously in the off-chip memory of the vector processor according to an O*KC-order matrix, where KC = kH*kW*C is the number of pixel data of a single convolutional kernel.

3. The method for implementing multi-sample multi-channel convolution back propagation vectorization according to claim 1, characterized in that: In step S02, the total number of times of extracting V rows of data by row to form the first sub-matrix is H*W, and the calculation of the entire output error gradient matrix is completed, where: nH = (H + 2pH - kH) / S + 1, nW = (W + 2pW - kW) / S + 1, nnH = nH + (nH - 1)*(S - 1), nnW = nW + (nW - 1)*(S - 1), H' = H + 2pH, W' = W + 2pW M = nH*nW*O Where, H and W are respectively the image height and image width of the input feature map of the forward calculation of the current convolutional layer, pH and pW are respectively the number of 0 elements filled in the height and width directions, H' and W' respectively represent the logical height and logical width after padding 0 to the input feature map, and M represents the number of pixel points of a single output feature map.

4. The method for implementing multi-sample multi-channel convolution back propagation vectorization according to claim 3, characterized in that: In step S02, the specific steps of extracting V rows of data by row to form the first sub-matrix are as follows: Step S201: Let valid = 0, h = 0, w = 0, the value range of h is {0,..., H' - 1}, the value range of w is {0,..., W' - 1}, and determine whether it satisfies h < pH or satisfies h >= (pH + H) or satisfies w < pW or satisfies w >= (pW + W). If so, go to step S202, otherwise go to S204; Step S202: Increment w by 1, and determine whether it satisfies w > pW - 1. If not, go to step S201. If so, go to step S2.3; Step S203: Increment h by 1, w = 0, and determine whether it satisfies h > pH - 1. If so, end. If not, go to step S201; Step S204: Construct a vector flag with a length of K, let o = 0, m = 0, n = 0, the value range of o is {0,..., O - 1}, the value range of m is {0,..., kH - 1}, and the value range of n is {0,..., kW - 1}; Step S205: Determine whether (h+m>=(kH-1)&&(w+n)>=(kW-1)&&(h+m)<=(kH+nnH-2)&&(w+n)<=(kW+nnW-2)) is true. If so, go to step S206. If so, continue to determine whether (h+m-kH+1)%S==0&&(w+n-kW+1)%S==0 is true, where % is a modulo operation. If not, set the o*kH*kW+m*kW+nth element of the vector flag to 0. If true, define t as follows: t=(h+m-kH+1) / s*nW+(w+n-kW+1) / S+o*nH*nW, And set the o*kH*kW+m*kW+nth element of the vector flag to 1, extract the tth row of the input error gradient matrix, and add 1 to valid; Step S206: n is incremented by 1, and it is determined whether n>=kW is satisfied. If not, go to step S205; if yes, go to step S207; Step S207: m increases by 1, n = 0, and determines whether m>=kH is satisfied. If not, go to step S205; if yes, go to step S208; Step S208: o increases by 1, m=0, n=0, and determines whether the value of o>=0 is satisfied. If not, go to step S205; if yes, go to step S03.

5. The method for implementing multi-sample multi-channel convolution back propagation vectorization according to claim 4, characterized in that: In step S03, the specific steps of each core of the vector processor performing vectorized matrix multiplication on the first sub-matrix of the input error gradient matrix and the convolution kernel matrix include: Step S301: The vector processor transmits the extracted first sub-matrix of the input error gradient matrix to a preset input buffer in the vector array memory AM of each core of the vector processor. The scale of the first sub-matrix of the input error gradient matrix transmitted by each core is (valid*kH*kW)*p, where p is the number of vector processing units (VPEs) of each core. Step S302: The vector processor transmits the convolution kernel matrix to the convolution kernel data buffer preset in the scalar memory SM of each core of the vector processor respectively. The scale of the convolution kernel matrix transmitted by each core is O*KC order, and i=0, j=0, invalid=0; Step S303: Let indx = i*O*kH*kW+j, o = indx / (O*kH*kW), u = indx%(kH*kW), c = indx / (kH*kW)-o*O, the scalar processing unit SPU of each core of the vector processor sequentially reads the c-th row and kH*kW-u-1+o*kH*kW column elements from the convolution kernel data buffer into a scalar register, and determines whether the input error gradient matrix row corresponding to the read convolution kernel data is 0. If it is invalid, increment by 1 and continue to read the next element; if not, broadcast it to a vector register through a scalar broadcast instruction; Step S304: The vector processing unit (VPU) of each core of the vector processor sequentially reads the (j-invalid)-th row of input error gradient data from the input error gradient buffer into a vector register, performs multiplication and accumulation calculation on the vector register and the vector register obtained in step S303, and j increases by 1; Step S305: Determine whether j is greater than or equal to O*kH*kW. If not, jump to step S303, and the reading position in step S304 moves to the next line. If so, jump to step S306; Step S306: Determine whether i is greater than or equal to C. If not, set j = 0 and jump to step S303. The read position in step S304 returns to the starting address of the input error gradient data buffer. If so, the vector processor completes the calculation of the C*N order output error gradient data.

6. The method for implementing multi-sample multi-channel convolution back propagation vectorization according to claim 1, characterized in that: In step S03, the specific steps of storing the calculation result of the output error gradient matrix in the off-chip memory of the vector processor are: The vector processor transmits the obtained output error gradient matrix to the preset output buffer of each core of the vector processor. The matrix scale of each core is C*p order, where p is the number of vector processing units (VPEs) of each core. The i-th row in the matrix is ​​transmitted to the (h-pH)*W+(w-pW)+i*H*W row of the output buffer.

7. The method for implementing multi-sample multi-channel convolution back propagation vectorization according to any one of claims 1 to 6, characterized in that: In step S04, the specific steps of performing vectorized matrix summation calculation on the second sub-matrix include: Step S401: The vector processor transmits the second sub-matrix of the input error gradient matrix to a preset input buffer in the vector array memory AM of each core of the vector processor. The scale of the input error gradient matrix transmitted by each core is (nH*nW)*p order, where p is the number of vector processing units (VPEs) of each core. Step S402: The vector processing unit VPU of each core of the vector processor traverses the second submatrix of the input error gradient matrix, and sequentially reads a row from the second submatrix of the input error gradient matrix into a vector register, performs vector summation calculation, and writes the result into a scalar register; the scalar processing unit SPU of each core uniformly stores the value of the scalar register into the scalar register of one of the cores and performs accumulation calculation; after traversing (nH*nW) rows, the vector processor completes the calculation of the corresponding component in the convolution layer bias parameter gradient; In step S04, updating the convolution layer bias parameter according to the learning rate includes: a vector processor traversing the convolution layer bias parameter and updating it according to the following formula: convolution layer bias parameter - (convolution layer bias parameter gradient * learning rate).

8. The method for implementing multi-sample multi-channel convolution back propagation vectorization according to any one of claims 1 to 6, characterized in that: In step S05, extracting (N*nnH*nnW)*C order submatrices from the input feature data matrix obtained in step S01 and transmitting them to the vector array memory AM of each core includes: Step S501: Let h=0, w=0, where the value range of h is {0, ..., kH-1}, the value range of w is {0, ..., kW-1}, and let c=0, m=0, n=0, the value range of c is {0, ..., N-1}, the value range of m is {0, ..., nnH-1}, and the value range of n is {0, ..., nnW-1}, and create a vector flag of size (N*nnH*nnW); Step S502: Determine whether h+m>pH-1&&h+m<H+pH&&w+n> pW-1 and w+n<(pW+W), if so, go to S503, otherwise go to S507; Step S503: construct a vector temp of length C, set i = 0, the value range of i is {0, ..., C-1}, t = (h + m - pH) * W + (w + n - pW); Step S504: Determine whether i is greater than C-1. If so, go to step 10.

6. If not, extract the pixel at the i*H*W+tth row and the cth column in the input data matrix to the i-th position in temp, and go to step 10.

5. Step S505: i is increased by 1, and the process goes to step S504; Step S506: transferring the temp vector to the sub-matrix; Step S507: n is incremented by 1, and it is determined whether n is greater than nnW-1. If so, the process goes to S508; if not, the process goes to step S502. Step S508: m increases by 1, n=0, determine whether m is greater than nnH-1, if so, go to S509, if not, go to step S502; Step S509: c increases by 1, m=0, n=0, and determines whether c is greater than N-1. If so, go to step S511; if not, go to step S502; Step S510: w increases by 1, c=0, m=0, n=0, determine whether w is greater than kW-1, if so, go to S511, if not, go to step S502; Step S511: h increases by 1, w=0, c=0, m=0, n=0, determine whether h is greater than kH-1, if so, the current process is terminated, if not, go to step S502.

9. The method for implementing multi-sample multi-channel convolution back propagation vectorization according to any one of claims 1 to 6, characterized in that: In step S06, the vector processor performs vectorized matrix multiplication calculation on the input error gradient matrix and the sub-matrix of the input feature data in step S05, specifically comprising the following steps: Step S601: the vector processor transmits input data to the input data buffers preset in the vector array memory AM of the vector processor, and the input data matrix size is (N*nnH*nnW)*C order; Step S602: The vector processor transmits the input error gradient matrix to a preset data buffer in the scalar memory SM of the vector processor. The scale of the input error gradient matrix is ​​(O*nnH*nnW)*N, and i=0, j=0. Step S603: Let x=i*N*nnH*nnW+j, o=x / (nnH*nnW*N), c=x / (nnH*nnW)-o*N, s=x / nnW-o*N*nnH-c*nnH, r=x%nnW, st=o*nnH*nnW+(s / S)*nnW+(r / S), if s%S is not equal to 0 or r%S is not equal to 0, jump to step S605, otherwise set: yc=j / (nnH*nnW), ym=j / nnW–yc*nnH, yn=j%nnW, if h+ym is greater than pH–1 and h+ym is less than pH+H and w+yn is greater than pW-1 and w+yn is less than pW+W, then the vector processing unit VPU of the vector processor reads the j-th row data of the input feature data matrix into a vector register; Step S604: the scalar processing unit SPU of the vector processor reads the element in the st-th row and the c-th column from the input error gradient matrix into a scalar register, and performs a multiplication-accumulation calculation on the scalar register and the scalar register obtained in step S603; Step S605: j is incremented by 1 to determine whether j is greater than or equal to N*nnH*nnW. If yes, jump to step S606; if no, jump to step S603. Step S606: i is incremented by 1, and it is determined whether i is greater than or equal to 0. If yes, the current process ends; if not, j=0, and the process jumps to step S603.

10. The method for implementing multi-sample multi-channel convolution back propagation vectorization according to any one of claims 1 to 6, characterized in that: In step S06, the specific step of transmitting the gradient of the convolution layer weight parameter includes: the vector processor extracts the element of the mth row and nth column of the two-dimensional matrix obtained by the current vectorized matrix multiplication calculation to the mth row and n*kH*kW+h*kW+w column of the convolution layer weight gradient matrix, where the value range of m is {0,…O-1}, and the value range of n is {0,…,C-1}; the specific step of updating the convolution layer weight parameter according to the learning rate includes: the vector processor transmits the convolution kernel matrix and the convolution layer weight parameter gradient to the input data buffer preset in the scalar array memory SM of each core of the vector processor respectively, the vector accelerator reads the convolution kernel matrix and the convolution layer weight parameter gradient matrix row by row, and updates them according to the following formula: current convolution kernel matrix row - (current convolution layer bias parameter gradient matrix row * learning rate), and completes the update of the convolution layer weight parameter after repeating O times.

Images