A convolutional neural network data processing method and related device

Abstract

A convolutional neural network (CNN) data processing method and related equipment are disclosed, specifically for use in the field of CNNs. The method includes: acquiring an input matrix, where each element of the input matrix is ​​a partial derivative of the loss function with respect to the output feature map obtained by the CNN in the forward pass; acquiring a first convolutional kernel of the CNN, wherein the size of the first convolutional kernel is n*n and the stride of the first convolutional kernel is m; splitting the first convolutional kernel into multiple sub-convolutional kernels according to the stride m; performing convolution multiplication on each of the multiple sub-convolutional kernels with the input matrix to obtain multiple sub-result matrices; and combining the elements of the multiple sub-result matrices according to the stride m to obtain an output result. This allows for efficient calculation of the output result of the reverse propagation, realizing the reverse propagation of the loss value.

Description

Technical Field

[0001] This application relates to convolutional neural networks (CNNs), and more particularly to a convolutional neural network data processing method and related equipment. Background Technology

[0002] In the field of artificial intelligence, CNNs are widely used in image classification, image recognition, feature extraction, and other areas. The convolution operator is the core of the convolutional neural network, determining its functionality and performance. The input to the convolution operator mainly consists of three parts: input features... Figure X The convolution matrix consists of a weight matrix W (convolution kernel) and a bias term b, with the output being the output feature map Y. The convolution is calculated by first applying an image-to-column (img2col) algorithm to the features. Figure X The data is unfolded and transformed into a two-dimensional data matrix X'. Then, the data matrix X' is multiplied by the convolution kernel (general matrix multiplication, GEMM). Finally, the bias term b is added to obtain the output feature map Y.

[0003] To achieve autonomous learning in a convolutional neural network, the output feature map Y needs to be evaluated using a loss function. During the backward propagation of the convolutional neural network, the partial derivative dE / dY of the loss function with respect to the output feature map Y is used as input. Then, dE / dY is reverse-propagated to obtain the output dE / dX, thereby realizing the backward propagation of the loss function. Finally, based on dE / dX, dE / dW, and dE / db, various optimizer algorithms are used to iteratively update the convolutional kernel W and the bias term b, completing the autonomous learning process of the neural network.

[0004] In the calculation of forward convolution, the img2col algorithm is used to process the features. Figure X To perform data unfolding, the reverse operation naturally requires the inverse process of the img2col algorithm to restore the original data, that is, to reconstruct the original features by accumulating the unfolded two-dimensional data at corresponding positions. Figure X The partial derivative data feature maps dE / dX are of the same size. In the img2col algorithm, the overlapping region is very large during the movement of the convolution kernel on the feature map, resulting in a large amount of data redundancy in the output feature map Y. Therefore, in the inverse process, the data at the overlapping positions must be accumulated and calculated to reconstruct the original feature map. Figure X The partial derivative data feature map of the size will result in a large cumulative calculation overhead, which will seriously affect the chip's computing performance. Summary of the Invention

[0005] This application provides a convolutional neural network data processing method and related equipment, which is used to efficiently obtain dE / dX based on dE / dY and realize the reverse propagation of the loss value.

[0006] The first aspect of this application provides a convolutional neural network data processing method, including:

[0007] In the backpropagation process of a convolutional neural network (CNN), the input to a certain layer of the CNN is the input matrix. This input matrix can be the partial derivative of the loss function with respect to the output feature map obtained by the forward pass of the CNN. When performing the backpropagation operation on the input matrix, it is necessary to obtain the first convolutional kernel corresponding to that layer, where the size of the first convolutional kernel is n*n and the stride is m. First, according to the stride m, the first convolutional kernel is split into multiple sub-convolutional kernels. Then, each sub-convolutional kernel performs a convolution multiplication operation on the input matrix, thus obtaining multiple sub-result matrices. Finally, according to the stride m, the elements in the multiple sub-result matrices are recombined to obtain the output result. The output result is used as the input to the next layer of the CNN to continue the backpropagation operation, ultimately completing the backpropagation of the loss value.

[0008] By splitting the convolution kernel and then performing convolution operations with the input matrix separately, and finally recombining the resulting sub-results, dE / dX can be efficiently calculated without the need for the col2img process and without redundant computation during the backpropagation of the convolutional neural network. This reduces computational overhead and improves the computing performance of AI chips.

[0009] In conjunction with the first aspect of the embodiments of this application, in a first implementation of the first aspect of the embodiments of this application:

[0010] The forward propagation process of a convolutional neural network first involves processing the input features... Figure X Two-dimensional data is unfolded to obtain matrix X', and then matrix multiplication is performed between matrix X' and the second convolution kernel to obtain the output feature map Y. Therefore, in the backpropagation process, dE / dX needs to be obtained from dE / dY. According to mathematical operations, dE / dY needs to be convolved with the transpose of the second convolution kernel. Therefore, the first convolution kernel corresponding to the forward operation can be obtained by transposing the second convolution kernel along the subdiagonal.

[0011] In conjunction with the first aspect to the first implementation of the embodiments of this application, in the second implementation of the first aspect of the embodiments of this application:

[0012] When splitting the first convolutional kernel, it can be split according to the stride m of the convolutional kernel, splitting the first convolutional kernel into m*m sub-convolutional kernels; in each sub-convolutional kernel, the positions of two adjacent elements in the original first convolutional kernel are separated by a stride m.

[0013] In conjunction with the second embodiment of the first aspect of the embodiments of this application, in the third embodiment of the first aspect of the embodiments of this application:

[0014] The specific splitting steps are as follows: First, determine the data at the top left corner of the first convolutional kernel as the first element; then, determine the next position with a step size of m in the horizontal / vertical direction; and determine the data at this position as the second element; next, starting from the position of the second element, determine the third position with a step size of m, and determine the element at the third position as the third element... repeat this step until the entire first convolutional kernel has been moved, so that there are no more elements to be taken in the horizontal and vertical directions of the first convolutional kernel; then rearrange the obtained elements according to their positions in the first convolutional kernel and combine them to obtain the first sub-convolutional kernel; next, determine the element in the remaining position in the top left direction of the first convolutional kernel as the first element of the second sub-convolutional kernel; also take the data of the entire convolutional kernel with a step size of m to obtain the second sub-convolutional kernel; finally, repeat the data taking steps until the first convolutional kernel is completely split to obtain multiple sub-convolutional kernels.

[0015] In conjunction with the third embodiment of the first aspect of the embodiments of this application, in the fourth embodiment of the first aspect of the embodiments of this application:

[0016] After splitting the first convolutional kernel into multiple sub-convolutional kernels, each sub-convolutional kernel can perform a convolution multiplication operation on the input matrix with a stride of 1 to obtain multiple sub-result matrices.

[0017] When the first convolutional kernel is split into multiple sub-convolutional kernels, the size of the sub-convolutional kernels will become smaller. At the same time, performing a convolutional multiplication operation with a stride of 1 with the input matrix will greatly reduce the computational complexity, ensuring that there is no redundant computation in the matrix calculation. In addition, the characteristics of matrix multiplication will integrate matrix addition, eliminating the need to accumulate elements at the same position, completely eliminating the computational task of the VCU, and improving the computing power of the AI ​​chip.

[0018] In conjunction with the fourth embodiment of the first aspect of the embodiments of this application, in the fifth embodiment of the first aspect of the embodiments of this application:

[0019] After obtaining multiple sub-result matrices, the elements of the multiple sub-result matrices need to be rearranged to obtain a combined matrix. First, the size of the combined matrix can be determined based on the size of the sub-result matrices. Then, all elements in the multiple sub-result matrices are rearranged in the combined matrix, with two horizontally / vertically adjacent elements in each sub-result matrix separated by a movement step m in the horizontal / vertical direction of the combined matrix. Finally, the final output result is obtained based on the combined matrix.

[0020] In conjunction with the fifth embodiment of the first aspect of the embodiments of this application, in the sixth embodiment of the first aspect of the embodiments of this application:

[0021] The rearrangement steps are as follows: First, start arranging elements from the top left corner of the combined matrix, defining the top left corner as the first position. Fill this first position with the top left element of the first sub-result matrix; understandably, the first sub-result matrix is ​​obtained by convolving the first sub-convolution kernel with the input matrix. Then, determine the next position in the horizontal / vertical direction with a step size m, and fill this position with the second element of the first sub-result matrix that is horizontally / vertically adjacent to the top left element. Continue filling in this order until all elements of the first sub-result matrix are filled into the combined matrix. Then, fill in the elements of the second sub-result matrix... The elements, as can be understood, the second sub-result matrix is ​​obtained by convolving the second sub-convolution kernel with the input matrix; the position in the upper left direction where no element is filled is determined as the second position, and the upper left corner element of the second sub-result matrix is ​​filled into the second position; then, in the horizontal / vertical direction, the next position of the second position is determined with the movement step size m as the step size, and the second element of the second sub-result matrix that is adjacent to the upper left corner element in the horizontal / vertical direction is filled into this position, and the number of elements is filled in order until all elements of the second sub-result matrix are filled into the combined matrix. Finally, the above steps are repeated until all elements of all sub-result matrices are filled into the combined matrix.

[0022] In conjunction with the fifth to sixth embodiments of the first aspect of this application, in the seventh embodiment of the first aspect of this application:

[0023] After reorganizing the multiple sub-result matrices, it is necessary to consider the input features from the forward pass. Figure X The elements in the combination matrix are filtered according to the dimension, and then the final output dE / dX is obtained based on the filtered elements. It can be understood that the final dE / dX is related to the input features in the forward propagation. Figure X They have the same dimensions.

[0024] A second aspect of this application provides a convolutional neural network data processing apparatus, comprising:

[0025] An acquisition unit is used to acquire an input matrix, wherein the elements of the input matrix are the partial derivatives of the loss function with respect to the output feature map obtained by the convolutional neural network in the forward direction;

[0026] The acquisition unit is further configured to acquire the first convolutional kernel of the convolutional neural network; wherein the size of the first convolutional kernel is n*n, and the stride of the first convolutional kernel is m;

[0027] The processing unit is configured to split the first convolutional kernel into multiple sub-convolutional kernels according to the movement step m;

[0028] The computing unit is used to perform convolution multiplication operations on the input matrix by each of the plurality of sub-convolution kernels to obtain a plurality of sub-result matrices;

[0029] The arrangement unit is used to combine the elements in the multiple sub-result matrices according to the movement step size m to obtain the output result.

[0030] In conjunction with the second aspect of the embodiments of this application, in the first embodiment of the second aspect of the embodiments of this application:

[0031] The first convolutional kernel is the convolutional kernel corresponding to the reverse operation of the convolutional neural network, and the convolutional kernel corresponding to the forward operation of the convolutional neural network is the second convolutional kernel; wherein, the first convolutional kernel is obtained by transposing the elements in the second convolutional kernel along the subdiagonal.

[0032] In conjunction with the second aspect to the first embodiment of the embodiments of this application, in the second embodiment of the second aspect of the embodiments of this application:

[0033] The processing unit is specifically used to split the first convolutional kernel into m*m sub-convolutional kernels; wherein, in each of the m*m sub-convolutional kernels, two horizontally / vertically adjacent elements are separated by a movement step m in the horizontal / vertical direction of the first convolutional kernel.

[0034] In conjunction with the second embodiment of the second aspect of the embodiments of this application, in the third embodiment of the second aspect of the embodiments of this application:

[0035] The processing unit is specifically used for: determining the element at the top left corner of the first convolutional kernel as the first element; taking the next element in both the horizontal and vertical directions with a step size of m; combining all the obtained elements according to their positions in the first convolutional kernel to obtain the first sub-convolutional kernel; determining the element not taken in the top left direction as the first element in the second sub-convolutional kernel; taking the next element in both the horizontal and vertical directions with a step size of m; combining all the obtained elements according to their positions in the first convolutional kernel to obtain the second sub-convolutional kernel; repeating the above steps to obtain m*m sub-convolutional kernels.

[0036] In conjunction with the third implementation of the second aspect of the embodiments of this application, in the fourth implementation of the second aspect of the embodiments of this application:

[0037] The computing unit is specifically used to perform a convolution multiplication operation with a stride of 1 on the input matrix by each of the plurality of sub-convolution kernels to obtain a plurality of sub-result matrices.

[0038] In conjunction with the fourth embodiment of the second aspect of the embodiments of this application, in the fifth embodiment of the second aspect of the embodiments of this application:

[0039] The arrangement unit is specifically used to determine the size of the combined matrix based on the size of the plurality of sub-result matrices; rearrange all elements in the plurality of sub-result matrices in the combined matrix so that two elements that are horizontally / vertically adjacent in each sub-result matrix are separated by a movement step m in the horizontal / vertical direction of the combined matrix; and obtain the output result based on the combined matrix.

[0040] In conjunction with the fifth embodiment of the second aspect of the embodiments of this application, in the sixth embodiment of the second aspect of the embodiments of this application:

[0041] The arrangement unit is specifically used for: determining the top-left corner of the combined matrix as the first position, and filling the first position with the top-left element of the first sub-result matrix; the first sub-result matrix corresponds to the first sub-convolution kernel; determining the next position of the first position in the horizontal / vertical direction of the combined matrix with a step size of m, and filling the next position with the element of the first sub-result matrix that is adjacent to the top-left element in the horizontal / vertical direction, until all elements of the first sub-result matrix are filled into the combined matrix; determining the position where no element is filled in the top-left direction as the second position, and filling the second position with the top-left element of the second sub-result matrix; the second sub-result matrix corresponds to the second sub-convolution kernel; determining the next position of the second position in the horizontal / vertical direction of the combined matrix with a step size of m, and filling the next position with the element of the second sub-result matrix that is adjacent to the top-left element in the horizontal / vertical direction, until all elements of the second sub-result matrix are filled into the combined matrix; repeating the above steps until all elements of all sub-result matrices are filled into the combined matrix.

[0042] In conjunction with the fifth to sixth embodiments of the second aspect of this application, in the seventh embodiment of the second aspect of this application:

[0043] The arrangement unit is specifically used to filter the elements in the combination matrix according to the size n*n of the first convolution kernel; and to obtain the output result based on the elements of the filtered combination matrix.

[0044] A third aspect of this application provides an execution device that may include a processor and a memory coupled together. The memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the convolutional neural network data processing method described in the first aspect is implemented. For details regarding the steps executed by the execution device in various possible implementations of the first aspect, please refer to the first aspect; further details will not be repeated here.

[0045] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed on a computer, causes the computer to perform the convolutional neural network data processing method described in the first aspect.

[0046] A fifth aspect of this application provides a circuit system including a processing circuit configured to perform the convolutional neural network data processing method described in the first aspect.

[0047] A sixth aspect of this application provides a computer program that, when run on a computer, causes the computer to execute the convolutional neural network data processing method described in the first aspect.

[0048] A seventh aspect of this application provides a chip system including a processor for supporting an execution device in implementing the functions involved in the foregoing aspects, such as transmitting or processing data and / or information involved in the foregoing methods. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for a server or communication device. This chip system may be composed of chips or may include chips and other discrete devices.

[0049] In the technical solution provided in this application embodiment, the backpropagation process of the convolutional neural network can efficiently obtain dE / dX without the col2img process and without redundant calculations. By splitting the convolution kernel in the backpropagation process, and then using multiple sub-convolution kernels to perform simple convolution calculations on the input matrix respectively, the final dE / dX result can be obtained by rearranging the data, which reduces the computational overhead and improves the computing performance of the AI ​​chip. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the structure of a convolutional neural network computation model provided in an embodiment of this application;

[0051] Figure 2 This application provides an embodiment of a feature processing method using the img2col algorithm. Figure X A schematic diagram of the data expansion structure;

[0052] Figure 3 This is a schematic diagram of a convolution kernel for two-dimensional data unfolding, provided in an embodiment of this application.

[0053] Figure 4 A schematic diagram of another convolutional neural network computation model provided in this application embodiment;

[0054] Figure 5 This application provides a schematic diagram of a structure for restoring the position of dE / dX' data using the col2img algorithm, as an embodiment of the present application.

[0055] Figure 6 This is a schematic diagram of the structure of the AI ​​chip provided in the embodiments of this application;

[0056] Figure 7 A flowchart illustrating the convolutional neural network data processing method provided in this application embodiment;

[0057] Figure 8This is a schematic diagram of the structure of the transposed rearrangement of the convolution kernel provided in an embodiment of this application;

[0058] Figure 9 This is a schematic diagram of the structure of the 3*3 convolution kernel splitting provided in the embodiments of this application;

[0059] Figure 10 This is a schematic diagram of the 5*5 structure convolution kernel splitting provided in an embodiment of this application;

[0060] Figure 11 This is a schematic diagram of the sub-result matrix rearrangement provided in the embodiments of this application;

[0061] Figure 12 A schematic diagram of a specific inverse convolution operation provided for the implementation of this application;

[0062] Figure 13 This is a schematic diagram of the structure of the convolutional neural network data processing device provided in the embodiments of this application. Detailed Implementation

[0063] This application provides a convolutional neural network data processing method and related equipment for the reverse computation process of a convolutional neural network; it efficiently calculates dE / dX based on dE / dY, thereby realizing the reverse propagation of the loss function.

[0064] In today's field of artificial intelligence, convolutional neural networks (CNNs) are widely used in image classification, image recognition, feature extraction, and other areas, receiving widespread attention and application from both academia and industry. The convolution operator is the core of a CNN, and its functions are primarily determined by the convolution operator. The input to a convolution operator can consist of three parts: input features... Figure X The convolution operator consists of a weight matrix W (convolution kernel) and a bias term b, and its output is the output feature map Y. The mathematical relationship of the convolution operator can be expressed as Y = Conv(X,W) + b.

[0065] Generally, the most common computation method for convolutional neural networks is to first process the features using an image-to-column (img2col) algorithm. Figure X The data is expanded, and then the convolution kernel W is also expanded into a matrix, followed by matrix multiplication (GEMM), and finally addition with the bias term b. This process yields the output feature map Y. Figure 1 This is a schematic diagram of the structure of a convolutional neural network computation model provided in an embodiment of this application, such as... Figure 1 As shown, firstly, input features Figure XThe data X is expanded using the img2col algorithm, resulting in X'. Then, the data matrix X' is multiplied by the convolution kernel to obtain Y0. Finally, the AddBias operation is performed, adding a bias term b to Y0 to obtain the final output feature map Y. In summary, the convolution calculation process can be represented by the following mathematical formula:

[0066] Y = img 2col(X)*W + P*b

[0067] exist Figure 1 In this context, P is a unit vector, representing the input features. Figure X The dimensions are [N, Hi, Wi, Ci], i.e., features. Figure X The number of features is N, and each feature map has a length of Hi, a width of Wi, and a number of channels Ci; where the number of channels Ci is used to indicate the features. Figure X The structure, when Ci>1, indicates that the input feature map is three-dimensional (3D), requiring convolution with 3D kernels, and the number of channels in the kernels must be the same as the number of channels in the input feature map. Before performing GEMM, the 3D features must first be... Figure X The convolution kernel is unfolded into a two-dimensional matrix and then multiplied.

[0068] In terms of features Figure X When performing matrix expansion, the img2col algorithm is used, and the features are adjusted according to the convolution kernel size. Figure X Perform matrix expansion; Figure 2 This application provides an embodiment of a feature processing method using the img2col algorithm. Figure X A schematic diagram of the data expansion structure; such as Figure 2 As shown, features Figure X The quantity is N, in each feature Figure X In the diagram, the feature map has a length of Hi, a width of Wi, and the number of channels Ci. Specifically, Ci takes a value of 3. Therefore, the feature map... Figure X The corresponding convolution kernel also has 3 channels; the convolution kernel in the feature Figure X Slide on top, and each slide will display a feature. Figure X The feature values ​​are extracted at the corresponding positions, and then the feature values ​​obtained from each layer of the feature map are arranged in a row from left to right according to the channels; then the convolution kernel is applied to the feature map. Figure X Slide the slider up again to take a second value, and record the second value in the second row. Repeat this process of taking feature values, and finally expand the N feature maps into a feature matrix X' of [N*Ho*Wo,Ci*Hk*Wk], thus completing the feature extraction. Figure X The two-dimensional data unfolds.

[0069] Understandably, the length of the feature matrix X' is Ci*Hk*Wk, and Ci is the same as the original feature matrix. Figure X The number of channels is the same, and Hk*Wk is the same as the size of the convolution kernel. That is, in the feature matrix X', the number of data in each row is the number of features obtained at that position after the convolution kernel moves one position. Figure X The eigenvalues ​​in all channels; the width of the feature matrix X' is N*Ho*Wo, which means that the convolution kernel changes its position Ho*Wo times in a feature map. If the convolution kernel is in the original feature map... Figure X If the step size is 1, meaning the sliding interval of the convolution kernel on the original feature map is 1, then Ho and Wo will be related to the feature map. Figure X The length Hi and width Wi are the same; however, the convolution kernel does not take values ​​at a sliding interval of 1 during the sliding process, and the regions of the feature maps covered by the convolution kernel at adjacent time points overlap. Therefore, the img2col process involves a large amount of repeated data reading, and the unfolded X' contains a lot of repeated data. Ho and Wo are not equal to Hi and Wi, but there is a linear correspondence between them.

[0070] Similarly, the convolution kernel also needs to be unfolded into two dimensions. Figure 3 This application provides a schematic diagram of a convolution kernel for two-dimensional data unfolding, as illustrated in an embodiment of the present application. Figure 3 As shown, the number of convolutional kernels is Nk, the number of channels in each convolutional kernel is Ci, specifically Ci is 3, the length of the convolutional kernel is Hk, and the width is Wk. The convolutional kernel is expanded into a weight matrix W of [Ci*Hk*Wk, Nk], and then convolutionally multiplied with the feature matrix X'.

[0071] Therefore, the forward computation process of a convolutional neural network can be represented as:

[0072] Y=img 2col(X)*W+P*b=X'*W+P*b

[0073] Convolutional neural networks (CNNs) require continuous training. Based on given training samples, the parameters of the neural network are adjusted to optimize the model. Training mainly involves two processes: forward propagation and backward propagation. Forward propagation, as described above, involves multiple layers of the neural network and numerous calculations to obtain the output feature map Y and derive the loss value from it. Backward propagation, on the other hand, involves obtaining the gradient from the loss value and propagating it backward through the multiple layers of the neural network. This gradient value is used to update the various parameters of the neural network. The backward propagation process of a CNN uses the partial derivative of the loss function with respect to the feature map Y, dE / dY, as input to calculate the loss function with respect to the input feature map. Figure XThe partial derivatives dE / dX with respect to the convolution kernel W, dE / dW, and dE / db with respect to the bias term b; where the loss function calculated by one layer of the neural network is related to the input features. Figure X The partial derivatives dE / dX will be used as input to the next layer of the neural network to calculate the partial derivatives again, ultimately achieving the reverse propagation of the loss value.

[0074] When calculating dE / dX in the reverse direction, the path is the same as that of the output feature map Y calculated in the forward direction, but the direction is opposite. Figure 4 A schematic diagram of another convolutional neural network computation model provided in this application embodiment is shown below. Figure 4 As shown, during the forward propagation process of this computational model, the input features Figure X The img2col algorithm unfolds X to obtain X'. Then, X' is multiplied by the unfolded convolution kernel, i.e., the weight matrix W, to obtain Y0. Finally, the AddBias operation is performed, which adds a bias term b to Y0 to obtain the final output feature map Y. Similarly, the backpropagation process of this computational model corresponds to the forward propagation. The output partial derivative dE / dY has the same dimension as the feature map Y. Then, it is multiplied by the bias vector to obtain dE / dY0. Then, it is multiplied by the transpose of the weight matrix W to obtain dE / dX', where the dimension of dE / dX' is the same as X'. To restore it to dE / dX with the same dimension as X, the inverse process of img2col is required, i.e., the matrix-to-image (col2img) process. The inverse computation process of the convolutional neural network can be represented as:

[0075] dE / dX=col2img(dE / dX')=col2img(dE / dY*W T )

[0076] Figure 5 This application provides a schematic diagram of a structure for reconstructing the position of dE / dX' data using the col2img algorithm, as shown in the embodiments of this application. Figure 5 As shown, after obtaining dE / dX', data padding is performed according to the stride of the convolution kernel during the forward operation. Based on the correspondence between the X' data and X's position, the row vectors in the dE / dX' data corresponding to X' are filled into the feature map region covered by the convolution kernel. This involves accumulating the dE / dX' data and then filling it into the corresponding positions, ultimately obtaining the feature map region. Figure X Data with the same dE / dX dimensions.

[0077] Since the col2img process is the inverse of the img2col process in the forward computation, the stride of the convolutional kernel on the feature map during img2col is not 1. When the size of the convolutional kernel is large, it can lead to issues with the feature map. Figure X The overlapping area caused by the upward sliding is very large, meaning that a position will be sampled multiple times. Therefore, the reverse col2img process will involve the accumulation and calculation of a large amount of overlapping position data, which will result in a large computational overhead and seriously affect the computing performance of AI chips.

[0078] This application provides a novel method for calculating dE / dX, which efficiently obtains dE / dX without the need for the col2img process and without redundant computation. Specifically, the transposed and rearranged convolutional kernels are split, and then the split convolutional kernels are used to perform convolution calculations on the dE / dY feature maps respectively. Finally, the data is rearranged to obtain the final dE / dX result.

[0079] Figure 6 This is a schematic diagram of the structure of the AI ​​chip provided in the embodiments of this application, such as... Figure 6 As shown, the main body of the AI ​​chip consists of a matrix computing unit (MCU) and a vector computing unit (VCU). The MCU is mainly responsible for matrix multiplication operations, while the VCU is responsible for vector processing of data, such as vector multiplication, addition, exponentiation, logarithmic operations, and comparisons. Specifically, it can also implement functions such as matrix splitting and combination as described in other embodiments of this application. The specific implementation of matrix splitting and combination is not further described here, but can be referred to other embodiments. The left matrix buffer, right matrix buffer, and result buffer are used to store input and output data related to AI chip operations; on-chip memory (OCM) is used for data caching; and the memory transfer engine... The engine (MTE) is mainly used to complete the data interaction between the AI ​​chip and the external DDR / HBM memory, as well as the data transfer between the various buffers on the chip. Specifically, the left matrix buffer obtains the left matrix data transferred from the DDR / HBM memory and transmits it to the MCU. The right matrix buffer obtains the right matrix data transferred from the DDR / HBM memory and also transmits it to the MCU. The MCU performs matrix multiplication on the left and right matrices and inputs the result into the result buffer. The result buffer exchanges data with the VCU, performs vector processing on it, and transmits the result to the OCM. The OCM and MTE exchange data, and finally, the MTE exchanges data with the outside world.

[0080] Figure 7 This is a flowchart illustrating the convolutional neural network data processing method provided in the embodiments of this application, as shown below. Figure 7As shown, convolutional neural network data processing methods include:

[0081] 701. Obtain the input matrix and the second convolution kernel.

[0082] In the backpropagation process of a convolutional neural network, the gradient needs to be backpropagated through multiple layers of the neural network. Understandably, we first need to determine the output feature map Y obtained from the forward operation of a certain layer of the neural network structure. Then, we obtain the partial derivative dE / dY of the loss value with respect to the feature map Y based on the loss function. Finally, we obtain the input matrix based on the partial derivative dE / dY. Understandably, the dimension of the dE / dY data is the same as that of the output feature map Y. For example, dE / dY can be expanded into two dimensions to obtain the input matrix. The size of this input matrix is ​​the same as the size of the matrix obtained after convolution multiplication of X' with the convolution kernel during the forward propagation process.

[0083] The second convolutional kernel is the convolutional kernel used in the forward propagation of this network layer. Since the output feature map Y is obtained by convolving X' with the second convolutional kernel, to reverse the process, it is necessary to perform calculations based on the second convolutional kernel to restore dE / dY to the input feature map. Figure X Dimensions.

[0084] 702. Transpose the second convolution kernel along the subdiagonal to obtain the first convolution kernel.

[0085] According to the principle of matrix multiplication, the second convolution kernel, which is multiplied by X' during the forward propagation, needs to be transposed and rearranged along its secondary diagonal to obtain the first convolution kernel. Only then can the convolution operation between the input matrix and the first convolution kernel be restored to the input features. Figure X The dimensions of the second convolution kernel. For example, the second convolution kernel is transposed along the subdiagonal in a two-dimensional plane of length and width. If the size of the second convolution kernel is n*n, then the shape dimensions of the first convolution kernel and the second convolution kernel obtained after transposition and rearrangement remain unchanged.

[0086] Figure 8 This is a schematic diagram of the transposed rearrangement of the convolution kernel provided in this embodiment, as shown below. Figure 8 As shown, if the size of the convolution kernel is a 3*3 structure, then the elements on the secondary diagonal of the convolution kernel are the elements numbered 3, 5, and 7. Then, the positions of each element are swapped along the secondary diagonal to obtain the transposed first convolution kernel. For example, in this convolution kernel, the element numbered 1 will be in the original position numbered 9 after transposition. It can be understood that the size of the transposed and rearranged convolution kernel is the same as that of the original convolution kernel, which is still a 3*3 structure.

[0087] Similarly, if the convolution kernel has a 5x5 structure, then the elements on the secondary diagonal of the kernel are the elements numbered 5, 9, 13, 17, and 21. Then, the elements are swapped along the secondary diagonal to obtain the transposed first convolution kernel. For example, in this convolution kernel, the element at position 1 will be transposed to the original position 25, and the element at position 7 will be transposed to the original position 19. It can be understood that the size of the transposed and rearranged convolution kernel is the same as that of the original convolution kernel, which is still a 5x5 structure.

[0088] 703. Split the first convolutional kernel to obtain multiple sub-convolutional kernels.

[0089] After transposing the original convolutional kernel from the forward propagation to obtain a new first convolutional kernel, it needs to be split according to the stride m of the convolutional kernel. Specifically, the size of the first convolutional kernel needs to be determined first, and it is divided into m*m sub-convolutional kernels. For example, if the stride of the first convolutional kernel is 2, it needs to be split into 4 sub-convolutional kernels. If the stride of the first convolutional kernel is 3, it needs to be split into 9 sub-convolutional kernels. In each sub-convolutional kernel, the two adjacent elements are separated by a stride m in the original convolutional kernel.

[0090] Optionally, the specific splitting method can be divided into the following steps:

[0091] 1) First, determine the data at the top left corner of the first convolution kernel as the first element;

[0092] 2) Then, determine the next position in the horizontal / vertical direction with a step size of m; and determine the data at that position as the second element;

[0093] 3) Next, starting from the position of the second element, with a step size of m, determine the third position and set the element at the third position as the third element... Repeat this step until the entire first convolution kernel has been moved so that there are no more elements to be taken in the horizontal and vertical directions of the first convolution kernel;

[0094] 4) Then rearrange the obtained elements according to their positions in the first convolution kernel, and combine them to obtain the first sub-convolution kernel;

[0095] 5) Next, determine the element in the remaining position in the top left direction of the first convolution kernel as the first element of the second sub-convolution kernel; also take the number of the entire convolution kernel according to m to obtain the second sub-convolution kernel;

[0096] 6) Finally, repeat the data extraction steps until the first convolution kernel is completely split into multiple sub-convolution kernels.

[0097] For example, such as Figure 9 As shown, the convolution kernel has a 3*3 structure and a stride of 2. Therefore, it needs to be split into 4 sub-convolution kernels. It can be understood that the numbers on the convolution kernel only represent the positions of the data taken. Every two positions after a stride belong to the same data take. Therefore, the convolution kernel can mark the first element position in the upper left corner as 1. Every time a data take is required after a stride, this position is also marked as 1. Finally, the element marked as 1 is determined as the element of the first sub-convolution kernel and combined in order. Then, the upper left corner of the remaining elements is marked as 2, and the same method is used to mark the positions. Finally, the element marked as 2 is determined as the element of the second sub-convolution kernel, until it is split into 4 sub-convolution kernels.

[0098] For example, such as Figure 10 As shown, the convolutional kernel has a 5x5 structure and a stride of 3. Therefore, it needs to be split into 9 sub-convolutional kernels. It's understandable that the numbers on the convolutional kernel only represent the position of the data being taken; every two positions after a stride belong to the same data taking. Therefore, the convolutional kernel can mark the first element at the top left corner as 1. After each stride, the position is marked again, and the element at the marked position is also marked as 1. Finally, the element at the marked position 1 is determined as the element of the first sub-convolutional kernel, and they are combined sequentially. Then, the top left corner of the remaining elements is marked as 2, and the same method is used to mark the positions. Finally, the element at the marked position 2 is determined as the element of the second sub-convolutional kernel, and so on, until it is split into 9 sub-convolutional kernels.

[0099] 704. Perform convolution multiplication operations between each of the multiple sub-convolution kernels and the input matrix to obtain multiple sub-result matrices.

[0100] After splitting the sub-convolution kernels, since the splitting is performed according to the stride m, it is only necessary to perform a convolution multiplication operation with a stride of 1 on the input matrix for each sub-convolution kernel, so as to obtain multiple sub-result matrices.

[0101] When the first convolutional kernel is split into multiple sub-convolutional kernels, the size of the sub-convolutional kernels will become smaller. At the same time, performing a convolutional multiplication operation with a stride of 1 with the input matrix will greatly reduce the computational complexity, ensuring that there is no redundant computation in the matrix calculation. In addition, the characteristics of matrix multiplication will integrate matrix addition, eliminating the need to accumulate elements at the same position, completely eliminating the computational task of the VCU, and improving the computing power of the AI ​​chip.

[0102] 705. Reorganize the elements in multiple sub-result matrices to obtain the output result.

[0103] After obtaining multiple sub-result matrices, simply rearrange the sub-result matrices to obtain the output dE / dX. Specifically, the size of the combination matrix can be determined based on the size of the multiple sub-result matrices. Then, all elements in the multiple sub-result matrices are rearranged in the combination matrix. It can be understood that after rearranging any two adjacent elements in the sub-result matrices, they are separated by a movement step m in the combination matrix. Finally, the final output result is obtained based on the rearranged combination matrix.

[0104] Optionally, the specific rearrangement steps are as follows:

[0105] 1) First, start taking elements from the top left corner of the combined matrix, and determine the top left corner position as the first position. Fill the top left corner element of the first sub-result matrix at the first position. It can be understood that the first sub-result matrix is ​​obtained by convolving the first sub-convolution kernel with the input matrix.

[0106] 2) Then, determine the next position of the first position in the horizontal / vertical direction with the step size m, and fill in the second element of the first sub-result matrix that is adjacent to the top left element in the horizontal / vertical direction at that position. Continue filling in the numbers in order until all elements of the first sub-result matrix are filled into the combined matrix.

[0107] 3) Then fill in the elements of the second sub-result matrix. It can be understood that the second sub-result matrix is ​​obtained by convolving the second sub-convolution kernel with the input matrix. Determine the position in the upper left direction where no element is filled as the second position, and fill in the upper left corner element of the second sub-result matrix in the second position.

[0108] 4) Then, determine the next position of the second position in the horizontal / vertical direction with the step size m, and fill in the second element of the second sub-result matrix that is adjacent to the top left element in the horizontal / vertical direction at that position. Continue filling in the numbers in order until all elements of the second sub-result matrix are filled into the combined matrix.

[0109] 5) Repeat the above steps until all elements of all sub-result matrices are filled into the combined matrix.

[0110] For example, such as Figure 11As shown, the size of the first sub-result matrix is ​​4*4, the size of the second sub-result matrix is ​​3*4, the size of the third sub-result matrix is ​​4*3, and the size of the fourth sub-result matrix is ​​3*3. Therefore, the size of the rearranged combined matrix is ​​7*7. Then, the first element from the top-left corner of the first sub-result matrix is ​​filled into the top-left corner of the combined matrix. Then, the next element from the first sub-result matrix is ​​filled in after a one-step shift, until all elements in the first sub-result matrix are filled. Next, the elements of the second sub-result matrix are filled, and finally, all elements in all four sub-result matrices are filled. It is understandable that in the filled combined matrix, the elements of the first sub-result matrix marked as 1 are separated by a one-step shift, the elements of the second sub-result matrix marked as 2 are also separated by a one-step shift, and the elements marked as 3 and 4 are the same.

[0111] After reorganizing the multiple sub-result matrices, it is necessary to consider the input features from the forward pass. Figure X The elements in the combination matrix are filtered according to the dimension, and then the final output dE / dX is obtained based on the filtered elements. It can be understood that the final dE / dX is related to the input features in the forward propagation. Figure X They have the same dimensions.

[0112] The selection of elements in the combination matrix is ​​related to the forward propagation calculation process. To preserve boundary information, when performing convolution operations on the input feature values ​​X, it is necessary to optimize the feature matrix. Figure X Performing padding operations, for example, when the padding method is "same", the convolutional kernel on the features... Figure X Slide upwards; when the remaining block is smaller than the convolution kernel size, it is necessary to adjust the features. Figure X Zero-padding is performed; therefore, during the reverse propagation process, the elements in the combination matrix are filtered, which requires examining the features during the forward propagation process. Figure X The zero-padding operation determines the position of the zero padding and deletes the corresponding element in the combined matrix according to the position, so that dE / dX and X have the same dimension and the element positions correspond.

[0113] like Figure 12The diagram shows a specific reverse convolution operation provided in this application. The second convolution kernel W corresponding to the forward pass has a 3*3 structure, containing the elements 1, 0, 2, -1, 1, 0, 0, 2, -1 from left to right. First, W is transposed along the sub-diagonal, resulting in the first convolution kernel W'. W' is then split into four sub-convolution kernels. Since the stride of the convolution kernel is 2, it is divided into four sub-convolution kernels. The first convolution kernel contains the elements -1, 2, 0, 1, the second convolution kernel contains the elements 0 and -1, the third convolution kernel contains the elements 2 and 0, and the fourth convolution kernel contains the element 1. Each sub-convolution kernel is then multiplied by the input matrix dE / dY to obtain four sub-result matrices. The elements of the sub-result matrices are then rearranged to obtain a combination matrix. Finally, the elements of the combination matrix are filtered according to the dimension of X in the forward pass, and the final result is obtained based on the filtered combination matrix.

[0114] Using the method provided in this embodiment, the backpropagation process of the convolutional neural network can efficiently obtain dE / dX without the col2img process and redundant computation. By splitting the transposed and rearranged convolutional kernel, and then using multiple sub-convolutional kernels to perform simple convolution calculations on the input matrix with a stride of 1, the final dE / dX result can be obtained by rearranging the data, which reduces computational overhead and improves the computing performance of AI chips.

[0115] exist Figures 1 to 12 Based on the corresponding embodiments, in order to better implement the above-described solutions of this application, related equipment for implementing the above solutions is also provided below. See details. Figure 13 , Figure 13 This is a schematic diagram of the structure of a convolutional neural network data processing device provided in an embodiment of this application. The convolutional neural network data processing device 1300 includes: an acquisition unit 1301, a processing unit 1302, a calculation unit 1303, and an arrangement unit 1304.

[0116] The acquisition unit 1301 is used to acquire an input matrix, wherein the elements of the input matrix are the partial derivatives of the loss function with respect to the output feature map obtained by the convolutional neural network in the forward direction;

[0117] The acquisition unit 1301 is further configured to acquire the first convolutional kernel of the convolutional neural network; wherein the size of the first convolutional kernel is n*n, and the stride of the first convolutional kernel is m;

[0118] Processing unit 1302 is used to split the first convolutional kernel according to the moving step m to obtain multiple sub-convolutional kernels;

[0119] The calculation unit 1303 is used to perform convolution multiplication operations on the input matrix by each of the plurality of sub-convolution kernels to obtain a plurality of sub-result matrices;

[0120] The arrangement unit 1304 is used to combine the elements in the plurality of sub-result matrices according to the moving step size m to obtain the output result; the output result is the input of the next layer network structure that is in the reverse adjacent layer of the first layer network structure.

[0121] In one optional implementation, the first convolutional kernel is the convolutional kernel corresponding to the reverse operation of the convolutional neural network, and the convolutional kernel corresponding to the forward operation of the convolutional neural network is the second convolutional kernel; wherein, the first convolutional kernel is obtained by transposing the elements of the second convolutional kernel along the subdiagonal.

[0122] In an optional implementation, the processing unit 1302 is specifically used to split the first convolutional kernel into m*m sub-convolutional kernels; wherein, in each of the m*m sub-convolutional kernels, two horizontally / vertically adjacent elements are separated by a movement step m in the horizontal / vertical direction of the first convolutional kernel.

[0123] In an optional implementation, the processing unit 1302 is specifically used for:

[0124] The element at the top left corner of the first convolutional kernel is determined as the first element; the next element is taken in both the horizontal and vertical directions with a stride of m; all the elements obtained are combined according to their positions in the first convolutional kernel to obtain the first sub-convolutional kernel; the element not taken in the top left direction is determined as the first element in the second sub-convolutional kernel; the next element is taken in both the horizontal and vertical directions with a stride of m; all the elements obtained are combined according to their positions in the first convolutional kernel to obtain the second sub-convolutional kernel; the above steps are repeated to obtain m*m sub-convolutional kernels.

[0125] In an optional implementation, the computing unit 1303 is specifically used to perform a convolution multiplication operation with a stride of 1 on each of the plurality of sub-convolution kernels on the input matrix to obtain a plurality of sub-result matrices.

[0126] In an optional implementation, the arrangement unit 1304 is specifically used to determine the size of the combined matrix based on the size of the plurality of sub-result matrices; rearrange all elements in the plurality of sub-result matrices in the combined matrix such that two elements that are horizontally / vertically adjacent in each sub-result matrix are separated by a movement step m in the horizontal / vertical direction of the combined matrix; and obtain the output result based on the combined matrix.

[0127] In an optional implementation, the arranging unit 1304 is specifically used for:

[0128] The top-left corner of the combined matrix is ​​defined as the first position, and the top-left element of the first sub-result matrix is ​​filled into the first position; the first sub-result matrix corresponds to the first sub-convolution kernel; the next position of the first position is determined in the horizontal / vertical direction of the combined matrix with a step size of m, and the element of the first sub-result matrix that is horizontally / vertically adjacent to the top-left element is filled into the next position of the first position, until all elements of the first sub-result matrix are filled into the combined matrix; the position where no element is filled in the top-left direction is defined as the second position, and the top-left element of the second sub-result matrix is ​​filled into the second position; the second sub-result matrix corresponds to the second sub-convolution kernel; the next position of the second position is determined in the horizontal / vertical direction of the combined matrix with a step size of m, and the element of the second sub-result matrix that is horizontally / vertically adjacent to the top-left element is filled into the next position of the second position, until all elements of the second sub-result matrix are filled into the combined matrix; the above steps are repeated until all elements of all sub-result matrices are filled into the combined matrix.

[0129] In an optional implementation, the arrangement unit 1304 is specifically used to filter the elements in the combination matrix according to the size n*n of the first convolution kernel; and to obtain the output result based on the elements of the filtered combination matrix.

[0130] The methods disclosed in the above embodiments of this application can be applied to the convolutional neural network data processing device 1300. It should be noted that the information interaction, execution process, and other beneficial effects between the modules / units in the convolutional neural network data processing device 1300 are different from those described in this application. Figures 1 to 12 The various method embodiments are based on the same concept, and the details can be found in the descriptions of the method embodiments shown above in this application, which will not be repeated here.

[0131] This application also provides an execution device, including a processor coupled to a memory. The memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, they correspond to the instructions in this application. Figures 1 to 12 The steps included in the corresponding method embodiments.

[0132] This application also provides a computer-readable storage medium storing a program that causes a computer to execute the aforementioned program. Figures 1 to 12 The steps performed by the execution device in the method described in the illustrated embodiment.

[0133] This application also provides a computer program product that enables a computer to perform the aforementioned operations. Figures 1 to 12 The steps performed by the execution device in the method described in the illustrated embodiment.

[0134] This application embodiment also provides a circuit system, the circuit system including a processing circuit, the processing circuit being configured to perform the aforementioned... Figures 1 to 12 The steps performed by the execution device in the method described in the illustrated embodiment.

[0135] The execution device provided in this application embodiment can specifically be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit, causing the chip within the execution device to perform the aforementioned operations. Figures 1 to 12 The convolutional neural network data processing method described in the illustrated embodiment. Optionally, the storage unit is a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0136] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of a program in the first aspect of the method.

[0137] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0138] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CLUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0139] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0140] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

Claims

1. A convolutional neural network data processing method, characterized in that, The method is applied to an artificial intelligence (AI) chip, and the method comprises: obtaining an input matrix, wherein elements of the input matrix are partial derivatives of a loss function with respect to output feature maps obtained by forward calculation of a convolutional neural network, and the convolutional neural network is applied to at least one of image classification and image recognition; obtaining a first convolution kernel of the convolutional neural network, wherein a size of the first convolution kernel is n*n, and a moving step of the first convolution kernel is m; splitting the first convolution kernel according to the moving step m to obtain a plurality of sub convolution kernels; performing convolution multiplication operation on the input matrix by each of the plurality of sub convolution kernels to obtain a plurality of sub result matrices; combining elements in the plurality of sub result matrices according to the moving step m to obtain an output result.

2. The method of claim 1, wherein, The first convolution kernel is a convolution kernel corresponding to backward calculation of the convolutional neural network, and a convolution kernel corresponding to forward calculation of the convolutional neural network is a second convolution kernel; wherein the first convolution kernel is obtained by transposing elements in the second convolution kernel along a sub-diagonal line.

3. The method according to any one of claims 1 to 2, characterized in that, The splitting the first convolution kernel according to the moving step m to obtain a plurality of sub convolution kernels comprises: splitting the first convolution kernel into m*m sub convolution kernels; wherein in each of the m*m sub convolution kernels, two elements adjacent in a horizontal / vertical direction are separated by one moving step m in the horizontal / vertical direction of the first convolution kernel.

4. The method of claim 3, wherein, The splitting the first convolution kernel according to the moving step m to obtain a plurality of sub convolution kernels comprises: determining an element in the upper left corner of the first convolution kernel as a first element; taking the next element in the horizontal and vertical directions with the moving step m; combining all the obtained elements according to their positions in the first convolution kernel to obtain a first sub convolution kernel; determining an element not taken in the upper left direction as a first element in a second sub convolution kernel; taking the next element in the horizontal and vertical directions with the moving step m; combining all the obtained elements according to their positions in the first convolution kernel to obtain a second sub convolution kernel; repeating the above steps to obtain m*m sub convolution kernels.

5. The method of claim 4, wherein, The performing convolution multiplication operation on the input matrix by each of the plurality of sub convolution kernels to obtain a plurality of sub result matrices comprises: performing convolution multiplication operation on the input matrix by each of the plurality of sub convolution kernels with a moving step of 1 to obtain a plurality of sub result matrices.

6. The method of claim 5, wherein, The combining elements in the plurality of sub result matrices according to the moving step m to obtain an output result comprises: determining a size of a combination matrix according to a size of the plurality of sub result matrices; rearranging all the elements in the plurality of sub result matrices in the combination matrix so that two elements adjacent in a horizontal / vertical direction in each sub result matrix are separated by one moving step m in the horizontal / vertical direction of the combination matrix; obtaining an output result according to the combination matrix.

7. The method of claim 6, wherein, The rearranging all the elements in the plurality of sub result matrices in the combination matrix comprises: determining a first position as a position of a top-left corner of the combination matrix, and filling an element of a top-left corner of a first sub-result matrix into the first position; the first sub-result matrix corresponds to the first sub-convolution kernel; determining a next position of the first position in a horizontal / vertical direction of the combination matrix by the m as a step length, and filling an element adjacent to the element of the top-left corner in the horizontal / vertical direction in the first sub-result matrix into the next position of the first position, until all elements of the first sub-result matrix are filled into the combination matrix; determining a second position as a position of an unfilled element in the top-left direction, and filling an element of a top-left corner of a second sub-result matrix into the second position; the second sub-result matrix corresponds to the second sub-convolution kernel; determining a next position of the second position in the horizontal / vertical direction of the combination matrix by the moving step length m as a step length, and filling an element adjacent to the element of the top-left corner in the horizontal / vertical direction in the second sub-result matrix into the next position of the second position, until all elements of the second sub-result matrix are filled into the combination matrix; repeating the above steps until all elements of all sub-result matrices are filled into the combination matrix.

8. The method according to any one of claims 6 to 7, characterized in that, combining elements in the plurality of sub-result matrices according to the moving step length m to obtain an output result, including: screening elements in the combination matrix according to the size n*n of the first convolution kernel; obtaining the output result according to the screened elements of the combination matrix.

9. A convolutional neural network data processing apparatus, comprising: The device is applied to an artificial intelligence (AI) chip, and the processing device includes: an obtaining unit, configured to obtain an input matrix, an element of the input matrix being a partial derivative of an output feature map obtained by a loss function on a forward operation of a convolutional neural network, the convolutional neural network being applied to at least one of the following: image classification, image recognition; the obtaining unit is further configured to obtain a first convolution kernel of the convolutional neural network; wherein the size of the first convolution kernel is n*n, and the moving step length of the first convolution kernel is m; a processing unit, configured to split the first convolution kernel according to the moving step length m to obtain a plurality of sub-convolution kernels; a calculating unit, configured to perform convolution multiplication operation on the input matrix by each of the plurality of sub-convolution kernels to obtain a plurality of sub-result matrices; an arranging unit, configured to combine elements in the plurality of sub-result matrices according to the moving step length m to obtain an output result.

10. The processing device of claim 9, wherein, The first convolution kernel is a convolution kernel corresponding to a backward operation of the convolutional neural network, and a convolution kernel corresponding to a forward operation of the convolutional neural network is a second convolution kernel; wherein the first convolution kernel is obtained by transposing elements in the second convolution kernel along a sub-diagonal line.

11. The processing device according to any one of claims 9 to 10, characterized in that, The processing unit is specifically configured to split the first convolution kernel into m*m sub-convolution kernels; wherein in each of the m*m sub-convolution kernels, two elements adjacent in a horizontal / vertical direction are apart from each other by one moving step length m in the horizontal / vertical direction of the first convolution kernel.

12. The processing device of claim 11, wherein, The processing unit is specifically configured to: determine an element of a top-left corner of the first convolution kernel as a first element; Take the next element in the horizontal and vertical direction with the moving step m as a step; Combine all the elements obtained according to the positions in the first convolution kernel to obtain a first sub-convolution kernel; Determine the element not taken in the upper left direction as the first element in the second sub-convolution kernel; Take the next element in the horizontal and vertical direction with the moving step m as a step; Combine all the elements obtained according to the positions in the first convolution kernel to obtain a first sub-convolution kernel; Repeat the above steps to obtain m*m sub-convolution kernels.

13. The processing device of claim 12, wherein, The calculation unit is specifically configured to perform convolution multiplication operation with a moving step of 1 on the input matrix by each of the plurality of sub-convolution kernels to obtain a plurality of sub-result matrices.

14. The processing device of claim 13, wherein, The arrangement unit is specifically configured to determine the size of a combination matrix according to the size of the plurality of sub-result matrices; and rearrange all elements in the plurality of sub-result matrices in the combination matrix, so that two elements adjacent in the horizontal / vertical direction in each sub-result matrix are separated by one moving step m in the horizontal / vertical direction of the combination matrix. Obtain an output result according to the combination matrix.

15. The processing device of claim 14, wherein, The arrangement unit is specifically configured to: Determine the position of the upper left corner of the combination matrix as a first position, and fill the upper left element of a first sub-result matrix in the first position; the first sub-result matrix corresponds to the first sub-convolution kernel; Determine the next position of the first position in the horizontal / vertical direction of the combination matrix with the moving step m, and fill the element adjacent to the upper left element in the horizontal / vertical direction in the first sub-result matrix in the next position of the first position, until all elements of the first sub-result matrix are filled in the combination matrix; Determine the position not filled in the upper left direction as a second position, and fill the upper left element of a second sub-result matrix in the second position; the second sub-result matrix corresponds to the second sub-convolution kernel; Determine the next position of the second position in the horizontal / vertical direction of the combination matrix with the moving step m, and fill the element adjacent to the upper left element in the horizontal / vertical direction in the second sub-result matrix in the next position of the second position, until all elements of the second sub-result matrix are filled in the combination matrix; Repeat the above steps until all elements of all sub-result matrices are filled in the combination matrix.

16. The processing device according to any one of claims 14 to 15, characterized in that, The arrangement unit is specifically configured to filter elements in the combination matrix according to the size n*n of the first convolution kernel; and obtain the output result according to the elements of the filtered combination matrix.

17. A computer-readable storage medium, characterized in that, The computer program product comprises a program which, when executed on a computer, causes the computer to perform the method of any one of claims 1 to 8.

18. An execution device, characterized by The processor and the memory are coupled, and the memory stores program instructions, when the program instructions stored in the memory are executed by the processor, the method of any one of claims 1 to 8 is realized.

Images