Deep learning processing device, method, equipment and storage medium

Abstract

The embodiments of the present disclosure propose deep learning processing devices, methods, equipment and storage media, and relate to the field of artificial intelligence. A deep learning processing device includes at least one matrix multiplication and addition module, which is configured to perform matrix multiplication and addition operations of a convolution kernel parameter value matrix of a convolution layer in a convolutional neural network and a first error gradient value matrix to obtain multiple intermediate matrices; a storage device, which is used to store multiple intermediate matrices without reshaping the elements in the multiple intermediate matrices; and multiple matrix accumulation modules, which are configured to read multiple intermediate matrices from the storage device and perform matrix accumulation operations based on multiple intermediate matrices in parallel according to the convolution scheme of the convolution layer to obtain a second error gradient value matrix for the convolution layer. Such a deep learning processing device can significantly improve the computational efficiency of the convolution reverse operation, and can be improved in terms of computing speed and energy consumption.

Description

Technical Field

[0001] Embodiments of the present disclosure relate generally to the field of data processing, and more specifically, to the field of artificial intelligence. Background Art

[0002] Deep learning is a research direction of artificial neural networks. In recent years, with the continuous improvement of hardware and software in the field of artificial intelligence, deep learning technology has developed rapidly. Deep learning technology can be applied in various fields, such as computer vision, natural language processing, audio analysis, etc. Convolutional neural network (CNN) is an influential network model in deep learning technology, especially suitable for applications involving image and text data. The calculations involved in convolutional neural networks mainly include convolution calculation, full connection (FC) calculation, pooling calculation, vector calculation, activation calculation, etc., among which the most important calculation is convolution operation. In the training process of CNN, in order to achieve model optimization, in addition to using training data to perform forward calculations, the model parameter optimization is also achieved by using the backward propagation method.

[0003] The training process of CNN involves a large number of convolution operations and inverse convolution operations of the convolution layer. In some CNN architectures, convolution operations and inverse convolution operations may occupy most of the computing resources and time of the entire architecture operation. Deep learning processors can be used to implement deep learning calculations and support the training of deep learning networks. It is expected that deep learning processors can process convolution operations and / or inverse convolution operations more quickly and efficiently, which will help accelerate the training of the entire deep learning network, especially CNN. Summary of the invention

[0004] According to an example embodiment of the present disclosure, a scheme for performing a deep learning process is provided.

[0005] In a first aspect of the present disclosure, a deep learning processing device is provided. The deep learning processing device includes at least one matrix multiplication and addition module, configured to perform matrix multiplication and addition operations of a convolution kernel parameter value matrix of a convolution layer in a convolutional neural network and a first error gradient value matrix to obtain multiple intermediate matrices; a storage device, used to store multiple intermediate matrices without reshaping the elements in the multiple intermediate matrices; and multiple matrix accumulation modules, configured to read multiple intermediate matrices from the storage device and perform matrix accumulation operations based on the multiple intermediate matrices in parallel according to the convolution scheme of the convolution layer to obtain a second error gradient value matrix for the convolution layer.

[0006] In a second aspect of the present disclosure, a method for performing deep learning processing is provided. The method includes causing at least one matrix multiplication and addition module of a deep learning processing device to perform a matrix multiplication and addition operation of a convolution kernel parameter value matrix of a convolution layer in a convolutional neural network and a first error gradient value matrix to obtain a plurality of intermediate matrices; storing the plurality of intermediate matrices to a storage device without reshaping the elements in the plurality of intermediate matrices; reading the plurality of intermediate matrices from the storage device; and causing a plurality of matrix accumulation modules of the deep learning processing device to perform a matrix accumulation operation based on the plurality of intermediate matrices in a parallel manner according to a convolution scheme of the convolution layer to obtain a second error gradient value matrix for the convolution layer.

[0007] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes a processing device, including at least one deep learning processing device according to the first aspect; and at least one general processing device, coupled to the at least one deep learning processing device and configured to provide instructions for performing training of a convolutional neural network (CNN) to the at least one deep learning processing device.

[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and when the program is executed by a processor, the method according to the second aspect of the present disclosure is implemented.

[0009] In a fifth aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the method according to the second aspect of the present disclosure when executed by a processor.

[0010] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0012] Figure 1 A block diagram schematically illustrates the structure of an example convolutional neural network (CNN);

[0013] Figure 2 schematically illustrates a block diagram of an example processing architecture according to some embodiments;

[0014] Figure 3A A block diagram illustrating an example structure of a deep learning processing device according to some example embodiments of the present disclosure;

[0015] Figure 3B A block diagram illustrating an example structure of a deep learning processing device according to some other example embodiments of the present disclosure;

[0016] Figure 3C A block diagram illustrating an example structure of a deep learning processing device according to some further example embodiments of the present disclosure;

[0017] Figure 4A A schematic diagram showing the convolution operation of the convolutional layer of a CNN is shown;

[0018] Figure 4B A schematic diagram showing the reverse operation of the convolutional layer of a CNN;

[0019] Figure 5 A schematic diagram illustrating a matrix accumulation process according to some example embodiments of the present disclosure;

[0020] Figure 6 A block diagram illustrating an example structure of a matrix calculation module according to some other example embodiments of the present disclosure;

[0021] Figure 7 A block diagram illustrating an example structure of a matrix accumulation module according to some other example embodiments of the present disclosure;

[0022] Figure 8 A flowchart schematically illustrates a method for executing a processing task according to an example embodiment of the present disclosure; and

[0023] Fig. 9 A block diagram of a computing device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0024] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for illustrative purposes and are not intended to limit the scope of protection of the present disclosure.

[0025] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0026] As mentioned above, very complex operations need to be processed during the training and reasoning of convolutional neural networks (CNNs), especially the convolution operations of the convolutional layers and the reverse convolution operations. If a general-purpose processor is used to perform the training or reasoning of CNNs, the processing efficiency is very low. Some current solutions are to implement the processing of CNNs, especially the training of CNNs, through a graphics processing unit (GPU). GPUs use single instruction multiple threads (SIMT) technology to schedule and implement operations through a large number of threads, which can increase the calculation speed. However, GPUs require a large number of register stacks, complex thread scheduling mechanisms, and cache management, resulting in high power consumption and poor performance during calculations. Therefore, it is expected to provide a more efficient processing device for deep learning processing, especially related processing of CNNs.

[0027] Before introducing the processing device for CNN related processing, CNN is briefly introduced. CNN is a deep learning model. The values ​​of the parameter set used for processing in the deep learning model are determined through a training process. The machine learning model uses the trained parameter set to map the received input to the corresponding output. Therefore, the training process of the machine learning model can be considered as learning the mapping or association relationship between input and output from the training data.

[0028] CNN can usually include input layer, convolution layer, pooling layer, activation layer, fully connected layer and output layer. Depending on the specific processing task requirements and configuration, the number and connection relationship of convolution layers, pooling layers, activation layers and fully connected layers in CNN can vary. Figure 1 An example of the structure of CNN 100 is shown, which shows input layer 110, convolution layer 120, activation layer 130, pooling layer 140, ... and fully connected layer 150 and output layer 160. The input to be processed by the model is provided to CNN 100 as input layer 110 for processing. In the forward processing 101, the input data is processed by multiple intermediate layers, and finally the output result of output layer 160 is obtained.

[0029] During the training process of CNN 100, the training data needs to be processed in the forward direction 101 and also needs to be processed in the reverse direction 102. In the reverse process 102, the error between the output obtained by processing the input training data under the current value of the parameter set of CNN 100 and the ideal output is usually calculated, and then the error is propagated in the opposite direction (i.e., from the output layer 160 to the input layer 110). During the back propagation process, the current value of the parameters of each layer in CNN 100 can be adjusted by relying on the gradient descent algorithm. According to multiple rounds of training, the error between the output of CNN 100 and the ideal output will become smaller and smaller until the model converges. The training process is completed.

[0030] It should be understood that Figure 1 The structure of the CNN is only an example. In practical applications, the CNN may have any other network configuration as needed, may have more network layers, other types of network layers, and different connection relationships of various types of network layers. The embodiments of the present disclosure are not limited in this respect.

[0031] In some embodiments, a dedicated deep learning processor may be utilized to handle CNN training-related operations. Figure 2 A schematic diagram of an example processing architecture 200 according to some embodiments is shown. The example processing architecture 200 is a heterogeneous architecture including a general purpose processing device 210 and a deep learning processing device 220 coupled thereto.

[0032] The general processing device 210 may include, for example, one or more general purpose processor (CPU) cores, one or more digital signal processor (DSP) cores, etc. The general processing device 210 may be, for example, a general purpose scalar processor. The general processing device 210 may execute general computer instructions, such as reduced instruction set computer (RISC) type instructions, and may also parse and execute customized instructions related to deep learning processing. The general processing device 210 may provide instructions related to deep learning processing to the deep learning processing device 220 for implementing related processing of the deep learning model.

[0033] The deep learning processing device 220 (sometimes also referred to as a deep learning processor or a deep learning processing device) may be, for example, a dedicated deep learning coprocessor, including software components and hardware circuits for implementing deep learning calculations. The deep learning processing device 220 may be implemented, for example, by a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like. The deep learning processing device 220 includes a plurality of modules, and the operations of the plurality of modules are scheduled by the relevant instructions defined for deep learning, and data can also be exchanged between the plurality of modules. The modules in the deep learning processing device 220 may be configured according to the deep learning processing tasks to be implemented. In some implementations, the deep learning processing device 220 may be configured to perform the training tasks of CNN. In such an implementation, the general processing device 210 provides the deep learning processing device 220 with corresponding instructions for performing the training of CNN.

[0034] As mentioned above, the training process of CNN involves a large number of convolution operations and inverse convolution operations of the convolution layer, which consumes a lot of computing resources and time. Therefore, improvements in convolution operations and inverse convolution operations will significantly accelerate the training of CNN.

[0035] According to an example embodiment of the present disclosure, a deep learning processing device is proposed. The deep learning processing device is capable of performing convolution direction operations of a convolution layer. The deep learning processing device includes one or more matrix multiplication and addition modules, which are configured to perform matrix multiplication and addition operations of a convolution kernel parameter value matrix of a convolution layer in a CNN and a first error gradient value matrix to obtain multiple intermediate matrices. Multiple intermediate matrices are stored in a storage device without reshaping. The deep learning processing device also includes multiple matrix accumulation modules, which are configured to read multiple intermediate matrices from a storage device and perform matrix accumulation operations based on multiple intermediate matrices in a parallel manner according to the convolution scheme of the convolution layer to obtain a second error gradient value matrix for the convolution layer. In this scheme, during the training process of the CNN, matrix multiplication and matrix accumulation operations are implemented by specific modules, and multiple matrix accumulation modules can perform operations in parallel, which can significantly improve the computational efficiency of the reverse operation of the convolution, and can be improved in terms of both computational speed and energy consumption.

[0036] Figure 3A An example structure of a deep learning processing device according to some example embodiments of the present disclosure is shown. The deep learning processing device may be a deep learning processing device 220, which includes a matrix calculation module 301 for implementing matrix-related calculations of a convolutional layer of a CNN. The matrix calculation module 301 includes one or more matrix multiplication and addition modules 310-1, ... 310-N, where N may be an integer greater than or equal to 1. For ease of discussion, the matrix multiplication and addition modules 310-1, ... 310-N may be collectively or individually referred to as matrix multiplication and addition modules 310. One or more matrix multiplication and addition modules 310 are configured to implement matrix multiplication and addition operations of a convolution kernel parameter value matrix of a convolutional layer of a CNN and a first error gradient value matrix. The matrix multiplication and addition operations of the convolutional layer will produce multiple intermediate matrices.

[0037] The matrix calculation module 301 also includes a plurality of matrix accumulation modules 320-1, ... 320-M, where M may be an integer greater than or equal to 2. For ease of discussion, the matrix accumulation modules 320-1, ... 320-M may be collectively or individually referred to as matrix accumulation modules 320. These matrix accumulation modules 320 are configured to perform matrix accumulation operations based on a plurality of intermediate matrices in a parallel manner according to the convolution scheme of the convolution layer to obtain an error gradient value matrix (also referred to as a "second error gradient value matrix") for the current convolution layer.

[0038] In operation, multiple intermediate matrices generated by the matrix multiplication and addition operation are stored in the storage device 330 of the deep learning processing device 210. The storage device 330 may be an on-chip storage device, for example, an on-chip random access memory (RAM), such as a static random access memory (SRAM) or other types of memory. Multiple intermediate matrices may not be generated at the same time, so each time the matrix multiplication and addition module 310 generates a corresponding intermediate matrix, the intermediate matrix is ​​stored in the corresponding storage location of the storage device 330. In some embodiments, if there are multiple matrix multiplication and addition modules 310, the multiple matrix multiplication and addition modules 310 can perform matrix multiplication and addition operations of the convolution kernel parameter value matrix and the first error gradient value matrix in parallel.

[0039] In some embodiments, the matrix calculation module 301 may further include an instruction parsing module 340 configured to receive instructions for operations of various modules in the matrix calculation module 301. Such an instruction set may be, for example, a custom deep learning instruction set, including an instruction set for convolution direction operations of a convolution layer of a CNN. The instruction parsing module 340 may, for example, obtain the instruction set from a general processing device 210 coupled to the deep learning processing device 220. The instruction parsing module 340 may parse the instruction set into instructions executable by the deep learning processing device 220.

[0040] In some embodiments, the matrix calculation module 301 may also be a module related to instruction parsing and control signal generation. Figure 3B As shown, the matrix calculation module 301 may further include a control signal generation module 350, which is configured to generate control signals for at least one matrix multiplication and addition module 310 and multiple matrix accumulation modules 320 based on the parsed instruction set to control the corresponding operations of the at least one matrix multiplication and addition module 310 and multiple matrix accumulation modules 320. Figure 3A In the example of , the control signal generation module 350 includes a read-write control module 352, which is configured to read and write control signals according to the parsed instruction set, and the read-write control signals are used to control the reading and writing of data in the storage device 330 for at least one matrix multiplication and addition module 310 and multiple matrix accumulation modules 320. The control signal generation module 350 may also include a loop control module 354. When the matrix multiplication and addition module 310 is to loop to perform multiple matrix multiplication and addition operations and / or the matrix accumulation module 320 is to loop to perform multiple matrix accumulation operations, the loop control module 354 is configured to generate a loop control signal according to the parsed instruction set, which is used to control the loop of the matrix multiplication and addition operations of each matrix multiplication and addition module 310, and / or control the loop of the matrix accumulation operations of each matrix accumulation module 320.

[0041] References Figure 3A and Figure 3BSome example embodiments of the matrix calculation module 301 in the deep learning processing device 220 are generally introduced. In some embodiments, the deep learning processing device 220 may further include other modules for implementing the CNN training task together with the matrix calculation module 301. Figure 3C FIG. 2 shows an example structure of a deep learning processing device 220 according to some embodiments of the present disclosure. Figure 3C As shown, in addition to the matrix calculation module 301, the deep learning processing device 220 also includes a data reading and writing module 302, a data conversion module 303, a vector calculation module 304, and possibly a pooling module 305 and a transposition module 306. In addition to the matrix calculation module 301, the vector calculation module 304 and possibly a pooling module 305 and a transposition module 306 can access the storage device 330 to read the data to be processed therefrom and write the processed data to the storage device 330. Therefore, the storage device 330 is sometimes referred to as a shared storage device 330.

[0042] The data reading and writing module 302 is configured to read the data required for the training process of the CNN from the storage device / equipment outside the deep learning processing device 220 (also referred to as an off-chip storage device / equipment), and store the data in the storage device 330. The data conversion module 303 is configured to read the data to be converted from the storage device 330 and perform format conversion on the data, such as reshaping of each element in the data (for example, converting the data from a three-dimensional or higher-dimensional matrix to a two-dimensional matrix, or converting a matrix to a vector). The converted data is then stored in the storage device 330.

[0043] The matrix calculation module 301 is configured to perform the matrix calculation operations involved in the training process of the CNN, and the vector calculation module 304 is configured to perform the vector calculation operations involved in the training process of the CNN. The pooling module 305 is configured to perform the relevant operations of the pooling layer of the CNN, and the transposition module 306 is configured to perform the matrix transposition operations involved in the training process of the CNN. In some embodiments, the relevant operations and transposition operations of the pooling layer can also be converted into corresponding matrix calculation operations and vector calculation operations, thereby being implemented by the matrix calculation module 301 and the vector calculation module 304. The storage device 330 can be, for example, an on-chip random access memory (RAM), such as a static random access memory (SRAM) or other types of memory. The matrix calculation module 301, the vector calculation module 304, and the possible pooling module 305 and the transposition module 306 can all access the storage device 330 to read the data to be processed therefrom and write the processed data to the storage device 330. Therefore, the storage device 330 is sometimes referred to as a shared storage device 330.

[0044] In the following, in order to better understand the specific operations of the matrix multiplication and addition module 310 and the matrix accumulation module 320 in the matrix calculation module 301 in the convolution reverse operation of the convolution layer of the CNN, reference will be made to Figure 4A and Figure 4B Let's briefly introduce the convolution operation and convolution reverse operation of the convolution layer.

[0045] Figure 4A The convolution operation of the convolution layer is shown. The convolution operation occurs in the forward calculation of the CNN. For a convolution layer in a CNN, the input to be processed comes from the output of the previous layer or from the input layer of the CNN. The input of the convolution layer usually includes one or more feature maps 410, each of which can be represented by a two-dimensional matrix. The number of input feature maps 410 is the number of input channels (assuming c channels).

[0046] The convolution layer includes one or more convolution kernels 420 for implementing the convolution operation. The number of convolution kernels 420 can be arbitrarily configured in the CNN (assuming the number of convolution kernels is "k"). The size of each convolution kernel 420 is assumed to be c*f h *f w , where c is the number of channels and f h *f w represents the height and width of the convolution kernel. That is, each convolution kernel 420 can be represented as c*f h *f w The convolution kernel parameter value matrix of the convolution kernel. In the forward processing of the training process, the convolution kernel parameter value matrix is ​​the value determined at the current stage of training. When performing the convolution operation, each convolution kernel 420 moves on the input feature map 410 of the convolution layer according to the scheme of the convolution operation. For example, it can move from left to right and from top to bottom on the feature map at a certain step, and perform convolution operations on the obtained elements to finally obtain the output feature map of the convolution layer. The convolution operation can be expressed as:

[0047] [O h *O w , c*f h *f w ]*[c*f h *f w , k]=[O h *O w , k] Formula (1)

[0048] Among them, h Represents the height of the output of the convolutional layer, O w represents the width of the output of the convolutional layer, and k represents the number of convolution kernels. According to formula (1), the size of the output feature map of the convolutional layer is k*O h *O w , can be represented as k two-dimensional matrices Oh *O w .

[0049] In the convolution operation, each time c*f is extracted from the input feature map 410 h *f w The input sub-matrix (also called input window) includes c*f h *f w input elements, used to be the convolution kernel parameter value matrix (size c*f) of each convolution kernel 420 in the k convolution kernels. h *f w ) are multiplied. c*f h *f w The input submatrix and c*f h *f w The multiplication of the convolution kernel parameter value matrix can sometimes be expressed as the input elements of c f h *f w The two-dimensional matrix of the convolution kernel parameter value matrix c f h *f w ) (Of course, the two three-dimensional sub-matrices can be converted into two-dimensional matrices of other sizes to perform the matrix multiplication).

[0050] According to the scheme of the convolution operation, it is assumed that the convolution kernel can extract O on the input feature map 410 h *O w A window is used to perform the convolution operation. Figure 4A FIG. 4 shows a feature map of size c*f extracted from one of the c input feature maps 410. h *f w The input window 412 and a convolution kernel 420 have a size of c*f h *f w The convolution operation 430 is performed on the convolution kernel parameter value matrix to obtain an output element 440 in the output feature map of the convolution layer. If there are multiple convolution kernels 420, each convolution kernel performs similar processing. The convolution operation 430 is a matrix multiplication and addition operation, which can be decomposed into multiplication and addition operations of matrix elements. In the matrix calculation module 301, the convolution operation can be performed by the matrix multiplication and addition module 310. In some embodiments, in the forward convolution, the convolution operation of the convolution layer shown in formula (1) can be divided into corresponding matrix multiplication and addition operations, which are implemented by the matrix multiplication and addition module 310. For example, the O in the input feature map 410 can be h *O w The matrix multiplication of the input sub-matrices and the convolution kernel parameter value matrix of the convolution kernel is decomposed into O h Each time, the output feature map of the convolutional layer is calculated to obtain 1*O w output elements.

[0051] The above introduces the forward convolution operation of the convolution layer. The reverse convolution operation of the convolution layer is the reverse of the above convolution operation. Figure 4B The convolution reverse operation of the convolution layer is shown. In the reverse process, for a specific convolution layer in CNN, the next layer of the convolution layer (the next layer from the forward direction of CNN, such as Figure 1 The size of the first error gradient value matrix is ​​the same as the size of the output feature map of the convolution layer, that is, k*O. h *O w The convolution reverse operation performs the convolution reverse operation of the first error gradient value matrix and the convolution kernel parameter value matrix of the convolution layer (which can also be decomposed into multiplication and addition operations of matrix elements) to obtain multiple intermediate matrices, and then sequentially superimposes the multiple intermediate matrices to the corresponding positions of the input feature map of the convolution layer to obtain the error gradient value matrix for the current convolution layer (referred to as the "second error gradient value matrix" in this article). The convolution reverse operation of the convolution layer can be expressed as:

[0052] [O h *O w , k]*[k, c*f h *f w ]=[O h *O w , c*f h *f w ] Formula (2)

[0053] Among them, h Represents the height of the first error gradient value matrix of the next layer, O w represents the width of the error gradient value matrix, and k represents the number of channels of the error gradient value matrix (i.e., the two-dimensional matrix O h *O w According to formula (2), each convolution kernel c*f h *f w After multiplying with the corresponding elements in the first error gradient value matrix, the error gradient products of the k channels need to be accumulated together.

[0054] In the convolution reverse operation, for a convolution kernel 420, an error gradient value 450 is extracted from the first error gradient value matrix each time for comparison with c*f h *f wThe convolution kernel parameter value matrix of k convolution kernels 420 is used to perform the convolution reverse operation 460. For k convolution kernels 420, in the convolution reverse operation 460, the error gradients of k channels in the first error gradient value matrix and the convolution kernel parameter value matrix are multiplied and added together to obtain an intermediate matrix 462. The convolution reverse operation 460 can also be implemented by the matrix multiplication and addition module 310. The convolution reverse operation 460 can also be considered as a matrix multiplication and addition operation, which can be decomposed into multiplication and addition operations of matrix elements.

[0055] In some embodiments, if the matrix calculation module 301 includes a plurality of matrix multiplication and addition modules 310, when performing the matrix multiplication and addition operation of the convolution kernel parameter value matrix and the first error gradient value matrix, the plurality of matrix multiplication and addition modules 310 may perform the matrix multiplication and addition operation in parallel. h *O w , k]*[k, c*f h *f w ] can be decomposed into matrix multiplication and addition calculations of two-dimensional matrices of any size.

[0056] Assume that after matrix multiplication and addition operations, multiple intermediate matrices 462 are obtained, including O h *O w The size is c*f h *f w intermediate matrices. These intermediate matrices are not the final results of the reverse convolution operation and need to be accumulated according to the convolution scheme of the convolution layer. In the exemplary embodiment of the present disclosure, multiple matrix accumulation modules 320 are used to implement the matrix accumulation operation of multiple intermediate matrices in parallel. The convolution scheme of the convolution layer depends specifically on the way in which the convolution kernel extracts the input window in the input feature map of the convolution layer during the convolution operation, including the convolution kernel movement mode (e.g., from left to right, from top to bottom) and the movement stride (e.g., the window moves one element or other predetermined number of elements each time).

[0057] Figure 5 The accumulation method of multiple intermediate matrices is shown. Figure 5 In the example of , for the convenience of illustration, it is assumed that the number of channels c=1. Figure 5 It shows that multiple intermediate matrices (size f h *f w ) are superimposed to obtain the second error gradient value matrix. If the number of channels is greater than 1, there are O h *O w A two-dimensional matrix f h *f w can be superimposed accordingly to obtain a channel matrix. The c channel matrices finally obtained are used as the second error gradient value matrix.

[0058] like Figure 5As shown, if the first intermediate matrix 510 is calculated by the matrix multiplication and addition module 310, it will be accumulated to the first position (the position of the upper left corner) of the second error gradient value matrix 502 with the initial value. Assuming that the intermediate matrix 510 is a matrix including 3*3 elements, it is accumulated to the sub-matrix of 3*3 elements at the upper left corner of the initial two-dimensional matrix 502. During the matrix accumulation operation process of the matrix accumulation module 320, the second error gradient value matrix initially has all zero values ​​and is stored in the target storage position of the storage device 330. The matrix accumulation module 320 is configured to read the corresponding initial values ​​to be accumulated in the second error gradient value matrix 502 from the target storage position of the storage device 330, and accumulate each element in the intermediate matrix 510 with the corresponding initial value of the second error gradient value matrix 502. The accumulation result is written back to the target storage position of the storage device 330. The storage position of each element of the second error gradient value matrix 502 in the storage device 330 remains unchanged.

[0059] If the second intermediate matrix 520 is calculated by the matrix multiplication and addition module 310, it will be accumulated by the matrix accumulation module 320 to the second position of the second error gradient value matrix, which is offset to the right by one element relative to the first position (assuming that the stride of the convolution kernel 420 is one element). Some elements of the intermediate matrix 520 continue to be accumulated with the elements of the second error gradient value matrix 502 that have been accumulated with some elements of the intermediate matrix 510, and some elements are accumulated with the initial value (i.e., zero) of the second error gradient value matrix. The matrix accumulation module 320 can read out the elements to be accumulated with the second intermediate matrix 520 from the storage device 330.

[0060] If the third intermediate matrix 530 is calculated by the matrix multiplication and addition module 310, it will also be accumulated to the corresponding sub-matrix of the second error gradient value matrix 502 in a similar manner, and each element in the intermediate matrix 530 is accumulated with the accumulated value or the initial value at the corresponding position. h *O w The size is f h *f w After the intermediate matrices are superimposed, the superposition result forms the final second error gradient value matrix.

[0061] In an embodiment of the present disclosure, as mentioned above, after the matrix multiplication and addition module 310 calculates the intermediate matrix, the intermediate matrix can be stored in the storage device 330 without reshaping the elements in the intermediate matrix, that is, the intermediate matrix can still be stored according to the element ordering method generated by the matrix multiplication and addition module 310 without being reshaped or divided into other representations.

[0062] In some embodiments, each matrix accumulation module 320 can perform an accumulation operation of an intermediate matrix in each matrix accumulation operation, and multiple matrix accumulation modules 320 can perform accumulation operations in parallel. Since the second error gradient value matrix is ​​stored in a specific storage location of the storage device 330, and the intermediate matrix calculated by the matrix multiplication and addition module 310 does not need to be reshaped when stored, the matrix accumulation module 320 can quickly accumulate the intermediate matrix with the current value of the corresponding matrix element of the second error gradient value matrix stored in the storage device 330 after each intermediate matrix is ​​determined, without the need to accumulate each intermediate matrix in series in sequence (the order here refers to the convolution scheme of the convolution kernel, that is, the movement method and stride of the convolution kernel on the input feature map). For example, in Figure 5 In the example of , if the second intermediate matrix 520 is calculated first, the intermediate matrix 520 is directly superimposed on the elements with initial values ​​at corresponding positions of the second error gradient value matrix 502, even if the first intermediate matrix 510 has not been accumulated yet.

[0063] In some embodiments, the loop control module 354 can be configured to control the matrix accumulation operation of the matrix accumulation module 320 to avoid multiple matrix accumulation modules 320 from simultaneously accumulating the same element position in the second error gradient value matrix. Such parallel matrix accumulation can further improve the calculation speed of the convolution reverse operation, and is particularly suitable when there are multiple matrix multiplication and addition modules 310 and these matrix multiplication and addition modules 310 perform matrix multiplication and addition operations in parallel, because in this case some matrix multiplication and addition modules 310 may output intermediate matrices faster.

[0064] In some embodiments, during the parallel matrix accumulation process, if the input of the convolutional layer has multiple channels, this means that the second error gradient value matrix includes multiple channel matrices for the multiple channels (each channel matrix is ​​a two-dimensional matrix, for example Figure 5 A two-dimensional matrix 502 is provided, and the plurality of matrix accumulation modules 320 may be configured to perform matrix accumulation operations in parallel according to the plurality of channels. The plurality of matrix accumulation modules 320 may implement a plurality of parallel paths corresponding to the plurality of channels, and one or more matrix accumulation modules 320 may be present in each parallel path to perform matrix accumulation operations. The matrix accumulation module 320 on each parallel path is configured to perform accumulation of the intermediate matrices associated with one of the plurality of channels in the plurality of intermediate matrices, thereby obtaining a channel matrix for the channel.

[0065] In some embodiments, the parallel matrix accumulation operation can be implemented by intermediate matrix, rather than by channel division. Each matrix accumulation module 320 is configured to accumulate one of the intermediate matrices into an intermediate result for one of the channel matrices in each accumulation.

[0066] In some embodiments, the matrix calculation module 301 may also include a cache area (sometimes referred to herein as a "second cache area"). In the matrix multiplication and addition operation process and the matrix accumulation operation process of the two matrices, a large number of intermediate operation results will be generated, and these intermediate operation results will be called again in the later operation. Therefore, setting a cache area in the module can effectively reduce the data transfer between the matrix calculation module 301 and the storage device 330, further improve the matrix multiplication and addition operation speed, and reduce power consumption. Figure 6 Such an embodiment is shown. Figure 6 As shown, the matrix calculation module 301 also includes a cache area 610, which is configured to cache intermediate operation results (sometimes referred to as "first intermediate operation results") generated by at least one matrix multiplication and addition module 310 during the matrix multiplication and addition operation process, and / or intermediate operation results (sometimes referred to as "second intermediate operation results") generated by multiple matrix accumulation modules 320 during the matrix accumulation operation process.

[0067] In the matrix calculation process of the matrix multiplication and addition module 310 and the matrix accumulation module 320, the vector calculation function of the vector calculation module 304 of the deep learning processing device 220 can also be used, which can avoid the increase in complexity caused by the involvement of corresponding functions in the matrix calculation module 310. In some embodiments, the vector calculation module 304 can be configured to help the matrix calculation module 310 perform zeroing and / or data reading and writing operations on the buffer area 610 at the vector level, such as Figure 6 shown.

[0068] In some embodiments, the vector calculation module 304 may be configured to zero the storage area in the buffer 610 for storing the intermediate operation results of the matrix multiplication and addition module 310 and / or the intermediate operation results of the matrix accumulation module 320 at the vector level. That is, the vector calculation module 304 may zero the storage area corresponding to the intermediate operation results by row or by column. Alternatively or additionally, the vector calculation module 304 may also be configured to selectively write the final operation result of the matrix accumulation operation performed by each matrix accumulation module 320 to the storage device 304 at the vector level as at least a part of the second error gradient value matrix. This is to solve the situation where padding elements may be added during the convolution process. In the reverse operation of the convolution, the padding elements added in the forward direction need to be deleted and not used as elements in the final error gradient value matrix. The vector calculation module 304 can better filter the padding elements of the matrix row by row or column by column at the vector level.

[0069] The processing of the vector calculation module 304 can be controlled by a control signal, and such a control signal can be determined by parsing a custom deep learning related instruction set. It should be understood that in other embodiments, the function of the vector calculation module 304 may not be used, but the corresponding function may be configured in the matrix calculation module 301 to realize the zeroing of the buffer area and the selective writing of data.

[0070] In some embodiments, if the matrix accumulation module 320 needs to read and write the storage device 330 when performing the matrix accumulation operation, a "data hazard" situation may occur due to a certain delay between the matrix accumulation operation and the data reading and writing. When performing data accumulation, the matrix accumulation module 320 may need to sequentially accumulate multiple elements, and the accumulation results are stored in the same storage location of the storage device 330. For example, for an element in the second error gradient value matrix finally obtained, it may be the result of the accumulation of elements in two or more intermediate matrices. The "data hazard" situation refers to the situation where, when performing the accumulation of the current two elements, it is necessary to wait for the accumulation operation of the previous two elements to be completed, thereby causing the data accumulation pipeline to stop.

[0071] In some embodiments, it is proposed to utilize a bypass mechanism in the matrix accumulation module 320 to solve the “data hazard” problem. Figure 7 FIG. 3 shows an example structure of the matrix accumulation module 320 according to some embodiments of the present disclosure. Figure 7 As shown, the matrix accumulation module 320 includes an addition unit 710, which is configured to perform the accumulation of two elements in the matrix to be superimposed to obtain an accumulated element. The addition unit 710 can be, for example, a floating point addition unit, or an addition unit of any other format. The matrix accumulation module 320 also includes a data write unit 720, which is configured to write the accumulated element calculated by the addition unit 710 into a corresponding storage location of the storage device 330. The matrix accumulation module 320 also includes a data read unit 740, which is configured to read the accumulated element from the corresponding storage location of the storage device 330 for continued accumulation with the next element.

[0072] In the matrix accumulation operation, the addition unit 710 may need to perform the accumulation of more than two elements, and the accumulation result of each two elements will continue to be accumulated with the next element until the accumulation of all elements is completed. The matrix to be accumulated may be the intermediate matrix and the sub-matrix in the second error gradient value matrix to which the intermediate matrix is ​​accumulated, and its matrix elements are usually calculated. However, due to the data reading and writing of the data writing unit 720 to the storage device 330 and the operation of the addition unit 710, there is a certain delay. In some cases, the data writing unit 720 is writing the accumulated element after the first element and the second element calculated by the addition unit 710 into the storage device 330, and the next third element to be accumulated may be ready. In the conventional scheme, it is also necessary to wait for the data writing unit to continue to complete the writing of the accumulated element, and then activate the data reading unit to read the accumulated element from the corresponding storage position and provide it to the addition unit for accumulation, which leads to the "data hazard" problem and introduces a large delay.

[0073] exist Figure 7 In an embodiment of the present invention, the matrix accumulation module 320 includes a bypass module 730, which is configured to cache the accumulated elements calculated by the addition unit 710 into a cache area 732 (sometimes also referred to as a "first cache area"). The write speed of the cache area 732 is generally greater than the write speed of the storage device 330. If a "data hazard" occurs, that is, when it is determined that the accumulated element cannot be read from the corresponding storage position of the storage device 330 when the addition unit is to perform the accumulation of the third element and the previous accumulated element, the bypass module 730 is configured to provide the accumulated element cached in the cache area 732 to the addition unit 710. In this way, the addition unit 710 can quickly perform further accumulation of the third element and the accumulated element when the third element is available. In some cases, if it is determined that no "data hazard" occurs, that is, the data reading unit 740 can read the accumulated element from the storage device 330 in time, the bypass module 730 can provide the accumulated element read from the storage device 330 to the addition unit 710 for accumulation.

[0074] The cache time of the accumulated element in the cache area 732 may be multiple clock cycles of the deep learning processing device 220. The specific cached time may depend on the size of the cache area 732 and / or the time extension that may cause "data hazard" (i.e., the cache time must be greater than the time extension). This time extension usually depends on the operation delay of the addition unit 710, the data writing delay of the data writing unit 720, and the data reading delay of the data reading unit 740. In some embodiments, if the time difference between the time when the accumulated element is calculated and cached in the cache area 732 and the arrival time of the third element is less than the time extension, it can be determined that the accumulated element cannot be read from the storage device 330 when the addition unit 710 is to perform the accumulation of the third element and the accumulated element, so the accumulated element in the cache area 732 can be provided to the addition unit 710 as input.

[0075] In the following, we will refer to Figure 8 Further details of example embodiments of the present disclosure are described. Figure 8 The flowchart of a method 800 for performing deep learning processing according to an exemplary embodiment of the present disclosure is schematically shown. The method 800 is at least partially implemented by the deep learning processing device 220.

[0076] At 810, the deep learning processing device 220 enables at least one matrix multiplication and addition module to perform a matrix multiplication and addition operation of a convolution kernel parameter value matrix of a convolution layer in a convolutional neural network and a first error gradient value matrix to obtain a plurality of intermediate matrices. At 820, the deep learning processing device 220 stores the plurality of intermediate matrices in a storage device without reshaping the elements in the plurality of intermediate matrices. At 830, the deep learning processing device 220 reads the plurality of intermediate matrices from the storage device. At 840, the deep learning processing device 220 enables a plurality of matrix accumulation modules to perform a matrix accumulation operation based on the plurality of intermediate matrices in a parallel manner according to a convolution scheme of the convolution layer to obtain a second error gradient value matrix for the convolution layer.

[0077] In some embodiments, multiple intermediate matrices are associated with multiple channels of the input of the convolution layer, and the second error gradient value matrix includes multiple channel matrices for the multiple channels. In some embodiments, making multiple matrix accumulation modules perform matrix accumulation operations in parallel according to the convolution scheme of the convolution layer includes one of the following: making each matrix accumulation module in the multiple matrix accumulation modules accumulate the intermediate matrices associated with one channel in the multiple channels on one parallel path of the multiple parallel paths corresponding to the multiple channels to obtain a channel matrix for the channel; or making each matrix accumulation module in the multiple matrix accumulation modules accumulate one of the multiple intermediate matrices to an intermediate result of one channel matrix in the multiple channel matrices at each accumulation.

[0078] In some embodiments, causing multiple matrix accumulation modules to perform matrix accumulation operations in parallel according to a convolution scheme of a convolution layer includes causing at least one matrix accumulation module among the multiple matrix accumulation modules to: accumulate a first element and a second element to obtain an accumulated element; write the accumulated element to a storage device; cache the accumulated element to a first cache area, wherein a write speed of the first cache area is faster than a write speed of the storage device, and the accumulated element is cached in the first cache area for multiple clock cycles; and based on determining that the accumulated element cannot be read out from the storage device when the addition operation unit is to perform accumulation of a third element and the accumulated element, the accumulated element cached in the first cache area is provided for accumulation with the third element.

[0079] In some embodiments, the deep learning processing device 220 also caches the first intermediate operation results generated by at least one matrix multiplication and addition module during the matrix multiplication and addition operation and the second intermediate operation results generated by multiple matrix accumulation modules during the matrix accumulation operation into the second cache area.

[0080] In some embodiments, the deep learning processing device 220 also enables the vector calculation module of the deep learning processing device to perform at least one of the following: setting the storage area in the second cache area for storing the first intermediate operation result and the second intermediate operation result to zero at the vector level; and selectively writing the final operation result of the matrix accumulation operation to the storage device of the deep learning processing device at the vector level as at least a part of the second error gradient value matrix.

[0081] In some embodiments, the deep learning processing device 220 also obtains an instruction set for convolution direction operations of the convolution layer of the CNN from the general processing device and parses the instruction set; and generates control signals for at least one matrix multiplication and addition module and multiple matrix accumulation modules based on the parsed instruction set.

[0082] In some embodiments, at least one matrix multiply-add module includes multiple matrix multiply-add modules, wherein causing at least one matrix multiply-add module to perform matrix multiply-add operations includes causing multiple matrix multiply-add modules to perform matrix multiply-add operations in parallel.

[0083] Fig. 9 A schematic block diagram of a computing device 900 that can be used to implement an embodiment of the present disclosure is shown. As shown, the device 900 includes a processing device 901, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 902 or computer program instructions loaded from a storage unit 908 to a random access memory (RAM) 903. The processing device 901 may include one or more deep learning processing devices 220 described in the present disclosure. The processing device 901 may also include one or more general processing devices 210.

[0084] In the RAM 903, various programs and data necessary for the operation of the computing device 900 may also be stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0085] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0086] The processing device 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processing device 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processing device 901 performs the various methods and processes described above, such as method 800. For example, in some embodiments, the method 800 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the processing device 901, one or more steps of the method 800 described above may be performed. Alternatively, in other embodiments, the processing device 901 may be configured to perform the method 800 in any other appropriate manner (e.g., by means of firmware).

[0087] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip systems (SOCs), load programmable logic devices (CPLDs), and the like.

[0088] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0089] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0090] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination mode.

[0091] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A deep learning processing device for performing a convolution inverse operation, the deep learning processing device comprising: At least one matrix multiplication and addition module is configured to perform a matrix multiplication and addition operation of a convolution kernel parameter value matrix of a convolution layer in a convolutional neural network and a first error gradient value matrix to obtain a plurality of intermediate matrices, wherein the first error gradient value matrix is ​​an error gradient value matrix of a subsequent layer of the convolution layer in a forward direction; Storage means for storing the plurality of intermediate matrices without reshaping the elements in the plurality of intermediate matrices, so that the plurality of intermediate matrices are stored in an ordering manner of the elements generated by the at least one matrix multiplication-addition module; as well as A plurality of matrix accumulation modules are configured to read the plurality of intermediate matrices from the storage device and perform matrix accumulation operations based on the plurality of intermediate matrices in parallel according to the convolution scheme of the convolution layer to obtain a second error gradient value matrix for the convolution layer.

2. The deep learning processing device according to claim 1, wherein the plurality of intermediate matrices are associated with a plurality of channels of the input of the convolutional layer, and the second error gradient value matrix includes a plurality of channel matrices for the plurality of channels, Each matrix accumulation module in the plurality of matrix accumulation modules is configured as: On one parallel path of the plurality of parallel paths corresponding to the plurality of channels, accumulating the intermediate matrices associated with one of the plurality of channels among the plurality of intermediate matrices to obtain the channel matrix for the channel; or In each accumulation, one of the plurality of intermediate matrices is accumulated to an intermediate result of one of the plurality of channel matrices.

3. The deep learning processing device according to claim 1, wherein the storage device is configured to store the second error gradient value matrix in a target storage location, the second error gradient value matrix initially having all zero values; and The plurality of matrix accumulation modules are configured to accumulate the plurality of intermediate matrices and the second error gradient value matrix having the initial value stored at the target storage location in parallel.

4. The deep learning processing device according to claim 1, wherein at least one matrix accumulation module among the plurality of matrix accumulation modules comprises: an addition operation unit, configured to receive a first element and a second element in a matrix to be accumulated and accumulate the first element and the second element to obtain an accumulated element; a data writing unit, configured to write the accumulated element into the storage device; A first buffer area is configured to cache the accumulated element, the writing speed of the first buffer area is faster than the writing speed of the storage device, and the accumulated element is cached in the first buffer area for multiple clock cycles; as well as The bypass module is configured to provide the accumulation element cached in the first buffer area to the addition operation unit according to determining that the accumulation element cannot be read out from the storage device when the addition operation unit is to perform accumulation of the third element and the accumulation element. 5 . The deep learning processing device according to claim 1 , wherein the storage device comprises an on-chip random access memory.

6. The deep learning processing apparatus according to claim 1, further comprising: The second cache area is configured to cache the first intermediate operation result generated by the at least one matrix multiplication and addition module during the matrix multiplication and addition operation process and to cache the second intermediate operation results generated by the multiple matrix accumulation modules during the matrix accumulation operation process.

7. The deep learning processing apparatus according to claim 6, further comprising: The vector calculation module is configured to perform at least one of the following: Setting a storage area in the second buffer area for storing the first intermediate operation result and the second intermediate operation result to zero at the vector level; as well as The final operation result of the matrix accumulation operation at the vector level is selectively written into the storage device of the deep learning processing device as at least a part of the second error gradient value matrix, so as to filter and fill the elements of the second error gradient value matrix row by row or column by column at the vector level.

8. The deep learning processing device according to any one of claims 1 to 7, further comprising: An instruction processing module is configured to obtain an instruction set for a convolution direction operation of the convolution layer of the CNN from a general processor and parse the instruction set; as well as The control signal generating module is configured to generate control signals for the at least one matrix multiplication and addition module and the plurality of matrix accumulation modules based on the parsed instruction set.

9. The deep learning processing device according to any one of claims 1 to 7, wherein the at least one matrix multiplication-addition module comprises a plurality of matrix multiplication-addition modules configured to perform the matrix multiplication-addition operations in parallel.

10. A method for performing a deep learning process, for performing a convolution inverse operation, the method comprising: At least one matrix multiplication and addition module of the deep learning processing device performs a matrix multiplication and addition operation of a convolution kernel parameter value matrix of a convolution layer in a convolutional neural network and a first error gradient value matrix to obtain a plurality of intermediate matrices, wherein the first error gradient value matrix is ​​an error gradient value matrix of a subsequent layer of the convolution layer in a forward direction; storing the plurality of intermediate matrices in a storage device without reshaping the elements in the plurality of intermediate matrices so that the plurality of intermediate matrices are stored in an ordering of the elements generated by the at least one matrix multiplication-addition module; reading the plurality of intermediate matrices from the storage device; as well as The multiple matrix accumulation modules of the deep learning processing device perform matrix accumulation operations based on the multiple intermediate matrices in parallel according to the convolution scheme of the convolution layer to obtain a second error gradient value matrix for the convolution layer.

11. The method according to claim 10, wherein the plurality of intermediate matrices are associated with a plurality of channels of the input of the convolutional layer, the second error gradient value matrix comprises a plurality of channel matrices for the plurality of channels, The step of causing the plurality of matrix accumulation modules to perform the matrix accumulation operation in parallel according to the convolution scheme of the convolution layer comprises one of the following: Each matrix accumulation module in the plurality of matrix accumulation modules accumulates, on one of the plurality of parallel paths corresponding to the plurality of channels, the intermediate matrices associated with one of the plurality of channels in the plurality of intermediate matrices to obtain the channel matrix for the channel; or Each matrix accumulation module in the plurality of matrix accumulation modules accumulates one of the plurality of intermediate matrices into an intermediate result for one of the plurality of channel matrices during each accumulation.

12. The method according to claim 10, wherein causing the plurality of matrix accumulation modules to perform the matrix accumulation operation in parallel according to the convolution scheme of the convolution layer comprises causing at least one matrix accumulation module of the plurality of matrix accumulation modules to: Add the first element and the second element to obtain the accumulated element; Writing the accumulated element into the storage device; caching the accumulated element in a first cache area, wherein a writing speed of the first cache area is faster than a writing speed of the storage device, and the accumulated element is cached in the first cache area for multiple clock cycles; as well as According to the determination that the accumulation element cannot be read out from the storage device when the addition operation unit is to perform accumulation of the third element and the accumulation element, the accumulation element cached in the first buffer area is provided for accumulation with the third element.

13. The method according to claim 10, further comprising: The first intermediate operation result generated by the at least one matrix multiplication and addition module during the matrix multiplication and addition operation and the second intermediate operation results generated by the multiple matrix accumulation modules during the matrix accumulation operation are cached in a second cache area.

14. The method according to claim 13, further comprising causing the vector calculation module of the deep learning processing device to perform at least one of the following: Setting the storage area in the second buffer area for storing the first intermediate operation result and the second intermediate operation result to zero at the vector level; and The final operation result of the matrix accumulation operation at the vector level is selectively written into the storage device of the deep learning processing device as at least a part of the second error gradient value matrix, so as to filter and fill the elements of the second error gradient value matrix row by row or column by column at the vector level.

15. The method according to any one of claims 10 to 14, further comprising: Acquire an instruction set for a convolution direction operation of the convolution layer of the CNN from a general-purpose processor and parse the instruction set; as well as Control signals for the at least one matrix multiplication-addition module and the plurality of matrix accumulation modules are generated based on the parsed instruction set.

16. The method according to any one of claims 10 to 14, wherein the at least one matrix multiply-add module comprises a plurality of matrix multiply-add modules, wherein causing the at least one matrix multiply-add module to perform the matrix multiply-add operation comprises: The plurality of matrix multiplication-addition modules are enabled to perform the matrix multiplication-addition operations in parallel.

17. An electronic device comprising: At least one deep learning processing device according to any one of claims 1 to 9; as well as At least one general purpose processor is coupled to at least one of the deep learning processing devices and is configured to provide instructions for performing training of a convolutional neural network (CNN) to at least one of the deep learning processing devices.

18. A computer-readable storage medium having a computer program stored thereon, wherein the program implements the method according to any one of claims 10 to 16 when executed by a processor.

19. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 10 to 16.

Images