A computing processing method and device based on a deep neural network processor

Abstract

The application provides a kind of based on deep neural network processor's computing processing method and device, the method includes: obtaining input data, weight data and instruction stream;According to input data, construct preset input matrix, according to weight data, construct weight matrix;In response to instruction stream, the first matrix transformation is done to preset input matrix, obtain input matrix;The first calculation is executed to input matrix and weight matrix, and output matrix is obtained;Output matrix is transmitted to the next layer of quantization deep neural network, to be used as the input matrix of the next layer of quantization deep neural network.The above-mentioned method, construct preset input matrix and weight matrix, can adapt to the data of variable precision and simultaneously adapt to pulsating array, improve computing efficiency;The first calculation is executed to input matrix and weight matrix, and the output matrix obtained after calculation is similar to input matrix form, can be directly transmitted to the next layer and inquired, without transformation, improve the performance and throughput of quantization DNN processor.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a computational processing method and apparatus based on a deep neural network processor. Background Technology

[0002] Deep Neural Networks (DNNs) construct deep convolutional neural networks and use sample data as input to obtain a model with analytical and recognition capabilities, which is applied in fields such as speech recognition and image recognition. In the field of artificial intelligence, extreme edge platforms such as wearable devices and in-vehicle intelligent devices are also beginning to use DNNs to meet intelligent requirements.

[0003] Due to the large computational load and parameter count of DNN models, they require significant memory and computational power during operation. However, extreme edge platforms such as wearable devices and in-vehicle intelligent devices have limited storage space, power consumption, and hardware resources, making it difficult to meet the demands of DNNs. To enable the operation of various intelligent applications and meet customized needs under conditions of limited energy consumption, memory space, and computational resources, variable-precision quantization DNNs are deployed on extreme edge platforms. Using processors that support variable-precision quantization DNNs for computation can effectively improve performance.

[0004] However, because variable precision quantization DNN processors need to be adapted to the deployment of DNNs with different precision quantization, the data flow and data storage methods during computation are complex. The data computation flow and pre-storage methods will cause time redundancy and reduce the performance of the processor. Furthermore, the operation process of variable precision quantization DNN processors is complex and the instruction flow is difficult to control, which reduces the throughput of the processor. Summary of the Invention

[0005] This application provides a computational processing method and apparatus based on a deep neural network processor, which can improve the performance and throughput of a quantized deep neural network processor.

[0006] In a first aspect, this application provides a computational processing method based on a deep neural network processor, including: acquiring input data, weight data, and instruction stream.

[0007] A preset input matrix is ​​constructed based on the input data, and a weight matrix is ​​constructed based on the weight data. The length of the preset input matrix is ​​equal to the number of input tensor channels, and the width of the preset input matrix is ​​equal to the product of the length and width of the input tensor. The length of the weight matrix is ​​equal to the number of output tensor channels, and the width of the weight matrix is ​​equal to the product of the length of the weight tensor, the width of the weight tensor, and the number of input tensor channels.

[0008] In response to the instruction stream, a first matrix transformation is performed on the preset input matrix to obtain the input matrix adapted to the systolic array. The length of the input matrix is ​​equal to the product of the length of the weight tensor, the width of the weight tensor, and the number of channels of the input tensor. The width of the input matrix is ​​the product of the length of the output tensor and the width of the output tensor.

[0009] Perform the first calculation on the input matrix and weight matrix to obtain the output matrix, where the length of the output matrix is ​​equal to the number of channels of the output tensor, and the width of the output matrix is ​​equal to the product of the length and width of the output tensor.

[0010] The output matrix is ​​transmitted to the next layer of the quantized DNN through a non-linear activation function, serving as the input matrix for the next layer of the quantized DNN.

[0011] It should be noted that the input tensor is the original data corresponding to the input matrix, the weight tensor is the original data corresponding to the weight matrix, and the output tensor is the original data corresponding to the output matrix.

[0012] In the above method, a preset input matrix is ​​constructed based on the input data, and a weight matrix is ​​constructed based on the weight data. This method can adapt to data with variable precision and simultaneously adapt to the systolic array used to perform the first calculation, thereby improving computational efficiency. The first calculation is performed on the input matrix and the weight matrix. The output matrix obtained after the calculation is similar in form to the input matrix and is directly passed to the next layer for inference, realizing the inference process of the quantized DNN. Moreover, the output matrix can be used to obtain the input matrix of the next layer without transformation, thereby improving the performance and throughput of the quantized DNN processor.

[0013] In one possible implementation of the first aspect, the method further includes: obtaining the input gradient matrix from the next layer of the quantized DNN as the output gradient matrix of the current layer of the quantized DNN, wherein the length of the output gradient matrix is ​​equal to the number of channels of the output tensor, and the width of the output gradient matrix is ​​equal to the product of the length and width of the output tensor.

[0014] A second matrix transformation is performed on the preset input matrix of the current layer of the quantized DNN to obtain the input matrix of the current layer of the quantized DNN adapted to the systolic array. The length of the input matrix of the current layer of the quantized DNN is equal to the product of the length and width of the output tensor, and the width of the input matrix of the current layer of the quantized DNN is equal to the product of the length of the weight tensor, the width of the weight tensor, and the number of channels of the input tensor.

[0015] Perform the first calculation on the input matrix and output gradient matrix of the current layer of the quantized DNN to obtain the preset weight gradient matrix of the current layer of the quantized DNN. The length of the preset weight gradient matrix is ​​equal to the number of output tensor channels, and the width of the preset weight gradient matrix is ​​equal to the product of the length of the weight tensor, the width of the weight tensor, and the number of input tensor channels.

[0016] A third matrix transformation is performed on the preset weight gradient matrix to obtain the weight gradient matrix, where the length of the weight gradient matrix is ​​equal to the number of input tensor channels, and the width of the weight gradient matrix is ​​equal to the product of the length of the weight tensor, the width of the weight tensor, and the number of output tensor channels. A fourth matrix transformation is performed on the output gradient matrix to obtain the transformed output gradient matrix, where the length of the transformed output gradient matrix is ​​equal to the product of the length and width of the input tensor, and the width of the transformed output gradient matrix is ​​equal to the product of the length of the weight tensor, the width of the weight tensor, and the number of output tensor channels.

[0017] Perform the first calculation on the weight gradient matrix and the transformed output gradient matrix to obtain the input gradient matrix, where the length of the input gradient matrix is ​​equal to the number of channels of the input tensor, and the width of the input gradient matrix is ​​equal to the product of the length and width of the input tensor.

[0018] The input gradient matrix is ​​transmitted to the layer above the quantized DNN through a nonlinear activation function, and then used as the output gradient matrix of the layer above the quantized DNN.

[0019] It should be noted that the input tensor is the original data corresponding to the input matrix, the weight tensor is the original data corresponding to the weight matrix, and the output tensor is the original data corresponding to the output matrix.

[0020] In the above method, the input gradient matrix of the next layer of the quantized DNN is first obtained as the output gradient matrix of the current layer of the quantized DNN. A first calculation is then performed with the input matrix of the current layer of the quantized DNN to obtain the weight gradient matrix of the current layer of the quantized DNN. This weight gradient matrix is ​​retained within the current layer of the quantized DNN for updating the weights. Next, the weight gradient matrix of the current layer of the quantized DNN is compared with the output gradient matrix of the current layer of the quantized DNN to perform the first calculation, resulting in the updated input gradient matrix of the current layer of the quantized DNN. This updated input gradient matrix is ​​then used as the output gradient matrix of the previous layer of the quantized DNN. The above process is repeated to update the weight gradient matrix of each layer of the quantized DNN, thus realizing the on-chip training of the quantized DNN. This method achieves efficient computation with fewer transformations, thereby improving the efficiency of on-chip training of the quantized DNN on a processor.

[0021] In one possible implementation of the first aspect, the method further includes: storing the input matrix in column-major order; storing the weight matrix in row-major order; and storing the output matrix obtained by the first calculation of the input matrix and the weight matrix in column-major order.

[0022] The above method describes the data storage approach for the inference process. To adapt to the systolic array performing the first calculation, the row height of the input matrix is ​​the same as the height of the systolic array. One row is fed into the systolic array at a time, and within each row, the input data is stored in column-major order. The column width of the weight matrix is ​​the same as the width of the systolic array. One column is fed into the systolic array at a time, and within each column, the weight data is stored in row-major order. This ensures that data is efficiently fed into the systolic array and correct calculation results are obtained. Furthermore, the output matrix of the systolic array is stored in the same way as the input matrix, facilitating the passing of the output matrix to the next layer of the quantized DNN as the input matrix for calculation.

[0023] In one possible implementation of the first aspect, the method further includes: storing the output gradient matrix in row-major order; storing the input matrix in column-major order; and storing the weight gradient matrix obtained by the first calculation of the input matrix and the output gradient matrix in column-major order.

[0024] The above method is the data storage method for the weight update process. The input matrix is ​​stored in column-major order. In order to adapt to the systolic array that performs the first calculation, the other matrix that participates in the first calculation, namely the output gradient matrix, is stored in row-major order to ensure that the data can be efficiently sent into the systolic array and obtain the correct calculation results.

[0025] In one possible implementation of the first aspect, the method further includes: storing the weight gradient matrix in row-major order; storing the transformed output gradient matrix in column-major order; and storing the input gradient matrix obtained by the first calculation of the weight gradient matrix and the output gradient matrix in column-major order.

[0026] The above method is the data storage method for the input update process. The weight gradient matrix is ​​stored in row-major order. In order to adapt to the systolic array that performs the first calculation, the transformed output gradient matrix is ​​stored in column-major order to ensure that the data can be efficiently sent into the systolic array and obtain the correct calculation results.

[0027] In one possible implementation of the first aspect, the method further includes: adjusting the input data to have a consistent data bit width and maximum parallelism, and adjusting the weight data to have a consistent data bit width and maximum parallelism.

[0028] The adjusted input data is split by row and stored in the row in a column-major manner. The row height after splitting by row is the same as the height of the systolic array. The adjusted weight data is split by column and stored in the column in a row-major manner. The column width after splitting by column is the same as the column width of the systolic array. The systolic array is the unit for performing the first calculation.

[0029] In the above method, adjusting the data to have a consistent bit width and maximizing the parallelism can ensure that data of different precisions are stored in a uniform unit of a certain number of bits, thereby maximizing the parallelism and ensuring the consistency of data storage. The adjusted input data and weight data are stored in column-major and row-major order respectively, and the height of the rows and the width of the columns after partitioning are the same as the height and width of the pulsation array, which improves the efficiency and accuracy of the first calculation.

[0030] In one possible implementation of the first aspect, the instruction stream includes: initialization configuration instruction, input matrix start address configuration instruction, weight matrix start address configuration instruction, data length configuration instruction, data loading instruction, and data storage instruction.

[0031] Using the instruction flow described above can effectively reduce the number of instructions and cycles while maintaining the same number of units required for computation, thereby improving computational efficiency and increasing the throughput of quantized DNN processors.

[0032] In one possible implementation of the first aspect, the method further includes: in response to a data loading instruction, summing the width and height of the input matrix to obtain first data; adding a preset first constant to the first data to obtain second data; multiplying the width or height of the pulsating array by a preset second constant to obtain third data; adding a preset third constant to the third data to obtain fourth data; and summing the second and fourth data to obtain fifth data, wherein the fifth data is the number of cycles executed by the data loading instruction.

[0033] In response to the data storage command, the width or height of the pulsating array is multiplied by two to obtain the sixth data; the sixth data is subtracted from the preset fourth constant to obtain the seventh data. The seventh data is the number of cycles in which the data storage command is executed, where the pulsating array is the unit that performs the first calculation.

[0034] In the above method, the number of instructions and cycles required to perform the first calculation on the input matrix and weight matrix of different dimensions using systolic arrays of different dimensions can be calculated by using the height and width of the input matrix and the height and width of the systolic array.

[0035] In one possible implementation of the first aspect, the method further includes: performing regularization processing on the data to be processed and storing the data to be processed in matrix form.

[0036] In the above method, before the input data and weight data are constructed into the input matrix and weight matrix, the input data and weight data are pre-stored as matrices according to specific rules, which facilitates the subsequent processing of the input matrix and weight matrix and improves the efficiency of quantized DNN in processing data.

[0037] In a second aspect, a computing device based on a deep neural network processor is provided, comprising: a data acquisition module configured to acquire input data, weight data, and instruction stream;

[0038] The forward propagation module is configured to: construct a preset input matrix based on the input data and construct a weight matrix based on the weight data, wherein the length of the preset input matrix is ​​equal to the number of input tensor channels; the width of the preset input matrix is ​​equal to the product of the length and width of the input tensor; the length of the weight matrix is ​​equal to the number of output tensor channels; and the width of the weight matrix is ​​equal to the product of the length of the weight tensor, the width of the weight tensor, and the number of input tensor channels.

[0039] In response to the instruction stream, a first matrix transformation is performed on the preset input matrix to obtain the input matrix adapted to the systolic array. The length of the input matrix is ​​equal to the product of the length of the weight tensor, the width of the weight tensor, and the number of channels of the input tensor. The width of the input matrix is ​​the product of the length of the output tensor and the width of the output tensor.

[0040] Perform the first calculation on the input matrix and weight matrix to obtain the output matrix, where the length of the output matrix is ​​equal to the number of channels of the output tensor, and the width of the output matrix is ​​equal to the product of the length and width of the output tensor.

[0041] The output matrix is ​​transmitted to the next layer of the quantized DNN through a non-linear activation function, serving as the input matrix for the next layer of the quantized DNN.

[0042] As can be seen from the above technical solutions, this application provides a computational processing method and apparatus based on a deep neural network processor. The computational processing method of the quantized DNN processor includes: acquiring input data, weight data, and instruction stream; constructing a preset input matrix based on the input data, and constructing a weight matrix based on the weight data, wherein the length of the preset input matrix is ​​equal to the number of input tensor channels, and the width of the preset input matrix is ​​equal to the product of the length and width of the input tensor; the length of the weight matrix is ​​equal to the number of output tensor channels, and the width of the weight matrix is ​​equal to the product of the length of the weight tensor, the width of the weight tensor, and the number of input tensor channels; responding to the instruction stream... The method involves performing a first matrix transformation on a preset input matrix to obtain an input matrix adapted to the systolic array. The length of the input matrix is ​​equal to the product of the length, width, and number of channels of the weight tensor, and the width is the product of the length and width of the output tensor. A first calculation is then performed on the input and weight matrices to obtain an output matrix. The length of the output matrix is ​​equal to the number of channels of the output tensor, and the width is equal to the product of the length and width of the output tensor. The output matrix is ​​then transmitted to the next layer of the quantized DNN using a non-linear activation function as its input matrix. This method constructs a preset input matrix based on the input data and a weight matrix based on the weight data, adapting to variable precision data and systolic arrays, thus improving computational efficiency. The first calculation on the input and weight matrices yields an output matrix similar in form to the input matrix, which can be directly transmitted to the next layer for inference without transformation, improving the performance and throughput of the quantized DNN processor. Attached Figure Description

[0043] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart of the quantization DNN processor inference process in some embodiments of this application;

[0045] Figure 2 This is a schematic diagram of the quantization DNN processor inference process in some embodiments of this application;

[0046] Figure 3 This is a flowchart illustrating the on-chip training process of a quantized DNN processor in some embodiments of this application;

[0047] Figure 4 This application explains the principle of the on-chip training process of the quantized DNN processor in some embodiments. Figure 1 ;

[0048] Figure 5 This application explains the principle of the on-chip training process of the quantized DNN processor in some embodiments. Figure 2 ;

[0049] Figure 6 This is a schematic diagram of the data pre-storage method in some embodiments of this application;

[0050] Figure 7 This is a schematic diagram of the instruction flow in some embodiments of this application. Detailed Implementation

[0051] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0052] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0053] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0054] Deep Neural Networks (DNNs) construct deep convolutional neural networks and use sample data as input to obtain a model with analytical and recognition capabilities. This model uses multiple hidden layers to process the data, and each hidden layer can learn and optimize higher-level features of the data. By passing information layer by layer, DNNs can handle complex nonlinear problems and are applied in fields such as speech recognition and image recognition. In the field of artificial intelligence, DNNs can also be used on extreme edge platforms such as wearable devices and in-vehicle intelligent devices to achieve intelligent analysis and recognition.

[0055] Due to the large computational load and parameter count of DNN models, they require significant memory and computational power during operation. However, extreme edge platforms such as wearable devices and in-vehicle intelligent devices have limited storage space, power consumption, and hardware resources, making it difficult to meet the demands of DNNs. Quantized DNNs, on the other hand, utilize less storage overhead, lower bandwidth requirements, and lower power consumption, offering advantages such as faster computation speeds. To meet the diverse needs of intelligent applications and customization requirements under conditions of limited energy consumption, memory space, and computational resources, variable-precision quantized DNNs are deployed on extreme edge platforms. Using processors that support variable-precision quantized DNNs for computation can effectively improve performance.

[0056] However, because variable precision quantization DNN processors need to be adapted to the deployment of DNNs with different precision quantization, the data flow and data storage methods during computation are complex. The data computation flow and pre-storage methods will cause time redundancy and reduce the performance of the processor. Furthermore, the operation process of variable precision quantization DNN processors is complex and the instruction flow is difficult to control, which reduces the throughput of the processor.

[0057] This application provides a computational processing method and apparatus based on a deep neural network processor, which enables the quantized DNN processor to support variable precision quantized DNN computation and improves the performance and throughput of the quantized DNN processor.

[0058] See Figure 1 This application provides a computational processing method based on a deep neural network processor, comprising:

[0059] S101, Obtain input data, weight data, and instruction stream.

[0060] The input data and weight data can be data of various precisions, such as data with precisions of INT16, INT8, INT4, INT2, FP16, BF16, etc.; the instruction stream is used to control the computation process of the quantized DNN processor.

[0061] S102. Construct a preset input matrix based on the input data, and construct a weight matrix based on the weight data.

[0062] For example, to adapt to a systolic array, the input data and weight data need to be converted into matrix form. Taking a 4x4 systolic array as an example, see [link to relevant documentation]. Figure 2The input data 201 is processed and transformed into a preset input matrix 204. The length of the preset input matrix 204 is equal to the number of channels of the input data 201 (e.g., the number of channels is 3), and the width of the preset input matrix 204 is equal to the product of the length (e.g., the length is 4) and the width (e.g., the width is 4) of the input data 201 (i.e., 16). The weight data 202 is processed and transformed into a weight matrix 206. The length of the weight matrix 206 is equal to the number of channels of the output data 203 (e.g., the number of channels is 4), and the width of the weight matrix 206 is equal to the product of the length (e.g., the length is 2), the width (e.g., the width is 2), and the number of channels of the input data 201 (e.g., the number of channels is 3) (i.e., 12).

[0063] S103. In response to the instruction stream, perform a first matrix transformation on the preset input matrix to obtain the input matrix adapted to the systolic array.

[0064] For example, after the input data is converted into a preset input matrix, a first matrix transformation is performed to convert it into a matrix with a width equal to the height of the pulsation array. See [link to relevant documentation]. Figure 2 A first matrix transformation is performed on the preset input matrix 204 to transform it into an input matrix 205. The length of the input matrix 205 is equal to the product of the length of the weight data 202 (e.g., length 2), the width of the weight data 202 (e.g., width 2), and the number of channels of the input data 201 (e.g., number of channels 3) (i.e., 12). The width of the input matrix 205 is the product of the length of the output data 203 (e.g., length 2) and the width of the output data 203 (e.g., width 2) (i.e., 4). The output data 203 is the original data corresponding to the output matrix 207.

[0065] It is understood that the first matrix transformation represents any matrix transformation that can change the position of each element in the matrix without changing the value of the matrix elements. For example, elementary row transformations, elementary column transformations, etc., are not specifically limited in this application.

[0066] S104. Perform the first calculation on the input matrix and weight matrix to obtain the output matrix.

[0067] For example, see Figure 2The input matrix 205 and weight matrix 206 are input into a 4*4 systolic array to perform the first calculation, resulting in an output matrix 207. The output data 203 is the original data corresponding to the output matrix 207. The length of the output matrix 207 is equal to the number of channels in the output data 203 (e.g., 4 channels), and the width of the output matrix 207 is equal to the product of the length (e.g., 2) and width (e.g., 2) of the output data 203 (i.e., 4). Specifically, the width of the input matrix 205 and the height of the systolic array are both 4, the length of the weight matrix 206 and the width of the systolic array are both 4, and the length and width of the output matrix 207 are equal to the width and height of the systolic array, both being 4.

[0068] It is understood that the first calculation refers to matrix multiplication calculation. In other embodiments, it can also be other calculation methods, such as matrix addition, matrix scalar multiplication, etc., which are not specifically limited in this application.

[0069] S105. The output matrix is ​​transmitted to the next layer of the quantized DNN through a non-linear activation function, so as to serve as the input matrix of the next layer of the quantized DNN.

[0070] For example, see Figure 2 The output matrix 207 is transmitted to the next layer of the quantized DNN through a non-linear activation function to continue inference.

[0071] As can be seen from the above embodiments, the input matrix 205 and the weight matrix 206 are used to obtain the output matrix 207 through the first calculation. Then, the output matrix 207 is passed to the next layer of the quantized DNN through a nonlinear activation function as the input matrix 205 to continue the first calculation, so as to realize the inference process of the quantized DNN.

[0072] In some embodiments, see Figure 3 The aforementioned computational processing method based on deep neural network processors also includes:

[0073] S201. Obtain the input gradient matrix from the next layer of the quantized DNN, and use it as the output gradient matrix of the current layer of the quantized DNN.

[0074] For example, see Figure 4 The output gradient matrix 407 receives the input gradient matrix of the next layer of the self-quantized DNN, and the output gradient data 403 is the original data corresponding to the output gradient matrix 407. The length of the output gradient matrix 407 is equal to the number of channels of the output gradient data 403 (e.g., the number of channels is 4), and the width of the output gradient matrix 407 is equal to the product of the length (e.g., the length is 2) and the width (e.g., the width is 2) of the output gradient data 403 (i.e., 4).

[0075] S202. Perform a second matrix transformation on the preset input matrix of the current layer of the quantized DNN to obtain the input matrix of the current layer of the quantized DNN adapted to the systolic array.

[0076] For example, to adapt to a 4x4 pulsating array, the matrices involved in the first calculation must be transformed into an appropriate form. See [link to documentation]. Figure 4 A second matrix transformation is performed on the preset input matrix 404 to transform it into an input matrix 405. The length of the input matrix 405 is equal to the product of the length (e.g., length 2) and width (e.g., width 2) of the output data 403 (e.g., the product is 4). The width of the input matrix 405 is equal to the product of the length (e.g., length 2) of the weight data 402, the width of the weight data 402, and the number of channels of the input data 401 (e.g., the number of channels is 3) (i.e., 12).

[0077] It is understood that the second matrix transformation represents any matrix transformation that can change the position of each element in the matrix without changing the value of the matrix elements. For example, elementary row transformations, elementary column transformations, etc., are not specifically limited in this application.

[0078] S203. Perform the first calculation on the input matrix and output gradient matrix of the current layer of the quantized DNN to obtain the preset weight gradient matrix of the current layer of the quantized DNN.

[0079] For example, see Figure 4 The input matrix 405 and the output gradient matrix 407 are input into a 4*4 pulsating array to perform the first calculation, resulting in a preset weight gradient matrix 406. The weight gradient data 402 is the original data corresponding to the preset weight gradient matrix 406. The length of the preset weight gradient matrix 406 is equal to the number of channels of the output gradient data 403 (e.g., the number of channels is 4). The width of the preset weight gradient matrix 406 is equal to the product of the length of the weight gradient data 402 (e.g., the length is 2), the width of the weight gradient data 402 (e.g., the width is 2), and the number of channels of the input data (e.g., the number of channels is 3) (i.e., 12).

[0080] S204. Perform a third matrix transformation on the preset weight gradient matrix to obtain the weight gradient matrix, and perform a fourth matrix transformation on the output gradient matrix to obtain the transformed output gradient matrix.

[0081] For example, to adapt to a 4x4 pulsating array, the matrices involved in the first calculation must be transformed into an appropriate form. See [link to documentation]. Figure 5A third matrix transformation is performed on the preset weight gradient matrix 504 to transform it into a weight gradient matrix 505. The length of the weight gradient matrix 505 is equal to the number of channels in the input gradient data 503 (e.g., 3 channels), and the width of the weight gradient matrix 505 is equal to the product of the length of the weight data 501 (e.g., 2 channels), the width of the weight data 501 (e.g., 2 channels), and the number of channels in the output gradient data 502 (e.g., 4 channels) (i.e., 16). The output gradient matrix... 506 performs the fourth matrix transformation to obtain the transformed output gradient matrix 507. The length of the transformed output gradient matrix 507 is equal to the product of the length (e.g., length 4) and width (e.g., width 4) of the input gradient data 503 (i.e., 16). The width of the transformed output gradient matrix 507 is equal to the product of the length (e.g., length 2) of the weight data 501, the width (e.g., width 2) of the weight data 501, and the number of channels (e.g., number of channels 4) of the output gradient data 502 (i.e., 16).

[0082] It is understood that the third and fourth matrix transformations represent any matrix transformation that can change the position of each element in the matrix without changing the value of the element. For example, elementary row transformations, elementary column transformations, etc., are not specifically limited in this application.

[0083] S205. Perform the first calculation on the weight gradient matrix and the transformed output gradient matrix to obtain the input gradient matrix.

[0084] For example, see Figure 5 The weight gradient matrix 505 and the output gradient matrix 507 are input into a 4*4 pulsating array to perform the first calculation, resulting in the input gradient matrix 508. The input gradient data 503 is the original data corresponding to the input gradient matrix 508. The length of the input gradient matrix 508 is equal to the number of channels of the input gradient data 503 (e.g., the number of channels is 3), and the width of the input gradient matrix 508 is equal to the product of the length (e.g., the length is 4) and the width (e.g., the width is 4) of the input gradient data 503 (i.e., 16).

[0085] It is understood that the first calculation refers to matrix multiplication calculation. In other embodiments, it can also be other calculation methods, such as matrix addition, matrix scalar multiplication, etc., which are not specifically limited in this application.

[0086] S206. The input gradient matrix is ​​transmitted to the layer above the quantized DNN through a nonlinear activation function, so as to serve as the output gradient matrix of the layer above the quantized DNN.

[0087] For example, see Figure 5 The input gradient matrix 508 is passed to the next layer of the quantized DNN through a non-linear activation function to continue training.

[0088] As can be seen from the above embodiments, the input gradient matrix of the next layer of the quantized DNN is transmitted back to the previous layer as the output gradient matrix 407 to update the weight gradient matrix 406 and the input gradient matrix 508. The input gradient matrix 508 is then transmitted to the next higher layer, and so on, to realize the training process of the quantized DNN.

[0089] It is understood that the systolic array mentioned in the foregoing embodiments is used to perform the first calculation. The dimension of the systolic array can also be other dimensions, as long as the systolic array is compatible with the matrix participating in the first calculation, such as an 8*8 systolic array, a 16*16 systolic array, etc. In addition to the systolic array, the unit that performs the first calculation can also have other architectures, such as a multiplication tree, a half-systolic array, etc.

[0090] In some embodiments, during the inference process of a quantized DNN, it is necessary to store the data required for inference. Data storage methods include:

[0091] Adjust the input data to have a consistent data bit width and maximize parallelism, and adjust the weight data to have a consistent data bit width and maximize parallelism.

[0092] The adjusted input data is split by row and stored in the row in a column-major manner. The row height after splitting by row is the same as the height of the systolic array. The adjusted weight data is split by column and stored in the column in a row-major manner. The column width after splitting by column is the same as the column width of the systolic array. The systolic array is the unit for performing the first calculation.

[0093] See Figure 6 First, according to precision, multiple sets of data are packaged in 32-bit units.

[0094] For example, Figure 6 The method shown in 601 packs 4 INT8 data into 1 32-bit data for low-precision data, 8 INT4 data into 1 32-bit data, and 16 INT2 data into 1 32-bit data.

[0095] Figure 6 The method shown in 601 adds 16 zeros before each FP16, BF16 and INT16 data for relatively high precision 16-bit data, and packs them into a 32-bit data.

[0096] It is understood that the above packaging method can also package data of other precisions, and can also package data with other bit lengths, such as 64 bits, 128 bits, etc., which are not specifically limited in this application.

[0097] Next, the input data and weight data are stored in the form of an adaptive systolic array.

[0098] For example, consider a 4x4 pulsating array. Figure 6 The input matrix 6021 has a row height of 4 and is stored in column-major order.

[0099] Taking a 4x4 pulsating array as an example, Figure 6 The weight matrix 6022 in the data has a column width of 4 and is stored in row-major order.

[0100] Finally, the output matrix calculated from the input matrix and weight matrix using a systolic array is stored as a matrix with the same structure as the input matrix, so that the output matrix can be transmitted to the next layer of the quantized DNN as the input matrix for calculation.

[0101] For example, Figure 6 The output matrix 6023 has a row height of 4 and is stored in column-major order, consistent with the form of the input matrix 6021.

[0102] In other embodiments, the training process of the quantized DNN can also adopt the data storage method described in the above embodiments, including:

[0103] During the stage of updating the weight gradient matrix, the output gradient matrix is ​​stored in row-major order; the input matrix is ​​stored in column-major order; and the weight gradient matrix obtained by the first calculation of the input matrix and the output gradient matrix is ​​stored in column-major order.

[0104] During the stage of updating the input gradient matrix, the weight gradient matrix is ​​stored in row-major order; the transformed output gradient matrix is ​​stored in column-major order; and the input gradient matrix obtained by the first calculation of the weight gradient matrix and the transformed output gradient matrix is ​​stored in column-major order.

[0105] For example, see Figure 4 The output gradient matrix 407 is stored in row-major order; the input matrix 405 is stored in column-major order; and the weight gradient matrix 406 is stored in column-major order.

[0106] See Figure 5 The weight gradient matrix 505 is stored in row-major order; the transformed output gradient matrix 507 is stored in column-major order; and the input gradient matrix 508 is stored in column-major order.

[0107] It is understandable that the storage method of each matrix participating in the first calculation may be row-major or column-major. To ensure that each matrix can adapt to the systolic array, the selection is made according to the actual calculation process.

[0108] In some embodiments, the data to be processed is regularized and stored in matrix form.

[0109] Understandably, systolic arrays only support matrix multiplication, so the data input to a systolic array needs to be converted into matrix form. Regularizing the data before processing it into matrix form facilitates subsequent processing of the input and weight matrices, improving the efficiency of quantized DNNs in processing data.

[0110] In some embodiments, the instruction stream in the computation processing method of this application includes:

[0111] Initialization configuration instructions, input matrix starting address configuration instructions, weight matrix starting address configuration instructions, data length configuration instructions, data loading instructions, and data storage instructions.

[0112] For example, see Figure 7 The first calculation is performed using a 4x4 systolic array to connect matrix A (4 channels, 4 lengths, 4 widths) and matrix B (4 channels, 4 lengths, 4 widths). This requires a total of 7 instructions: initialization configuration instruction, matrix B start address configuration instruction, data length configuration instruction, data loading instruction, and data storage instruction. Specifically, the initialization configuration instruction has 4 cycles, the matrix A start address configuration instruction has 1 cycle, the matrix B start address configuration instruction has 1 cycle, the data length configuration instruction has 1 cycle, the data loading instruction has 19 cycles, and the data storage instruction has 7 cycles.

[0113] In some embodiments, in response to a data loading instruction, the width and length of the input matrix are summed to obtain first data; the first data is added to a preset first constant to obtain second data; the width or height of the pulsating array is multiplied by a preset second constant to obtain third data; the third data is added to a preset third constant to obtain fourth data; and the second and fourth data are summed to obtain fifth data, whereby the fifth data is the number of cycles in which the data loading instruction is executed.

[0114] In response to the data storage command, the width or height of the pulsating array is multiplied by two to obtain the sixth data; the sixth data is subtracted from the preset fourth constant to obtain the seventh data, which is the number of cycles of the data storage command being executed.

[0115] For example, see Figure 7 The pulsating array has a dimension of 4*4, the matrix A has a width of 4 and a length of 4, the first preset constant is 1, the second preset constant is 2, and the third preset constant is 2. Then the number of cycles for executing the data loading instruction is 19.

[0116] See Figure 7The pulsating array has a dimension of 4*4, the matrix A has a width of 4 and a length of 4, and the fourth constant is preset to be 1. Then the number of cycles for executing the data storage instruction is 7.

[0117] It is understood that the number of cycles required for each instruction in the above embodiments is not fixed and must be determined based on the dimension of the pulsation array and the dimension of the matrix involved in the first calculation.

[0118] As can be seen from the above technical solutions, this application provides a computational processing method and apparatus based on a deep neural network processor. The apparatus includes: a data acquisition module configured to acquire input data, weight data, and instruction stream; and a forward propagation module configured to construct a preset input matrix based on the input data and a weight matrix based on the weight data, wherein the length of the preset input matrix is ​​equal to the number of input tensor channels; the width of the preset input matrix is ​​equal to the product of the length and width of the input tensor; the length of the weight matrix is ​​equal to the number of output tensor channels; and the width of the weight matrix is ​​equal to the product of the length of the weight tensor, the width of the weight tensor, and the number of input tensor channels. In response to the instruction stream, a first matrix transformation is performed on the preset input matrix to obtain an input matrix adapted to the systolic array. The length of the input matrix is ​​equal to the product of the length of the weight tensor, the width of the weight tensor, and the number of channels in the input tensor. The width of the input matrix is ​​the product of the length and width of the output tensor. A first calculation is performed on the input matrix and the weight matrix to obtain an output matrix. The length of the output matrix is ​​equal to the number of channels in the output tensor, and the width of the output matrix is ​​equal to the product of the length and width of the output tensor. The output matrix is ​​then transmitted to the next layer of the quantized DNN using a nonlinear activation function as the input matrix for that layer. This device constructs a preset input matrix based on the input data and a weight matrix based on the weight data, adapting to variable precision data and systolic arrays, thus improving computational efficiency. The first calculation is performed on the input matrix and the weight matrix, and the resulting output matrix, similar in form to the input matrix, is directly transmitted to the next layer for inference, realizing the inference process of the quantized DNN. Furthermore, the output matrix does not require transformation to obtain the input matrix for the next layer, improving the performance and throughput of the quantized DNN processor.

[0119] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.

Claims

1. A method of computing processing based on a deep neural network processor, characterized by, include: Acquire input data, weight data, and instruction stream; A preset input matrix is ​​constructed based on the input data, and a weight matrix is ​​constructed based on the weight data. The length of the preset input matrix is ​​equal to the number of channels of the input tensor, and the width of the preset input matrix is ​​equal to the product of the length and width of the input tensor. The length of the weight matrix is ​​equal to the number of output tensor channels, and the width of the weight matrix is ​​equal to the product of the length of the weight tensor, the width of the weight tensor, and the number of input tensor channels. In response to the instruction stream, a first matrix transformation is performed on the preset input matrix to obtain an input matrix adapted to the systolic array. The length of the input matrix is ​​equal to the product of the length of the weight tensor, the width of the weight tensor, and the number of input tensor channels. The width of the input matrix is ​​the product of the length of the output tensor and the width of the output tensor. Perform a first calculation on the input matrix and the weight matrix to obtain an output matrix. The length of the output matrix is ​​equal to the number of channels of the output tensor, and the width of the output matrix is ​​equal to the product of the length and width of the output tensor. The output matrix is ​​transmitted to the next layer of the quantized deep neural network through a nonlinear activation function, so as to serve as the input matrix of the next layer of the quantized deep neural network. Obtain the input gradient matrix from the next layer of the quantized deep neural network, and use it as the output gradient matrix of the current layer of the quantized deep neural network. The length of the output gradient matrix is ​​equal to the number of channels of the output tensor, and the width of the output gradient matrix is ​​equal to the product of the length and width of the output tensor. A second matrix transformation is performed on the preset input matrix of the current layer of the quantized deep neural network to obtain the input matrix of the current layer of the quantized deep neural network adapted to the systolic array. The length of the input matrix of the current layer of the quantized deep neural network is equal to the product of the length and width of the output tensor, and the width of the input matrix of the current layer of the quantized deep neural network is equal to the product of the length of the weight tensor, the width of the weight tensor, and the number of channels of the input tensor. Perform the first calculation on the input matrix and the output gradient matrix of the current layer of the quantized deep neural network to obtain the preset weight gradient matrix of the current layer of the quantized deep neural network. The length of the preset weight gradient matrix is ​​equal to the number of output tensor channels, and the width of the preset weight gradient matrix is ​​equal to the product of the length of the weight tensor, the width of the weight tensor, and the number of input tensor channels. A third matrix transformation is performed on the preset weight gradient matrix to obtain a weight gradient matrix. The length of the weight gradient matrix is ​​equal to the number of input tensor channels, and the width of the weight gradient matrix is ​​equal to the product of the length of the weight tensor, the width of the weight tensor, and the number of output tensor channels. A fourth matrix transformation is performed on the output gradient matrix to obtain a transformed output gradient matrix. The length of the transformed output gradient matrix is ​​equal to the product of the length and width of the input tensor, and the width of the transformed output gradient matrix is ​​equal to the product of the length of the weight tensor, the width of the weight tensor, and the number of output tensor channels. The first calculation is performed on the weight gradient matrix and the transformed output gradient matrix to obtain the input gradient matrix. The length of the input gradient matrix is ​​equal to the number of channels of the input tensor, and the width of the input gradient matrix is ​​equal to the product of the length and width of the input tensor. The input gradient matrix is ​​transmitted to the layer above the quantized deep neural network through a nonlinear activation function, so as to serve as the output gradient matrix of the layer above the quantized deep neural network.

2. The deep neural network processor based computing processing method of claim 1, wherein, The method further includes: The output gradient matrix is ​​stored in row-major order. The input matrix is ​​stored in column-major order. The weight gradient matrix obtained by the first calculation of the input matrix and the output gradient matrix is ​​stored in column-major order.

3. The computational processing method based on a deep neural network processor according to claim 1, characterized in that, The method further includes: The weight gradient matrix is ​​stored in row-major order. The transformed output gradient matrix is ​​stored in column-major order. The input gradient matrix, obtained by performing the first calculation on the weight gradient matrix and the transformed output gradient matrix, is stored in column-major order.

4. The computational processing method based on a deep neural network processor according to claim 1, characterized in that, The method further includes: The input matrix is ​​stored in column-major order. The weight matrix is ​​stored in row-major order. The output matrix obtained by the first calculation of the input matrix and the weight matrix is ​​stored in column-major order.

5. The computational processing method based on a deep neural network processor according to claim 1, characterized in that, The method further includes: The input data is adjusted to have a consistent data bit width and maximum parallelism, and the weight data is adjusted to have a consistent data bit width and maximum parallelism. The adjusted input data is divided into rows and stored in rows in column-major order, and the height of the row after the row division is the same as the height of the pulsation array. The adjusted weight data is divided into columns and stored in columns in a row-major manner. The column width after column division is the same as the column width of the pulsation array, wherein the pulsation array is the unit that performs the first calculation.

6. The computational processing method based on a deep neural network processor according to claim 1, characterized in that, The instruction stream includes: Initialization configuration instructions, input matrix starting address configuration instructions, weight matrix starting address configuration instructions, data length configuration instructions, data loading instructions, and data storage instructions.

7. The computational processing method based on a deep neural network processor according to claim 6, characterized in that, The method further includes: In response to the data loading instruction, the width and height of the input matrix are summed to obtain the first data; Add a preset first constant to the first data to obtain the second data; Multiply the width or height of the pulsating array by a preset second constant to obtain the third data; Add a preset third constant to the third data to obtain the fourth data; The second data and the fourth data are summed to obtain the fifth data, which is the number of cycles of the data loading instruction execution; In response to the data storage instruction, the width or height of the pulsating array is multiplied by two to obtain the sixth data; Subtracting a preset fourth constant from the sixth data yields the seventh data, which is the number of cycles in which the data storage instruction is executed. The pulsating array is the unit that performs the first calculation.

8. The computational processing method based on a deep neural network processor according to claim 1, characterized in that, The method further includes: The data to be processed is processed according to rules, and the data to be processed is stored in matrix form.

9. A computing device based on a deep neural network processor, characterized in that, include: The data acquisition module is configured to acquire input data, weight data, and instruction streams. The forward propagation module is configured to: construct a preset input matrix based on the input data, construct a weight matrix based on the weight data, wherein the length of the preset input matrix is ​​equal to the number of input tensor channels, and the width of the preset input matrix is ​​equal to the product of the length and width of the input tensor; The length of the weight matrix is ​​equal to the number of output tensor channels, and the width of the weight matrix is ​​equal to the product of the length of the weight tensor, the width of the weight tensor, and the number of input tensor channels. In response to the instruction stream, a first matrix transformation is performed on the preset input matrix to obtain an input matrix adapted to the systolic array. The length of the input matrix is ​​equal to the product of the length of the weight tensor, the width of the weight tensor, and the number of input tensor channels. The width of the input matrix is ​​the product of the length of the output tensor and the width of the output tensor. Perform a first calculation on the input matrix and the weight matrix to obtain an output matrix. The length of the output matrix is ​​equal to the number of channels of the output tensor, and the width of the output matrix is ​​equal to the product of the length and width of the output tensor. The output matrix is ​​transmitted to the next layer of the quantized deep neural network through a nonlinear activation function, so as to serve as the input matrix of the next layer of the quantized deep neural network. The backpropagation module is configured to: obtain the input gradient matrix from the next layer of the quantized deep neural network as the output gradient matrix of the current layer of the quantized deep neural network, wherein the length of the output gradient matrix is ​​equal to the number of channels of the output tensor, and the width of the output gradient matrix is ​​equal to the product of the length and width of the output tensor; A second matrix transformation is performed on the preset input matrix of the current layer of the quantized deep neural network to obtain the input matrix of the current layer of the quantized deep neural network adapted to the systolic array. The length of the input matrix of the current layer of the quantized deep neural network is equal to the product of the length and width of the output tensor, and the width of the input matrix of the current layer of the quantized deep neural network is equal to the product of the length of the weight tensor, the width of the weight tensor, and the number of channels of the input tensor. Perform the first calculation on the input matrix and the output gradient matrix of the current layer of the quantized deep neural network to obtain the preset weight gradient matrix of the current layer of the quantized deep neural network. The length of the preset weight gradient matrix is ​​equal to the number of output tensor channels, and the width of the preset weight gradient matrix is ​​equal to the product of the length of the weight tensor, the width of the weight tensor, and the number of input tensor channels. A third matrix transformation is performed on the preset weight gradient matrix to obtain a weight gradient matrix. The length of the weight gradient matrix is ​​equal to the number of input tensor channels, and the width of the weight gradient matrix is ​​equal to the product of the length of the weight tensor, the width of the weight tensor, and the number of output tensor channels. A fourth matrix transformation is performed on the output gradient matrix to obtain a transformed output gradient matrix. The length of the transformed output gradient matrix is ​​equal to the product of the length and width of the input tensor, and the width of the transformed output gradient matrix is ​​equal to the product of the length of the weight tensor, the width of the weight tensor, and the number of output tensor channels. The first calculation is performed on the weight gradient matrix and the transformed output gradient matrix to obtain the input gradient matrix. The length of the input gradient matrix is ​​equal to the number of channels of the input tensor, and the width of the input gradient matrix is ​​equal to the product of the length and width of the input tensor. The input gradient matrix is ​​transmitted to the layer above the quantized deep neural network through a nonlinear activation function, so as to serve as the output gradient matrix of the layer above the quantized deep neural network.

Images