A neural network processor and data processing method

Abstract

The application provides a neural network processor and a data processing method. The neural network processor can include: a first cache unit configured to store first input data; a second cache unit configured to store weight data; an operation unit connected to the first cache unit and the second cache unit, and configured to perform matrix multiplication on the first input data and the weight data to obtain an intermediate calculation result; a compression unit configured to process the intermediate calculation result to obtain processed data; the processed data includes compressed data and uncompressed data; and a storage unit access controller connected to the compression unit and configured to store the processed data into a memory outside the neural network processor. By implementing the application, the IO bandwidth can be saved, and the complexity of the compression operation can be reduced.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a neural network processor and a data processing method. Background Technology

[0002] A neural network is an algorithmic mathematical model that mimics the behavioral characteristics of animal neural networks to perform distributed parallel information processing. This type of network consists of a large number of interconnected nodes (or neurons). By adjusting the relationships between the interconnections of these nodes, it uses input data and weights to generate output data, simulating the information processing process of the human brain to process information and generate the results after pattern recognition.

[0003] For neural network operations, if the neural network has multiple layers, the input and output neurons do not refer to the neurons in the entire input and output layers of the neural network. Instead, for any two adjacent layers in the network, the neuron in the lower layer of the forward operation is the input neuron, and the neuron in the upper layer of the forward operation is the output neuron. Taking a convolutional neural network model as an example, suppose a convolutional neural network has L layers, K = 1, 2, ..., L-1. For the K-th layer and the (K+1)-th layer, we call the K-th layer the input layer, and the neurons in it are input neurons. The (K+1)-th layer is called the output layer, and the neurons in it are output neurons. That is, except for the top layer, each layer can be used as an input layer, and the layer below it is the corresponding output layer.

[0004] In the computation process of a neural network model, the output of the previous layer serves as the input for the next layer. Normally, the processor stores the output of the previous layer in its internal output data buffer. When performing a subsequent layer's computation, the processor first reads the previously stored output from the buffer and then uses it as the input. However, when the amount of intermediate computation results from the previous layer is large, due to the capacity limitations of the output data buffer, these intermediate results must be stored in external memory. Therefore, when performing a subsequent layer's computation, the processor must first retrieve the stored intermediate results from external memory and then use them as the input.

[0005] In existing technologies, when processors compress intermediate computation results, they often compress the entire intermediate result and then store the compressed result in external memory. When performing the next layer of computation in the neural network, the compressed intermediate result needs to be decompressed to obtain the decompressed data, which is then used in the computation. While adding compression and decompression operations to the processor reduces the amount of data transferred from the processor to memory, the applicant has found that for certain neural network layers (e.g., the first layer), the sparsity of the first layer data is typically low. Processing the first layer data using the aforementioned compression and decompression methods results in minimal I / O bandwidth gains and also incurs power consumption losses due to the complexity of compression and decompression. Therefore, reducing the complexity of compression and decompression operations while ensuring I / O bandwidth gains is a pressing technical problem that needs to be solved. Summary of the Invention

[0006] This application provides a neural network processor and a data processing method that can reduce the amount of data transferred from the neural network processor to external memory, and also reduce the complexity of compression operations, thus facilitating the subsequent reduction of the complexity of decompression operations.

[0007] In a first aspect, a neural network processor is provided, which may include: a first cache unit, a second cache unit, a computation unit, a compression unit, and a storage unit access controller; wherein, the first cache unit is used to store first input data, specifically, the first input data may be the raw data initially input to the neural network processor for inference operations, such as image data, voice data, etc., or it may be intermediate data (also known as intermediate calculation results) generated by the neural network processor during the execution of neural network operations; the second cache unit is used to store weight data; the computation unit is connected to the first cache unit and the second cache unit, and is used to perform matrix multiplication operations on the first input data and the weight data to obtain intermediate calculation results, wherein the intermediate calculation results refer to the calculation results generated by the neural network processor during the execution of neural network operations; the compression unit is used to process the intermediate calculation results to obtain processed data; the processed data includes compressed data and uncompressed data; the storage unit access controller is connected to the compression unit, and is used to store the processed data in a memory outside the neural network processor.

[0008] In implementing the embodiments of this application, a compression unit is added to the neural network processor. When processing the intermediate calculation results obtained by the arithmetic unit, the compression unit can obtain both compressed and uncompressed data. Then, the processed data is stored in external memory through the storage unit access controller. Compared to the prior art, since it is not necessary to compress all intermediate calculation results, but only a portion of the data in the intermediate calculation results, on the one hand, the amount of data transferred from the neural network processor to external memory is reduced, saving the I / O bandwidth of the neural network processor; on the other hand, the complexity of the compression operation is reduced, facilitating the subsequent reduction of the complexity of the decompression operation.

[0009] In one possible implementation, the neural network processor may further include a decompression unit connected to the aforementioned memory access controller. This decompression unit receives data to be decompressed from storage by the memory access controller, decompresses the compressed data within the decompressed data, and leaves the uncompressed data uncompressed. The decompressed and uncompressed data are then stored as a second input data in a first cache unit. The arithmetic unit can then retrieve the second input data from the first cache unit to perform matrix multiplication. By implementing this embodiment, the addition of a decompression unit to the neural network processor reduces the amount of data transferred from memory to the processor for the next computation. Since the decompression unit only needs to decompress the compressed data and does not need to process the uncompressed data, the complexity of the decompression operation is reduced.

[0010] In one possible implementation, the length of the uncompressed data is less than the block length. The uncompressed data includes first data and second data, which are stored in the same memory space corresponding to the same starting address. The first data is the last block of intermediate calculation results from the previous layer, and the second data is the first block of intermediate calculation results from the current layer. For example, if the block length is 4kb, the first data length is 3kb, and the second data length is 1kb, the method described in this application stores the first and second data in a 4kb memory space. By implementing this embodiment, when storing data processed by the compression unit, the memory can store the last block of data from the previous layer and the first block of data from the current layer in the same memory space, thus saving memory space.

[0011] In one possible implementation, the last block of data is the data between the first address and the end address of the intermediate calculation result of the previous layer, where the first address is an address that is an integer multiple of the block length; the first block of data is the data between the start address and the second address of the intermediate calculation result of the current layer, where the second address is the address preceding the third address, and the third address is an address that is an integer multiple of the block length.

[0012] In one possible implementation, the compression unit can be specifically used to: when the starting address corresponding to the intermediate calculation result is not an integer multiple of the block length, divide the intermediate calculation result from the starting address to the first block address into blocks to obtain the first block data; starting from the second block address, divide the intermediate calculation result into blocks according to the block length and compress them to obtain the processed data; wherein, the second block address is an address that satisfies an integer multiple of the block length; the first block address is the address preceding the second block address; the first block data is not compressed; when the data length of the last block is less than the block length, the data of the last block is not compressed. For example, the starting address corresponding to the intermediate calculation result is 19kb, the data length is 5kb, the ending address is 23kb, and the block length is 4kb. The neural network processor determines that the starting address corresponding to the intermediate calculation result is not an integer multiple of the block length. It then designates the data corresponding to 19kb as the first block and the data between 20kb and 23kb as the second block. The first block has a length of 1kb, and the second block has a length of 4kb. In this case, the first block is not compressed, while the second block is compressed. Implementing this embodiment, the neural network processor can divide the data into blocks based on the starting address, data length, and determined block length of the intermediate calculation result. If the data length of a block equals the block length, the block is compressed; if the data length is less than the block length, it is not compressed. This results in compressed and uncompressed data. The processed data is then stored in external memory via a memory access controller. Compared to existing technologies, since it is not necessary to compress all intermediate calculation results, but only some data in the intermediate calculation results, on the one hand, it reduces the amount of data to be transferred from the neural network processor to external memory, saving the I / O bandwidth of the neural network processor; on the other hand, it reduces the complexity of the compression operation, which facilitates the subsequent reduction of the complexity of the decompression operation.

[0013] In one possible implementation, the data to be decompressed is stored in the memory according to a preset alignment. When processing the processed data, the decompression unit may include: if the starting address corresponding to the data to be decompressed is not an integer multiple of the block length, obtaining the second block data between the first alignment address and the second alignment address, obtaining the block data according to the block length starting from the second alignment address, and decompressing the block data; the first alignment address and the second alignment address are addresses that satisfy an integer multiple of the block length; the first alignment address is an alignment address before the starting address; the second alignment address is an alignment address after the starting address; wherein the second block data is not decompressed, and data after the starting address is obtained; in the last block, data between the alignment address corresponding to the last block and the termination address corresponding to the data to be decompressed is obtained. For example, the starting address of the data to be decompressed is 19kb, the data length is 5kb, the ending address is 23kb, and the block length is 4kb. The decompression unit determines that the starting address of the data to be decompressed is not an integer multiple of the block length, and receives the data to be decompressed between 16kb and 23kb obtained from the memory by the storage unit access controller. Here, the first alignment address is 16kb (the first alignment address is an alignment address before the starting address), and the second alignment address is 20kb. In the first block of data, the first block of data is not decompressed, and the data after the starting address 19kb is obtained; in the second block of data, the second decomposed data is decompressed to obtain the decompressed data. By implementing the embodiments of this application, the decompression unit can obtain the corresponding block data according to the starting address and block length of the data to be decompressed. When the block data is compressed, the compressed block data is decompressed; when the block data is uncompressed, there is no need to decompress the uncompressed block data. Since the decompression unit only needs to decompress the compressed data and does not need to process the uncompressed data, the complexity of the decompression operation can be reduced.

[0014] In one possible implementation, the compressed data includes compressed block data, and the uncompressed data includes uncompressed block data. For the compressed block data, a first identifier is generated corresponding to the compressed block data, which indicates successful compression and the length of the compressed data. The first identifier is controlled by a storage unit access controller and stored in memory. For the uncompressed block data, a second identifier is generated corresponding to the uncompressed block data, which indicates compression failure. The second identifier is controlled by a storage unit access controller and stored in memory. By implementing this embodiment, the block data processed by the compression unit includes corresponding identifiers, allowing the decompression unit to decompress each block data according to its respective identifier, thus improving decompression efficiency.

[0015] In one possible implementation, when processing the data to be decompressed, the decompression unit may receive the data to be decompressed from the memory by the storage unit access controller; when the identifier corresponding to the block data is a successful compression identifier, the block data is decompressed and the decompressed data is stored as the second input data in the first cache unit; when the identifier corresponding to the block data is a failed compression identifier, the block data is stored as the second input data in the first cache unit.

[0016] Secondly, embodiments of this application provide a data processing method that can be applied to a neural network processor. The neural network processor may include a first cache unit, a second cache unit, a computation unit, a compression unit, and a storage unit access controller. The method may include the following steps: First, the computation unit performs matrix multiplication on first input data obtained from the first cache unit and weight data obtained from the second cache unit to obtain intermediate calculation results; then, the compression unit processes the intermediate calculation results to obtain processed data; the processed data includes compressed data and uncompressed data; finally, the storage unit access controller stores the processed data in a memory outside the neural network processor.

[0017] In implementing this embodiment, a compression unit is added to the neural network processor. When processing intermediate calculation results obtained by the arithmetic unit, the compression unit can obtain both compressed and uncompressed data. Then, the processed data is stored in external memory via the storage unit access controller. Compared to existing technologies, since it is not necessary to compress all intermediate calculation results, but only a portion of the data, on the one hand, the amount of data transferred from the neural network processor to external memory is reduced, saving the I / O bandwidth of the neural network processor; on the other hand, the complexity of the compression operation is reduced, facilitating the subsequent reduction of the complexity of the decompression operation.

[0018] In one possible implementation, the neural network processor further includes a decompression unit; in this case, the method may further include the following steps: receiving data to be decompressed from memory by a memory access controller via the decompression unit, decompressing the compressed data in the data to be decompressed, and storing the decompressed data and uncompressed data as second input data in a first cache unit; performing matrix multiplication on the second input data obtained from the first cache unit via an arithmetic unit, thereby realizing the computation of the next layer of the neural network.

[0019] In one possible implementation, the length of the uncompressed data is less than the block length. The uncompressed data includes first data and second data, which are stored in the same memory space corresponding to the same starting address. The first data is the last block of intermediate calculation results from the previous layer, and the second data is the first block of intermediate calculation results from the current layer.

[0020] In one possible implementation, the last block of data is the data between the first address and the end address of the intermediate calculation result of the previous layer, where the first address is an address that is an integer multiple of the block length; the first block of data is the data between the start address and the second address of the intermediate calculation result of the current layer, where the second address is the address preceding the third address, and the third address is an address that is an integer multiple of the block length.

[0021] In one possible implementation, the process of processing intermediate calculation results through a compression unit to obtain processed data may include: when the starting address corresponding to the intermediate calculation result is not an integer multiple of the block length, dividing the intermediate calculation result from the starting address to the first block address into blocks to obtain the first block data; starting from the second block address, dividing the intermediate calculation result into blocks according to the block length and compressing them to obtain the processed data; wherein, the second block address is an address that satisfies an integer multiple of the block length; the first block address is the address preceding the second block address; the first block data is not compressed; when the data length of the last block is less than the block length, the data of the last block is not compressed.

[0022] In one possible implementation, the data to be decompressed is stored in memory according to a preset alignment. Decompressing the compressed data in the data to be decompressed includes: if the starting address corresponding to the data to be decompressed is not an integer multiple of the block length, obtaining the second block of data between the first alignment address and the second alignment address, obtaining the block data according to the block length starting from the second alignment address, and decompressing the block data; the first alignment address and the second alignment address are addresses that satisfy an integer multiple of the block length; the first alignment address is an alignment address before the starting address; the second alignment address is an alignment address after the starting address; wherein the second block of data is not decompressed, and the data after the starting address is obtained; in the last block, the data between the alignment address corresponding to the last block and the termination address corresponding to the data to be decompressed is obtained.

[0023] In one possible implementation, the compressed data includes compressed block data, and the uncompressed data includes uncompressed block data. The method may further include the following steps: for the compressed block data, generating a first identifier corresponding to the compressed block data, the first identifier being used to characterize successful block data compression and the length of the compressed block data; wherein the first identifier is controlled by a storage unit access controller and stored in a memory; for the uncompressed block data, generating a second identifier corresponding to the uncompressed block data, the second identifier being used to characterize block data compression failure; wherein the second identifier is controlled by a storage unit access controller and stored in a memory.

[0024] In one possible implementation, the above method may further include the following steps: receiving data to be decompressed obtained from the memory by the storage unit access controller; when the identifier corresponding to the block data is a successful compression identifier, decompressing the block data and storing the decompressed data as a second input data in the first cache unit; when the identifier corresponding to the block data is a failed compression identifier, storing the block data as a second input data in the first cache unit. Attached Figure Description

[0025] Figure 1 A schematic diagram of the structure of a convolutional neural network (CNN) 100 provided in this application embodiment;

[0026] Figure 2 A schematic diagram of another convolutional neural network (CNN) 100 provided in this application embodiment;

[0027] Figure 3 A schematic diagram illustrating the first application scenario provided in this application embodiment;

[0028] Figure 4 A schematic diagram illustrating a second application scenario provided in an embodiment of this application;

[0029] Figure 5a A hardware structure diagram of a neural network processor 50 provided in an embodiment of this application;

[0030] Figure 5b A hardware structure diagram of another neural network processor 50 provided in an embodiment of this application;

[0031] Figure 6a This is a schematic diagram of the structure of a computation unit 503 in a neural network processor provided in an embodiment of this application;

[0032] Figure 6b This is a schematic diagram of the structure of a vector computation unit in a neural network processor provided in an embodiment of this application;

[0033] Figure 7a A schematic diagram of a block result corresponding to an intermediate calculation result provided in an embodiment of this application;

[0034] Figure 7b A schematic diagram illustrating block data compression provided in an embodiment of this application;

[0035] Figure 7c A schematic diagram illustrating another form of block data compression provided in an embodiment of this application;

[0036] Figure 7d This is a schematic diagram illustrating a storage space alignment setting provided in an embodiment of this application;

[0037] Figure 7e A schematic diagram illustrating a memory for storing a first intermediate calculation result and a second intermediate calculation result, provided as an embodiment of this application;

[0038] Figure 8a A flowchart illustrating a data processing method provided in an embodiment of this application;

[0039] Figure 8bA flowchart illustrating another data processing method provided in an embodiment of this application;

[0040] Figure 9 This is a schematic flowchart of a method for controlling a neural network processor to perform compression, provided in an embodiment of this application. Detailed Implementation

[0041] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0042] The terms "first" and "second," etc., used in the specification and drawings of this application are used to distinguish different objects or to differentiate different treatments of the same object, rather than to describe a specific order of objects. Furthermore, the terms "comprising" and "having," and any variations thereof, mentioned in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. It should be noted that in the embodiments of this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design method described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being superior or more advantageous than other embodiments or design solutions. Specifically, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner. In the embodiments of this application, "A and / or B" means both A and B, or A or B. "A, and / or B, and / or C" means any one of A, B, and C, or means any two of A, B, and C, or means A, B, and C.

[0043] To facilitate a better understanding of the technical solutions described in this application, the technical terms involved in the embodiments of this application will be explained below:

[0044] (1) Convolutional Neural Network.

[0045] A Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. It is a deep learning architecture, which refers to learning at multiple levels of abstraction using machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network where each neuron responds to overlapping regions in the input image.

[0046] like Figure 1 As shown, the Convolutional Neural Network (CNN) 100 may include an input layer 110, convolutional / pooling layers 120 (where pooling layers are optional), and neural network layers 130. A detailed description follows:

[0047] Convolutional / pooling layers 120:

[0048] Convolutional layers:

[0049] like Figure 1 The convolutional / pooling layer 120 shown may include layers 121-126 as in Examples 121-126. In one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer. In another implementation, layers 121 and 122 are convolutional layers, layer 123 is a pooling layer, layers 124 and 125 are convolutional layers, and layer 126 is a pooling layer. That is, the output of the convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0050] Taking convolutional layer 121 as an example, it can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter to extract specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix typically processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features from the image. The size of the weight matrix is ​​related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, multiple weight matrices of the same dimension are applied instead of a single weight matrix. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract image edge information, another weight matrix can be used to extract specific colors from the image, and yet another weight matrix can be used to blur unwanted noise in the image. These multiple weight matrices have the same dimension, and the feature maps extracted by these multiple weight matrices with the same dimension also have the same dimension. The extracted feature maps with the same dimension are then merged to form the output of the convolution operation.

[0051] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions.

[0052] When a convolutional neural network 100 has multiple convolutional layers, the initial convolutional layers (e.g., 121) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by later convolutional layers (e.g., 126) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0053] Pooling layer:

[0054] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, i.e., ... Figure 1 Layers 121-126 in example 120 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. Average pooling calculates the average value of pixel values ​​within a specific range. Max pooling takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after pooling can be smaller than the size of the input image of the pooling layer. Each pixel in the output image represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0055] Neural network layer 130:

[0056] After processing by the convolutional / pooling layers 120, the convolutional neural network 100 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 120 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 100 needs to utilize neural network layers 130 to generate one or more outputs representing the required number of classes. Therefore, neural network layers 130 may include multiple hidden layers (such as...). Figure 1As shown in layers 131, 132 to 13n) and output layer 140, the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0057] After the multiple hidden layers in neural network layer 130, the final layer of the entire convolutional neural network 100 is the output layer 140. This output layer 140 has a loss function similar to classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 100 has undergone forward propagation (e.g., ...), the loss function is applied. Figure 3 The propagation from 110 to 140 is completed (forward propagation), and the reverse propagation (such as...) Figure 1 The propagation from 140 to 110 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 100 and the error between the output of the convolutional neural network 100 through the output layer and the ideal result.

[0058] It should be noted that, as Figure 1 The convolutional neural network 100 shown is merely an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models, such as... Figure 2 The multiple convolutional / pooling layers shown are used in parallel, and the extracted features are all input into the neural network layer 130 for processing.

[0059] (2) Operations in neural networks

[0060] In this embodiment, the operation in the neural network can be the operation of one layer in the neural network. For a multi-layer neural network, the implementation process is as follows: In the forward operation, after the previous layer of the artificial neural network has completed its execution, the operation instruction of the next layer will use the output neuron calculated in the operation unit as the input neuron of the next layer for operation (or perform some operations on the output neuron and then use it as the input neuron of the next layer), and at the same time, the weights will also be replaced with the weights of the next layer; In the backward operation, after the backward operation of the previous layer of the artificial neural network has been completed, the operation instruction of the next layer will use the gradient of the input neuron calculated in the operation unit as the gradient of the output neuron of the next layer for operation (or perform some operations on the gradient of the input neuron and then use it as the gradient of the output neuron of the next layer), and at the same time, the weights will be replaced with the weights of the next layer.

[0061] The aforementioned neural network operations can also include support vector machine operations, k-nearest neighbor (k-NN) operations, k-means operations, principal component analysis operations, and so on. For ease of description, the following uses artificial neural network operations as an example to illustrate the specific scheme of machine learning computation.

[0062] For artificial neural network operations, if the artificial neural network has multiple layers, the input and output neurons of these layers do not refer to the neurons in the entire input and output layers of the neural network. Instead, for any two adjacent layers in the network, the neuron in the layer below the forward operation is the input neuron, and the neuron in the layer above the forward operation is the output neuron. Taking a convolutional neural network as an example, suppose a convolutional neural network has L layers, K = 1, 2, ..., L. For the Kth layer and the (K+1)th layer, we call the Kth layer the input layer, and its neurons are input neurons; the (K+1)th layer is called the output layer, and its neurons are output neurons. That is, except for the top layer, each layer can be used as an input layer, and the layer below it is the corresponding output layer.

[0063] (3) Input data

[0064] In this embodiment of the application, the input data can be the original data initially input to the neural network processor for inference operations, such as image data, voice data, etc., or it can be the intermediate data (also known as intermediate calculation results) generated by the neural network processor during the execution of neural network operations. Since the amount of intermediate data is usually large, the neural network processor will store the intermediate data calculated by the previous layer of the neural network into external memory. When executing the next layer of the neural network calculation, the intermediate data will be read from the memory and loaded into the neural network processor for calculation.

[0065] (4) Weighted data

[0066] In this embodiment, weight data refers to the weight data obtained after training the neural network. Specifically, the training process of the neural network is a process of continuously adjusting the weights.

[0067] (5) Intermediate calculation results

[0068] In this embodiment, intermediate calculation results refer to the calculation results generated by the neural network processor during the execution of neural network operations. These results can be data output by the computational units in the neural network processor, or data obtained by the vector computational unit performing further computation on the data output by the computational units. It should be noted that intermediate calculation results are also a type of input data; intermediate calculation results from the previous layer of the neural network are often used as input data in the calculation of the next layer.

[0069] The following describes, for example, the application scenarios in which the convolutional neural network 100 can be applied.

[0070] First application scenario:

[0071] In this embodiment, the convolutional neural network 100 can be applied to various electronic devices. In a specific implementation scenario, such as... Figure 3 As shown, smartphones 302 and 304 have built-in processors related to the convolutional neural network 100. When mobile smartphone client 301 initiates a voice call to mobile smartphone client 305, the voice signal is transmitted via smartphone 302 and relayed to smartphone 304 via base station 303. Due to a sudden downpour accompanied by strong thunder and lightning during the voice call, the input signal 306 is severely weakened and contains significant noise. This input signal can be a one-dimensional digital voice signal. Smartphone 304 is equipped with the convolutional neural network 100, which can be implemented in a chip as a dedicated circuit or as program instructions running in a central processing unit (CPU) or other processor. The input signal 306 is processed in the convolutional neural network in smartphone 304, including noise removal and effective signal enhancement, resulting in an output signal 307. This output signal completely preserves the voice information transmitted by the calling user, avoiding interference from the harsh natural environment.

[0072] Second application scenario:

[0073] This application provides another specific implementation scenario for the convolutional neural network 100, such as... Figure 4 As shown, a car 403 is traveling at high speed on a road. A passerby 401 uses a digital camera 402 to photograph the license plate number of the car 403. However, due to the high speed v of the car 403, motion blur occurs in the input signal 404 of the digital camera. This input signal is a two-dimensional digital image signal. The digital camera 402 is equipped with a convolutional neural network 100, which can be implemented in a dedicated circuit chip or as a software module running in an image signal processor. After the input signal 404 is processed in the convolutional neural network of the digital camera 402, including car motion model estimation and motion blur removal, an output signal 405 is obtained. The clarity of the license plate information contained in the output signal is improved, and accurate identification is possible.

[0074] As mentioned earlier, convolutional neural networks (CNNs), widely used in image recognition, audio recognition, and other fields, often require a large number of matrix multiplication operations. Performing matrix multiplication requires very high storage bandwidth and involves a large amount of computation. In existing technologies, when a neural network processor determines that intermediate computation results do not need to be compressed, it directly stores the intermediate computation results in external memory and uses these results as input data in the next layer of the neural network. In this case, the neural network processor does not need to decompress the intermediate computation results. When a neural network processor determines that intermediate computation results need to be compressed, it often compresses all of the intermediate computation results and then stores the compressed intermediate computation results in external memory. When performing the next layer of the neural network, the compressed intermediate computation results need to be decompressed to obtain the decompressed data, which is then used in the computation. This application differs from these methods. To fully utilize the hardware's processing power, in this embodiment, the intermediate calculation results obtained from the previous layer are processed to obtain compressed and uncompressed data. These compressed and uncompressed data are then stored in external memory. When performing calculations in the next layer, the processor first retrieves the data to be decompressed from the external memory and then performs a decompression operation. For example, the compressed data in the decompressed data is decompressed, while the uncompressed data is not processed. The data processed by the decompression unit participates in the calculation of the next layer of the neural network. Compared to existing technologies, since it is not necessary to compress all intermediate calculation results, but only a portion of the data in the intermediate calculation results, on the one hand, the amount of data transferred from the neural network processor to external memory is reduced, saving the I / O bandwidth of the neural network processor; on the other hand, the complexity of the compression operation is reduced. Furthermore, since the decompression unit only needs to decompress the compressed data and does not need to process the uncompressed data, the complexity of the decompression operation is reduced.

[0075] Figure 5a This is a hardware structure diagram of a chip provided in an embodiment of this application. The chip may include a Neural Network Processing Unit (NPU) 50, also known as a neural network processor 50, used to implement the computational functions of a convolutional neural network 100. The neural network processor in this embodiment can be applied to various devices capable of performing matrix multiplication operations, such as mobile phones, tablets, servers, and wearable devices.

[0076] In this embodiment, the artificial intelligence processor 50 can be an NPU, Google's Tensor Processing Unit (TPU), or a Graphics Processing Unit (GPU), or any processor suitable for large-scale XOR operations. Taking an NPU as an example: the NPU can be mounted as a coprocessor on the host CPU, with the host CPU assigning tasks to it.

[0077] In this embodiment, the core of the NPU is the arithmetic circuit 50, which is controlled by the controller 504 to extract matrix data from the memory and perform multiplication operations. For example, the controller 504 controls the arithmetic circuit 503 to extract data from the input memory 501 or the weight memory 502 and perform operations.

[0078] In some implementations, the arithmetic circuit (also called the arithmetic unit) 503 internally includes multiple processing units (PEs). In some implementations, the arithmetic circuit 503 is a two-dimensional pulsating array. The arithmetic circuit 503 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 503 is a general-purpose matrix processor.

[0079] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 502 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 501 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 508.

[0080] Unified memory 506 is used to store input and output data. Weight data is directly accessed via the Direct Memory Access Controller (DMAC) 505 and moved to weight memory 502. Input data is also moved to unified memory 506 via DMAC.

[0081] BIU stands for Bus Interface Unit 510, which is used for interaction between the AXI bus and the DMAC and the instruction fetch buffer 509.

[0082] The Bus Interface Unit (BIU) 510 is used by the instruction fetch memory 509 to fetch instructions from external memory, and also by the memory access controller 505 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0083] The DMAC is mainly used to move input data from external memory DDR to unified memory 506, or to weight data to weight memory 502, or to input data to input memory 501.

[0084] The vector computation unit 507 can further process the output of the arithmetic circuit 503, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for computation in non-convolutional / FC layer networks of neural networks, such as pooling, batch normalization, local response normalization, etc.

[0085] In some implementations, the vector computation unit 507 can store the processed output vector into a unified memory 506. For example, the vector computation unit 507 can apply a nonlinear function to the output of the arithmetic circuit 503, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 507 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as an activation input to the arithmetic circuit 503, for example, for use in subsequent layers of a neural network.

[0086] The instruction fetch buffer 509 connected to the controller 504 is used to store the instructions used by the controller 504;

[0087] The controller 504 is used to retrieve instructions cached in the instruction fetch memory to control the operation of the neural network processor 50.

[0088] Generally, the unified memory 506, input memory 501, weighted memory 502, and instruction fetch memory 509 are all on-chip memories. External memory is proprietary to this NPU hardware architecture. For example, external memory can be Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), High Bandwidth Memory (HBM), or other readable and writable memory.

[0089] In this embodiment, the input memory 501 is the first cache unit, the weight memory 502 is the second cache unit, the instruction fetch memory 509 is the third cache unit, and the unified memory 506 is the fourth cache unit.

[0090] When the aforementioned neural network processor performs convolution and full-computation (FC) operations, the amount of intermediate calculation results involved in the operation is large and cannot all be stored in the unified memory 506. Therefore, during the execution of the operation, the neural network processor needs to import the intermediate calculation results from the external memory in real time. Both exporting and importing intermediate calculation results will occupy the input / output (I / O) bandwidth of the neural network processor. If the I / O bandwidth becomes a bottleneck, the computational function of the neural network processor will be idle, reducing the computational performance of the neural network processor.

[0091] It should be noted that, Figure 1 and Figure 2 The operations of each layer in the convolutional neural network shown can be performed by matrix computation unit 212 or vector computation unit 507.

[0092] Figure 5b This is a schematic diagram of the hardware structure of a neural network processor 50 provided in an embodiment of this application, compared to... Figure 5a The neural network processor 50 shown here has an added compression unit 511 and a decompression unit 512.

[0093] The system includes an input memory 501 for storing input data, a weight memory 502 for storing weight data, and an arithmetic circuit 503 for performing matrix multiplication on the input data obtained from the input memory 501 and the weight data obtained from the weight memory 502 to obtain intermediate calculation results. These intermediate calculation results are stored in an accumulator 508. The vector calculation unit 507 can retrieve the intermediate calculation results from the accumulator 508 for further processing, such as vector multiplication, vector addition, exponential operation, logarithmic operation, and size comparison. The vector calculation unit 507 then stores the processed intermediate calculation results in a unified memory 506.

[0094] Compression unit 511 is used to obtain intermediate calculation results from unified memory 506 and process the intermediate calculation results to obtain processed data. The processed data includes compressed data and uncompressed data. Then, memory access controller 505 transports the data processed by compression unit 511 to external memory.

[0095] Furthermore, since the processed data is the result of calculations performed by the neural network processor 50 on a certain layer of the neural network, the processed data can be used as input data in the next calculation of the neural network processor 50. Therefore, the neural network processor 50 may also include a decompression unit 512, used to retrieve the data processed by the compression unit 511 from the external memory through the storage unit access controller 505, decompress the compressed data, and store the decompressed data and uncompressed data as input data in the input memory 501. The arithmetic circuit 503 will perform matrix multiplication calculations based on the input data retrieved from the input memory 501 and the weight data retrieved from the weight memory 502.

[0096] As can be seen from the above, since the neural network processor 50 adds a compression unit 511 and a decompression unit 512, specifically, the compression unit 511 can process the intermediate calculation results obtained from the previous layer to obtain processed data. The processed data includes compressed data and uncompressed data. The storage unit access controller 505 transfers the processed data from the compression unit 511 to the external memory. When performing calculations for the next layer, the storage unit access controller 505 first retrieves the processed data from the external memory, and then processes it through the decompression unit 512. For example, it decompresses the compressed data and decompresses the uncompressed data. The data is not decompressed, and the decompressed data and the undecompressed data are stored in the input memory 501. Thus, the arithmetic unit 503 can obtain the second input data from the input memory 501 to perform the next layer of calculation. Compared with the prior art, since it is not necessary to compress all the intermediate calculation results, but only some of the data in the intermediate calculation results, on the one hand, the amount of data transferred from the neural network processor to the external memory is reduced, saving the IO bandwidth of the neural network processor 50, so that the arithmetic units in the neural network processor do not have idle phenomena, and the computing performance of the processor is improved; on the other hand, the complexity of compression and decompression operations is reduced.

[0097] like Figure 6a As shown, the arithmetic unit (also called arithmetic circuit) 503 may include one or more matrix multiplication circuits 5031 and one or more addition circuits 5032. One addition circuit 5032 may correspond to one matrix multiplication circuit 5031, or one addition circuit 5032 may correspond to multiple matrix multiplication circuits 5031. The multiple matrix multiplication circuits 5031 included in the arithmetic unit 503 can each perform matrix multiplication operations independently. See also... Figure 6a , Figure 6aThe diagram uses the arithmetic unit 503, which includes two matrix multiplication circuits 5031, as an example. The matrix multiplication circuit 5031 includes M arithmetic groups composed of arithmetic blocks. Each arithmetic group includes K arithmetic blocks, and each arithmetic block includes N arithmetic units. Each arithmetic unit has two inputs, used to receive data sent from different memories (such as input memory 501 and weight memory 502), and multiply the two inputs. The addition circuit 5032 is used to accumulate the calculation results of arithmetic units belonging to the same arithmetic block to obtain the calculation result of each arithmetic block.

[0098] Figure 6b The structure of vector computation unit 507 is as follows: vector computation unit 507 is used to generate normalized values, merged values, or both. The processed output vector can be used as the activation input to computation unit 503, for example, for use in subsequent layers in a neural network. It mainly includes activation circuits, normalization circuits, and pooling circuits.

[0099] The activation circuit 5071 applies a nonlinear function to each accumulated value to generate an activation value, such as the nonlinear hyperbolic function tanh(x).

[0100] The normalization circuit 5072 generates normalized values ​​based on the activation values;

[0101] The pooling circuit 5073 applies an aggregation function to normalized values ​​to generate pooled values. For example, in some implementations, the aggregation function is a function that returns the maximum, minimum, or average value of the set of normalized values.

[0102] In the embodiments of this application, in Figure 5b In the neural network processor shown, the instruction controller 504 parses the control instructions stored in the instruction fetch memory 509. Based on the parsing result, it performs different control operations on the compression unit 511, the unified memory 506, the decompression unit 512, and the memory access controller 505. In some possible implementations, the instruction controller 504 may not perform the parsing of the control instructions and instead delegate the parsing of the control instructions to the compression unit 511 and the decompression unit 512 in the neural network processor 50.

[0103] In one example, the instruction fetch memory 509 stores control instructions (e.g., compression instructions). Specifically, these control instructions instruct the compression unit 511 to process data 1, where the starting address of data 1 is A and the data length is M. The instruction controller 504 sends this control instruction to the compression unit 511. The compression unit 511 then parses the control instruction, processes the intermediate calculation results, and obtains the processed data. The processed data includes compressed and uncompressed data. Finally, the memory access controller 505 transfers the processed data to the memory 20 outside the neural network processor.

[0104] In one example, the instruction fetch memory 509 stores control instructions (e.g., decompression instructions). Specifically, these control instructions instruct the decompression unit 512 to process data 2, which has a starting address of B and a length of N in the memory. The instruction controller 504 sends this control instruction to the decompression unit 512. The decompression unit 512 then parses the control instruction, receives the data to be decompressed from the memory by the memory access controller, decompresses the compressed data, leaves the uncompressed data unprocessed, and stores the decompressed and uncompressed data as second input data in the first cache unit (i.e., input memory 501).

[0105] based on Figure 5b The hardware structure shown below will be explained in detail below, including how the compression unit 511 performs the compression function and how the decompression unit 512 performs the decompression function in the embodiments of this application.

[0106] In this embodiment, the compression unit 511 is used to divide the intermediate calculation result into multiple blocks based on the starting address, data length, and predetermined block length corresponding to the intermediate calculation result. Then, if the data length of a block is equal to the block length, the block is compressed; if the data length of a block is less than the block length, the block is not compressed. When compressing the block data, each block can be compressed using an independent compression algorithm.

[0107] It should be noted that this application does not specifically limit the compression algorithm used by the compression unit 511. For example, the compression unit 511 can compress the data based on the value of 0. Since the compression unit 511 runs on the hardware logic circuit of the neural network processor 50, the selection of the compression algorithm needs to take into account the hardware resources, power consumption and performance. This application does not limit which compression algorithm the compression unit 511 uses.

[0108] Specifically, when the starting address corresponding to the intermediate calculation result is not an integer multiple of the block length, compression unit 511 divides the intermediate calculation result from the starting address to the first block address into blocks to obtain the first block data. Starting from the second block address, it then divides the intermediate calculation result into blocks according to the block length and compresses them to obtain the processed data. Here, the second block address is an address that satisfies an integer multiple of the block length; the first block address is the address preceding the second block address; the first block data is not compressed; and when the data length of the last block is less than the block length, the last block data is not compressed. When the starting address corresponding to the intermediate calculation result is an integer multiple of the block length, compression unit 511 divides the intermediate calculation result into blocks according to the block length starting from the starting address and compresses them to obtain the processed data; and when the data length of the last block is less than the block length, the last block data is not compressed.

[0109] The following examples illustrate this point:

[0110] In one example, the starting address A of the intermediate calculation result is 19kb, the data length M is 13kb, and the block length is 4kb. Since the starting address A is not an integer multiple of the block length L, the data corresponding to the starting address 19kb is designated as block 1. Starting from address 20kb, the remaining intermediate calculation results are divided into blocks according to the block length, resulting in blocks 2, 3, and 4. Block 2 includes data between addresses 20kb and 23kb, block 3 includes data between addresses 24kb and 27kb, and block 4 includes data between addresses 28kb and 31kb. For example, this block division result can be as follows: Figure 7a As shown. Since the data length of block 1 is less than the block length L, block 1 is not compressed. Since the data lengths of blocks 2, 3, and 4 are each equal to the block length L, compression unit 511 compresses these three blocks respectively, resulting in three compressed block data. For example, as... Figure 7b As shown, block 1 is not compressed, block 2 is compressed to 3kb, block 3 is compressed to 1kb, and block 4 is compressed to 2.5kb. Since the data length of block 5 is less than the block length L, block 5 is not compressed.

[0111] In one example, the starting address A = 19kb for the intermediate calculation result, the data length M = 7kb, and the block length L = 4kb. Since the starting address A is not an integer multiple of the block length L, the data corresponding to the starting address 19kb is designated as block 1, the data between addresses 20kb and 23kb is designated as block 2, and the data between addresses 24kb and 25kb is designated as block 3. Because the data lengths of blocks 1 and 3 are less than the block length L, these two blocks are not compressed. Since the data length of block 2 is equal to the block length L, block 2 is compressed.

[0112] In one example, the starting address corresponding to the intermediate calculation result is A = 0kb, the data length is M = 19kb, and the block length is L = 4kb. The starting address A is an integer multiple of the block length L. Starting from this starting address, the intermediate calculation result is divided into blocks according to the block length L, resulting in block 1, block 2, block 3, block 4, and block 5. Block 1 includes data from address 0kb to address 3kb, block 2 includes data from address 4kb to address 7kb, block 3 includes data from address 8kb to address 11kb, block 4 includes data from address 12kb to address 15kb, and block 5 includes data from address 16kb to address 18kb. Since the data length corresponding to each of block 1, block 2, block 3, and block 4 is equal to the block length, the compression unit 511 compresses each of the four blocks to obtain four compressed block data. For example, as... Figure 7c As shown, block 1 is compressed to 2kb of data, block 2 to 3kb of data, block 3 to 1kb of data, and block 4 to 2.5kb of data. Since the data length of block 5 is less than the block length L, block 5 is not compressed.

[0113] In this embodiment of the application, when storing data processed by the compression unit 511, the memory can store the data according to a preset alignment method, for example, the preset alignment method can be a byte alignment method. Figure 7d As shown, the four storage spaces 1 through 4 are byte aligned. When storing data, block 1 is stored in storage space 1, block 2 is stored in storage space 2, block 3 is stored in storage space 3, and block 4 is stored in storage space 4.

[0114] In this embodiment of the application, taking the example of the first intermediate calculation result with a starting address of A = 0kb, a data length of M = 19kb, and a block length of L = 4kb, and the second intermediate calculation result with a starting address of A = 19kb, a data length of M = 13kb, and a block length of 4kb, the first intermediate calculation result is the intermediate calculation result of the previous layer, and the second intermediate calculation result is the intermediate calculation result of the current layer. When the compression unit 511 processes the first intermediate calculation result, it compresses the four blocks from block 1 to block 4, but does not compress block 5. When the compression unit 511 processes the second intermediate calculation result, it does not compress block 1, but compresses the three blocks from block 2 to block 4. At this time, the uncompressed data is block 5 in the first intermediate calculation result (that is, the first data, which is the last block data of the first intermediate calculation result) and block 1 in the second intermediate calculation result (that is, the second data, which is the first block data of the second intermediate calculation result). When the memory stores the above-mentioned first data and second data, such as Figure 7e As shown, the first and second data are stored in the same memory address. This implementation saves memory space.

[0115] Specifically, the last data block mentioned above consists of the data between the first address (16kb) and the end address (18kb) of the first intermediate calculation result, where the first address is an address that is an integer multiple of the block length L. The first data block consists of the data between the starting address (19kb) and the second address of the second intermediate calculation result, where the second address is the address preceding the third address, and the third address is an address that is an integer multiple of the block length.

[0116] In this embodiment, for compressed block data, a first identifier corresponding to the compressed block data is generated. This first identifier is used to characterize whether the block data compression was successful and the length of the compressed data. For uncompressed block data, a second identifier corresponding to the uncompressed block data is generated. This second identifier is used to characterize whether the block data compression failed. The identifiers for successful and failed compression are stored in the memory via the storage unit access controller 505.

[0117] As can be seen from the above, the data processed by the compression unit 511 may result in either successful compression or compression failure. For the decompression unit 512, after receiving the data to be decompressed from the memory by the memory access controller, it identifies the identifiers corresponding to each block of data. When the identifier corresponding to the block of data is a compression failure identifier, the block of data is directly stored as input data in the input memory 501 without decompression processing. When the identifier corresponding to the block of data is a compression failure identifier, the block of data is decompressed to obtain the decompressed data, and the decompressed data is stored as input data in the input memory 501.

[0118] It should be noted that the decompression algorithm used in the decompression unit 512 is the same as the compression algorithm used in the compression unit 511.

[0119] In this embodiment of the application, when the storage unit access controller 505 moves the data stored in the external memory DDR to the decompression unit 512, the data can be processed by the decompression unit 512 to realize the calculation of the next layer of neural network.

[0120] Specifically, when the starting address of the data to be decompressed is not an integer multiple of the block length, the decompression unit 512 obtains the second block of data between the first alignment address and the second alignment address, obtains block data according to the block length starting from the second alignment address, and decompresses the block data; the first alignment address and the second alignment address are addresses that satisfy the integer multiple of the block length; the first alignment address is an alignment address before the starting address; the second alignment address is an alignment address after the starting address; wherein, the second block of data is not decompressed, and data from the starting address to the second alignment address is obtained; in the last block, data between the alignment address corresponding to the last block and the termination address corresponding to the data to be decompressed is obtained. When the starting address of the data to be decompressed is an integer multiple of the block length, the decompression unit 512 obtains the data between the starting address and the third alignment address, obtains block data according to the block length starting from the starting address, and decompresses the block data; the third alignment address is an address that satisfies the integer multiple of the block length; in the last block, data between the alignment address corresponding to the last block and the termination address corresponding to the data to be decompressed is obtained.

[0121] The following examples illustrate this point:

[0122] In one example, the starting address B of the data to be decompressed is 1kb, the data length N is 7kb, and the block length L is 4kb. The starting address is not an integer multiple of the block length L. The first block of data (0kb-3kb) is obtained from the first aligned address (0kb) to the second aligned address (4kb). Starting from the second aligned address (4kb), the second block of data (4kb-7kb) is obtained from address 4kb to address 7kb. Within the first block, data after the starting address (1kb) (1kb-3kb) is directly obtained. The second block is then decompressed to obtain data of length 4kb. Thus, the decompression unit 512 can decompress the data and store the decompressed and uncompressed data as input data in the input memory 501.

[0123] In one example, the starting address B of the data to be decompressed is 1kb, the data length N is 10kb, and the block length L is 4kb. The starting address is not an integer multiple of the block length L. The first block of data (0kb-3kb) is retrieved from the first aligned address (0kb) to the second aligned address (4kb). Starting from the second aligned address (4kb), the second block of data (4kb-7kb) is retrieved from the address between 4kb and 7kb. The third block of data (8kb-11kb) is retrieved from the address between 8kb and the third aligned address (11kb). In the first data block, the data after the starting address (1kb) is directly obtained (i.e., 1kb-3kb); in the second data block, the second data block is decompressed to obtain data with a length of 4kb; in the third data block, the data between the address 8kb and the ending address (10kb) is directly obtained (i.e., 8kb-10kb). Thus, the decompression unit 512 can decompress the data and store the decompressed data and uncompressed data as input data in the input memory 501.

[0124] In one example, the starting address B of the data to be decompressed is 0kb, and the data length is 9kb. Data is obtained in blocks according to the block length, starting from the starting address. For example, blocks 1 (including data from address 0kb to address 3kb), blocks 2 (including data from address 4kb to address 7kb), and blocks 3 (including data from address 8kb to address 11kb) are decompressed. Blocks 1 and 2 are decompressed respectively to obtain the decompressed data. In block 3, the data corresponding to address 8kb is obtained. Thus, the decompression unit 512 can decompress the data and store the decompressed data and uncompressed data as input data in the input memory 501.

[0125] It should be noted that the examples above are merely illustrative and should not be construed as limiting factors. For instance, the block length L can be 2kb, 8kb, or other values.

[0126] In implementing embodiments of this application, a compression unit and a decompression unit are added to the neural network processor. When processing intermediate calculation results obtained by the computation unit, the compression unit can obtain both compressed and uncompressed data. Then, the processed data is stored in external memory via a memory access controller. Compared to existing technologies, since it is not necessary to compress all intermediate calculation results, but only a portion of the data, the amount of data transferred from the neural network processor to external memory is reduced, saving the I / O bandwidth of the neural network processor. Furthermore, the complexity of the compression operation is reduced. In addition, since the decompression unit only needs to decompress the compressed data and does not need to process the uncompressed data, the complexity of the decompression operation is also reduced.

[0127] It should be noted that, taking convolutional neural networks as an example, since the length of feature data generated in convolutional neural networks is largely irregular, the length of feature data used when the next layer performs calculations is also irregular. The method described in this application, which processes data through compression unit 511 to obtain compressed and uncompressed data, and processes the data to be decompressed through decompression unit 512, can support compression and decompression at any starting position and with any data length, thus improving the versatility of data compression and decompression.

[0128] based on Figure 5b The hardware structure diagram shown below, in conjunction with... Figure 8a The flowchart shown in this embodiment illustrates a data processing method, specifically explaining how data processing is implemented in this embodiment. This method is applied to a neural network processor, which includes a first cache unit, a second cache unit, a computation unit, a compression unit, and a storage unit access controller. The method may include, but is not limited to, the following steps:

[0129] Step S800 involves performing matrix multiplication on the first input data obtained from the first cache unit and the weight data obtained from the second cache unit through the arithmetic unit to obtain intermediate calculation results.

[0130] Step S802: The intermediate calculation results are processed by the compression unit to obtain the processed data; the processed data includes compressed data and uncompressed data.

[0131] Step S804: The processed data is stored in a memory outside the neural network processor via the storage unit access controller.

[0132] In the embodiments of this application, please refer to the foregoing description for the specific implementation of steps S800-S804, which will not be repeated here.

[0133] exist Figure 8a Based on the method flowchart shown, as Figure 8b As shown in the embodiments of this application, another data processing method is also provided. This method describes how to decompress data stored in external memory (DDR) to realize neural network computation. This method may include, but is not limited to, the following steps:

[0134] Step S806: The decompression unit receives the data to be decompressed from the memory by the memory access controller, decompresses the compressed data in the data to be decompressed, and stores the decompressed data and the uncompressed data as the second input data in the first cache unit.

[0135] Step S808: Perform matrix multiplication on the second input data obtained from the first cache unit through the arithmetic unit.

[0136] In the embodiments of this application, please refer to the foregoing description for the specific implementation of steps S806-S808, which will not be repeated here.

[0137] In implementing the embodiments of this application, a compression unit and a decompression unit are added to the neural network processor. When processing the intermediate calculation results obtained by the computation unit, the compression unit can obtain both compressed and uncompressed data. Then, the processed data is stored in external memory through the memory access controller. Compared to the prior art, since it is not necessary to compress all intermediate calculation results, but only a portion of the data in the intermediate calculation results, on the one hand, the amount of data transferred from the neural network processor to external memory is reduced, saving the I / O bandwidth of the neural network processor; on the other hand, the complexity of the compression operation is reduced. Furthermore, since the decompression unit only needs to decompress the compressed data and does not need to process the uncompressed data, the complexity of the decompression operation is reduced.

[0138] by Figure 5b For example, Figure 5b In addition to providing the structure of the neural network processor 50, the structure of the main CPU 10 is also given. Based on this, embodiments of this application also provide a method for controlling the neural network processor to perform compression, such as... Figure 9As shown, the method may include, but is not limited to, steps S901-S909 as shown in the figure below, wherein steps S901-S904 are executed by the main CPU, and steps S905-S909 are executed by the neural network processor.

[0139] Step S901: Determine whether to compress and decompress the intermediate calculation results of each layer in the neural network.

[0140] In practical applications, the CPU can analyze the characteristics of the neural network formed after training (this trained neural network is the neural network that the neural network processor performs inference operations on). For example, it can analyze the data through algorithms or actual test data, and then infer the sparsity rate of the input data of each layer in the neural network process based on the characteristics of the neural network obtained from the analysis. Based on the sparsity rate of the input data of each layer, it can determine whether to compress and decompress the intermediate calculation results of that layer. Specifically, it can determine whether to compress and decompress the intermediate calculation results of that layer by comparing the sparsity rate of the input data of each layer with a threshold. For example, when the CPU determines that the sparsity of the input data of the i-th layer of the neural network is greater than a threshold, it determines that the intermediate calculation result of the i-th layer needs to be processed by the compression unit 511, and that the decompression unit 512 needs to decompress the intermediate calculation result of the i-th layer as the input data of the (i+1)-th layer to perform the calculation; when the CPU determines that the sparsity of the input data of the i-th layer of the neural network is not greater than a threshold, it determines that the intermediate calculation result of the i-th layer does not need to be processed by the compression unit 511, and that the intermediate calculation result of the i-th layer does not need to be decompressed when used as the input data of the (i+1)-th layer to perform the calculation; where i is an integer greater than 1 and less than N, and N is the number of layers in the neural network.

[0141] The above thresholds can be determined based on the benefits of I / O bandwidth and the power consumption cost. The benefits of I / O bandwidth refer to the reduction of I / O bandwidth by the neural network processor in compressing and decompressing intermediate calculation results, while the power consumption cost refers to the increase in power consumption by the neural network processor in compressing and decompressing intermediate calculation results.

[0142] Specifically, this threshold can be predetermined. For example, in a preliminary test, when the sparsity of the input data equals a critical value, the gain in I / O bandwidth from enabling compression and decompression by the neural network processor 50 equals the power consumption cost, then this critical value can be determined as the threshold. In practical applications, considering the desire for greater I / O bandwidth gains, this critical value can be adjusted to determine the size of the threshold. This application does not specifically limit the determination of the threshold.

[0143] It should be noted that the preset thresholds mentioned above may vary for different neural network models.

[0144] Step S902: Generate control commands.

[0145] Specifically, the CPU can generate control instructions based on the above judgment results. These control instructions are used to instruct the neural network processor 50 whether to compress and decompress the intermediate calculation results obtained from each layer of the neural network.

[0146] Step S903: Store the control instructions in the memory.

[0147] Specifically, the CPU can store the generated control instructions in memory 20 outside the neural network processor.

[0148] Step S904: Place the control instructions into the value memory in the neural network processor.

[0149] Specifically, the CPU can move control instructions stored in memory to the instruction fetch memory 509 in the neural network processor 50.

[0150] Step S905: Read the control instructions from the value retrieval memory.

[0151] Specifically, the neural network processor 50 reads control instructions from the instruction fetch memory 509.

[0152] Step S906: Perform calculations on each layer of the neural network to obtain intermediate calculation results.

[0153] Specifically, the computation can be performed through the arithmetic circuit 503 in the neural network processor 50.

[0154] Step S907: Determine whether to compress and decompress the intermediate calculation results according to the control command.

[0155] Specifically, after receiving the control instruction, the neural network processor parses the control instruction and then determines whether to compress and decompress the intermediate calculation results of each layer in the neural network according to the control instruction. If yes, step S908 is executed; otherwise, step S909 is executed.

[0156] Step S908: The intermediate calculation results are processed by the compression unit to obtain processed data; the processed data includes compressed data and uncompressed data; the processed data is stored in the memory; when the next layer of neural network calculation is executed, the intermediate calculation results obtained from the memory are decompressed to obtain the input data.

[0157] Specifically, when the neural network processor determines that the intermediate calculation results need to be compressed, it processes the intermediate calculation results through the compression unit 511 to obtain the processed data and stores the processed data in the memory. When executing the next layer of the neural network calculation, the neural network processor retrieves the intermediate calculation results from the memory, decompresses them through the decompression unit 512, and uses the decompressed data and the uncompressed data as input data to participate in the next layer of the neural network calculation.

[0158] Step S909: Store the intermediate calculation results in the memory. When performing the next layer of neural network calculation, the intermediate calculation results obtained from the memory will be used as input data.

[0159] Specifically, when the neural network processor determines that there is no need to compress the intermediate calculation results, it directly stores the intermediate calculation results in the memory. When executing the next layer of the neural network calculation, the neural network processor reads the intermediate calculation results from the memory and uses the obtained intermediate calculation results as input data to participate in the next layer of the neural network calculation. At this time, the intermediate calculation results obtained by the neural network processor do not need to be processed by the decompression unit 512.

[0160] Therefore, it can be seen that for input data with low sparsity in neural networks, the compression benefit is low if the neural network processor initiates compression and decompression. Using the method described in this application, the CPU only controls the neural network processor to compress and decompress the intermediate calculation results obtained from a certain layer of the neural network when it determines that the sparsity of the input data in that layer is high, which can improve the compression benefit of I / O bandwidth.

[0161] It should be noted that the steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0162] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0163] It is understood that those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in the various embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0164] Those skilled in the art will appreciate that the functionality described by the various illustrative logic blocks, modules, and algorithmic steps disclosed in connection with the various embodiments of this application can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality described by the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may comprise a computer-readable storage medium, which corresponds to a tangible medium, such as a data storage medium, or a communication medium that includes any medium that facilitates the transfer of a computer program from one place to another (e.g., according to a communication protocol). In this way, the computer-readable medium may substantially correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium, such as a signal or carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this application. The computer program product may comprise a computer-readable medium.

[0165] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0166] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0167] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0168] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0169] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0170] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A neural network processor, characterized in that, include: The first buffer unit is used to store the first input data; The second cache unit is used to store weight data; The arithmetic unit connected to the first cache unit and the second cache unit is used to perform matrix multiplication on the first input data and the weight data to obtain intermediate calculation results; The compression unit is used to divide and compress the intermediate calculation result into blocks according to the starting address, data length, and determined block length of the intermediate calculation result to obtain processed data; the processed data includes compressed data and uncompressed data; the length of the uncompressed data is less than the block length. A storage unit access controller connected to the compression unit is used to store the processed data in a memory outside the neural network processor.

2. The neural network processor of claim 1, further comprising: The decompression unit connected to the storage unit access controller is used to receive the data to be decompressed obtained by the storage unit access controller from the memory, decompress the compressed data in the data to be decompressed, and store the decompressed data and the uncompressed data as the second input data in the first cache unit; The computation unit is used to obtain the second input data from the first cache unit to perform the matrix multiplication operation.

3. The neural network processor as described in claim 1 or 2, characterized in that, The uncompressed data includes first data and second data, and the first data and the second data are stored in the same storage space corresponding to the same starting address in the memory. Wherein, the first data is the last block of intermediate calculation results from the previous layer; the second data is the first block of intermediate calculation results from the current layer.

4. The neural network processor as described in claim 3, characterized in that, The last block of data is the data between the first address and the termination address of the intermediate calculation result of the previous layer, where the first address is an address that is an integer multiple of the block length; The first block of data is the data between the starting address of the intermediate calculation result of the current layer and the second address, where the second address is the address preceding the third address, and the third address is an address that is an integer multiple of the block length.

5. The neural network processor as described in claim 1, characterized in that, The compression unit is specifically used for: If the starting address corresponding to the intermediate calculation result is not an integer multiple of the block length, the intermediate calculation result from the starting address to the first block address is divided into blocks to obtain the first block data. Starting from the second block address, the intermediate calculation result is divided into blocks according to the block length and compressed to obtain the processed data. Wherein, the second block address is an address that is an integer multiple of the block length; the first block address is the address preceding the second block address; the first block data is not compressed; when the data length of the last block is less than the block length, the data of the last block is not compressed.

6. The neural network processor as described in claim 2, characterized in that, The data to be decompressed is stored in the memory according to a preset alignment; the decompression unit is specifically used for: If the starting address of the data to be decompressed is not an integer multiple of the block length, the second block of data between the first alignment address and the second alignment address is obtained, and the block data is obtained according to the block length starting from the second alignment address, and the block data is decompressed; the first alignment address and the second alignment address are addresses that satisfy the integer multiple of the block length; The first alignment address is an alignment address preceding the starting address; the second alignment address is an alignment address following the starting address; In this process, the second block of data is not decompressed, and the data after the starting address is obtained; in the last block, the data between the alignment address corresponding to the last block and the termination address corresponding to the data to be decompressed is obtained.

7. The neural network processor as described in claim 6, characterized in that, The compressed data includes compressed block data, and the uncompressed data includes uncompressed block data; For the compressed block data, a first identifier information corresponding to the compressed block data is generated. The first identifier information is used to characterize the successful compression of the block data and the length of the compressed data. The first identifier information is controlled by the storage unit access controller and stored in the memory. For the uncompressed block data, a second identifier is generated corresponding to the uncompressed block data. The second identifier is used to indicate that the block data compression failed. The second identifier is controlled by the storage unit access controller and stored in the memory.

8. The neural network processor as described in claim 7, characterized in that, The decompression unit is specifically used for: Receive the data to be decompressed obtained from the memory by the storage unit access controller; When the identifier corresponding to the block data is the identifier of successful compression, the block data is decompressed, and the decompressed data is stored as the second input data in the first cache unit; When the identifier corresponding to the block data is a compression failure identifier, the block data is stored as the second input data in the first cache unit.

9. A data processing method, characterized in that, The method is applied to a neural network processor, which includes a first cache unit, a second cache unit, a computation unit, a compression unit, and a storage unit access controller. The method includes: The processing unit performs matrix multiplication on the first input data obtained from the first cache unit and the weight data obtained from the second cache unit to obtain intermediate calculation results. The compression unit divides and compresses the intermediate calculation results into blocks based on the starting address, data length, and predetermined block length to obtain processed data. The processed data includes compressed data and uncompressed data. The length of the uncompressed data is less than the block length. The processed data is stored in a memory outside the neural network processor via the storage unit access controller.

10. The method as described in claim 9, characterized in that, The neural network processor further includes a decompression unit, and the method further includes: The decompression unit receives the data to be decompressed obtained from the memory by the storage unit access controller, decompresses the compressed data in the data to be decompressed, and stores the decompressed data and uncompressed data as the second input data in the first cache unit. The arithmetic unit performs matrix multiplication on the second input data obtained from the first cache unit.

11. The method as described in claim 9 or 10, characterized in that, The uncompressed data includes first data and second data, and the first data and the second data are stored in the same storage space corresponding to the same starting address in the memory. Wherein, the first data is the last block of intermediate calculation results from the previous layer; the second data is the first block of intermediate calculation results from the current layer.

12. The method as described in claim 11, characterized in that, The last block of data is the data between the first address and the termination address of the intermediate calculation result of the previous layer, where the first address is an address that is an integer multiple of the block length; The first block of data is the data between the starting address of the intermediate calculation result of the current layer and the second address, where the second address is the address preceding the third address, and the third address is an address that is an integer multiple of the block length.

13. The method as described in claim 9, characterized in that, The intermediate calculation result is divided into blocks and compressed by the compression unit according to the starting address, data length, and determined block length of the intermediate calculation result to obtain the processed data, including: If the starting address corresponding to the intermediate calculation result is not an integer multiple of the block length, the intermediate calculation result from the starting address to the first block address is divided into blocks to obtain the first block data. Starting from the second block address, the intermediate calculation result is divided into blocks according to the block length and compressed to obtain the processed data. Wherein, the second block address is an address that is an integer multiple of the block length; the first block address is the address preceding the second block address; the first block data is not compressed; when the data length of the last block is less than the block length, the data of the last block is not compressed.

14. The method as described in claim 10, characterized in that, The data to be decompressed is stored in the memory according to a preset alignment; the decompression of the compressed data in the data to be decompressed includes: If the starting address of the data to be decompressed is not an integer multiple of the block length, the second block of data between the first alignment address and the second alignment address is obtained. Starting from the second alignment address, the block data is obtained according to the block length, and the block data is decompressed. The first alignment address and the second alignment address are addresses that satisfy an integer multiple of the block length. The first alignment address is an alignment address before the starting address. The second alignment address is an alignment address after the starting address. In this process, the second block of data is not decompressed, and the data after the starting address is obtained; in the last block, the data between the alignment address corresponding to the last block and the termination address corresponding to the data to be decompressed is obtained.

15. The method as described in claim 14, characterized in that, The compressed data includes compressed block data, and the uncompressed data includes uncompressed block data; the method further includes: For the compressed block data, a first identifier information corresponding to the compressed block data is generated. The first identifier information is used to characterize the successful compression of the block data and the length of the compressed data. The first identifier information is controlled by the storage unit access controller and stored in the memory. For the uncompressed block data, a second identifier is generated corresponding to the uncompressed block data. The second identifier is used to indicate that the block data compression failed. The second identifier is controlled by the storage unit access controller and stored in the memory.

16. The method as described in claim 15, characterized in that, The method further includes: Receive the data to be decompressed obtained from the memory by the storage unit access controller; When the identifier corresponding to the block data is the identifier of successful compression, the block data is decompressed, and the decompressed data is stored as the second input data in the first cache unit; When the identifier corresponding to the block data is a compression failure identifier, the block data is stored as the second input data in the first cache unit.

Images