Method, device and system for neural network processing

Abstract

A method, apparatus (100) and system (1000) for processing neural networks, the apparatus (100) comprising: a first computing array (10) configured to perform a first type of neural network operation; a second computing array (20) configured to perform a second type of neural network operation, the second type of neural network operation being different from the first type of neural network operation; and a control module (30) configured to control the first computing array (10) to perform the first type of neural network operation, and to control the second computing array (20) to perform the second type of neural network operation. By including multiple computing arrays configured to perform different types of operations in neural networks, acceleration of multiple types of operations in neural networks can be achieved, and thus the computational efficiency of deep neural networks can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, in particular to a neural network processing method, device and system. BACKGROUND

[0002] Deep neural network learning has good application prospects in the fields of image processing, pattern recognition, speech recognition, natural language, etc. Deep neural network learning needs to process a large amount of data, and as the development of deep neural network learning, the requirements for processors become higher and higher. Traditional processors are already difficult to meet the computing needs of deep learning, and special-purpose acceleration chips for neural networks have been proposed.

[0003] In the current technology, a tensor processing unit (TPU) is a special-purpose processor for neural networks that can accelerate the computation of neural networks. The TPU mainly adopts a matrix computation unit based on a systolic array. The systolic array can effectively accelerate convolution operations.

[0004] In addition to convolution operations, deep neural network learning also includes a variety of other types of operations, such as pooling or fully connected operations. It is necessary to accelerate the computation of various types of operations in neural networks. SUMMARY

[0005] The embodiments of the present disclosure provide a neural network processing method, device and system, which can accelerate the computation of various types of operations in neural networks.

[0006] In a first aspect, a neural network processing device is provided, which includes: a first computation array configured to perform a first type of neural network operation; a second computation array configured to perform a second type of neural network operation, the second type of neural network operation being different from the first type of neural network operation; and a control module configured to control the first computation array to perform the first type of neural network operation, and control the second computation array to perform the second type of neural network operation.

[0007] It should be understood that the neural network processing device provided by the embodiments of the present disclosure includes multiple computation arrays for performing different types of operations in neural networks, so that the computation of various types of operations in neural networks can be accelerated, and the computing efficiency of deep neural networks can be improved.

[0008] In some implementations, the control module is configured to control the first computation array and the second computation array to perform the first type of neural network operation and the second type of neural network operation in parallel.

[0009] It should be understood that the apparatus for processing neural network provided by the embodiments of the present disclosure can effectively improve the calculation efficiency of the deep neural network by performing multiple types of operations in the neural network in parallel.

[0010] In some implementations, the first type of neural network operation is a convolution operation, and the first calculation array is a 3D MAC array.

[0011] In some implementations, the first type of neural network operation is a convolution operation, and the first calculation array is a systolic array.

[0012] In some implementations, the second type of neural network operation includes vector operations.

[0013] The vector operation represents an operation of multi-dimensional data on multi-dimensional data.

[0014] The vector operation in the embodiments of the present disclosure includes, but is not limited to, pooling operations and full connection operations.

[0015] In some implementations, the apparatus further includes a first storage module configured to store weights required for the convolution operation, and a second storage module configured to store feature data required for the convolution operation, wherein the control module is configured to read the weights required for the convolution operation from the first storage module to the first calculation array, and read the feature data required for the convolution operation from the second storage module to the first calculation array.

[0016] Optionally, the control module is configured to read the weights and the feature data required for the convolution operation from the first storage module and the second storage module to the first calculation array in parallel.

[0017] It should be understood that the apparatus for processing neural network provided by the embodiments of the present disclosure can effectively improve the calculation efficiency of the deep neural network by performing multiple types of operations in the neural network in parallel.

[0018] In some implementations, the control module includes a first control unit configured to read the weights required for the convolution operation from the first storage module to the first calculation array, and a second control unit configured to read the feature data required for the convolution operation from the second storage module to the first calculation array.

[0019] In some implementations, the apparatus further includes a cache configured to cache the feature data required for the convolution operation read from the second storage module, wherein the control module is configured to read the feature data required for the convolution operation from the cache to the first calculation array.

[0020] In some embodiments, the control module is configured to: determine whether the cache memory has the feature data required by the first computing array for convolution operation; if the determination is positive, read the feature data required by the convolution operation from the cache memory to the first computing array; and if the determination is negative, read the required feature data from the second storage module to the cache memory, and read the feature data required by the convolution operation from the cache memory to the first computing array.

[0021] In some embodiments, the distance from the cache memory to the first computing array is less than the distance from the second storage module to the first computing array.

[0022] It should be understood that the apparatus for processing neural network provided by the embodiments of the present disclosure can improve the reading efficiency of the feature data required by the convolution operation by using the cache memory to cache the feature data required by the convolution operation, and can also reduce the power consumption of data reading.

[0023] In some embodiments, the second storage module is further configured to store data required by the second type of neural network operation; and the control module is further configured to read the data required by the second type of neural network operation from the second storage module to the second computing array.

[0024] It should be understood that the first computing array and the second computing array share one storage module, i.e., the second storage module, which can reduce the occupied space of the storage module.

[0025] In some embodiments, the apparatus further comprises: a third storage module configured to store data required by the first type of neural network operation; and a fourth storage module configured to store data required by the second type of neural network operation, wherein the fourth storage module is arranged separately from the third storage module; and the control module is configured to: control the first computing array to read the data required by the first type of neural network operation from the third storage module; and control the second computing array to read the data required by the second type of neural network operation from the fourth storage module.

[0026] It should be understood that the apparatus for processing neural network provided by the embodiments of the present disclosure can improve the reading efficiency of the feature data required by the convolution operation by using the cache memory to cache the feature data required by the convolution operation, and can also reduce the power consumption of data reading.

[0027] In some embodiments, the distance from the third storage module to the first computing array is less than the distance from the fourth storage module to the first computing array; and / or the distance from the fourth storage module to the second computing array is less than the distance from the third storage module to the second computing array.

[0028] In some implementations, the apparatus includes a plurality of second compute arrays. Each second compute array can support a plurality of types of operations.

[0029] In some implementations, the apparatus includes a plurality of second compute arrays configured to perform a same type of operation in a neural network. For example, the plurality of second compute arrays are configured to perform a pooling operation.

[0030] It should be understood that the apparatus provided by the embodiments of the present disclosure can perform the same type of operation through the plurality of second compute arrays, which can effectively improve the computing efficiency.

[0031] In some implementations, the apparatus is a tensor processing unit (TPU).

[0032] In a second aspect, an apparatus for neural network processing is provided. The apparatus includes a first compute array configured to perform a first type of neural network operation, the first compute array being a 3D MAC array, and the first type of neural network operation being a convolution operation; a first storage module configured to store weights required by the convolution operation; a second storage module, separate from the first storage module, configured to store feature data required by the convolution operation; and a control module configured to read the weights required by the convolution operation from the first storage module to the first compute array, and read the feature data required by the convolution operation from the second storage module to the first compute array.

[0033] Optionally, the second storage module is further configured to store a result of the computation of the first compute array.

[0034] It should be understood that the apparatus for neural network processing provided by the embodiments of the present disclosure can store the feature data and the weights required by the convolution operation separately, so that the feature data and the weights required by the convolution operation can be read from two read ports respectively, and the efficiency of data reading can be improved.

[0035] In some implementations, the control module is configured to read the weights and the feature data required by the convolution operation from the first storage module and the second storage module to the first compute array in parallel.

[0036] In some implementations, the apparatus further includes a cache configured to cache the feature data required by the convolution operation read from the second storage module; and wherein the control module is configured to read the feature data required by the convolution operation from the cache to the first compute array.

[0037] In some embodiments, the control module is configured to: determine whether the cache memory has the feature data required by the first computing array for performing the convolution operation; if the determination is positive, read the feature data required by the convolution operation from the cache memory to the first computing array; and if the determination is negative, read the required feature data from the second storage module to the cache memory, and read the feature data required by the convolution operation from the cache memory to the first computing array.

[0038] In some embodiments, the distance from the cache memory to the first computing array is less than the distance from the second storage module to the first computing array.

[0039] In a third aspect, a system for processing a neural network is provided, which includes the following units.

[0040] One or more processing units configured to perform the neural network processing apparatus of the first aspect or the second aspect. In the case where the system includes a plurality of processing units, the plurality of processing units include the neural network processing apparatus of the first aspect and / or the second aspect.

[0041] A storage unit configured to store data required by the one or more processing units for performing the neural network operation.

[0042] A data transfer unit configured to read out the data required by the neural network operation stored in the storage unit to the one or more processing units.

[0043] It should be understood that the system provided by the embodiments of the present disclosure can support multiple neural networks, and has good versatility. In addition, the system can perform the operation of multiple neural networks in parallel, which can improve the computing efficiency of the neural network.

[0044] In some embodiments, the system includes a plurality of processing units; wherein the plurality of processing units are configured to perform the neural network operation in parallel.

[0045] In some embodiments, the data transfer unit includes: one or more intermediate units corresponding to the one or more processing units one by one, wherein each intermediate unit is configured to be responsible for the transfer of data between the corresponding processing unit and the storage unit.

[0046] Here, the transfer includes reading data from the storage unit to the processing unit, or writing the processing result of the processing unit to the storage unit.

[0047] In some embodiments, the system includes a plurality of processing units; wherein the data transfer unit further includes: a broadcast unit configured to broadcast the data stored in the storage unit and shared by the plurality of processing units to the plurality of processing units in the case where the plurality of processing units are configured to perform the operation of the same neural network.

[0048] In some embodiments, the storage unit comprises any one or more of the following: a first memory and a second memory, the second memory having a read speed greater than that of the first memory.

[0049] In some embodiments, the first memory is an internal memory and the second memory is an L2 cache.

[0050] In some embodiments, the system comprises a plurality of processing units, wherein different processing units of the plurality of processing units are configured to perform operations of different neural networks.

[0051] In some embodiments, the system further comprises an interconnection structure for transmitting data or commands between the one or more processing units and the storage unit.

[0052] The system provided by the embodiments of the present disclosure comprises a plurality of processing units, which can run different neural networks or the same neural network at the same time. Such flexible configuration can improve the utilization rate of each processing unit and the overall computing capability of the system.

[0053] The fourth aspect provides a neural network processing method corresponding to the neural network processing apparatus provided by the first aspect, the fifth aspect provides a neural network processing method corresponding to the neural network processing apparatus provided by the second aspect, and the sixth aspect provides a neural network processing method corresponding to the neural network processing system provided by the third aspect. The content not described in detail can be referred to the apparatus embodiment of the first aspect, and for brevity, will not be described here.

[0054] The related explanations and beneficial effects of the neural network processing methods provided by the fourth aspect, the fifth aspect and the sixth aspect can be correspondingly referred to the descriptions in the first aspect, the second aspect and the third aspect, and will not be described here.

[0055] The fourth aspect provides a neural network processing method, which comprises: performing a first type of neural network operation through a first computing array; and performing a second type of neural network operation through a second computing array, the second type of neural network operation being different from the first type of neural network operation.

[0056] The method can be performed by the neural network processing apparatus provided by the first aspect.

[0057] For the descriptions of the first computing array and the second computing array, refer to the descriptions above, which will not be described here.

[0058] In some embodiments, the method further comprises: reading the weights required by the convolution operation from the first storage module to the first computing array, and reading the feature data required by the convolution operation from the second storage module to the first computing array, the second storage module being arranged separately from the first storage module.

[0059] Optionally, the method further comprises: reading the data required by the second type of neural network operation from the second storage module to the second computing array.

[0060] In some embodiments, reading the weights required by the convolution operation from the first storage module to the first computing array, and reading the feature data required by the convolution operation from the second storage module to the first computing array comprises: reading the weights and the feature data required by the convolution operation from the first storage module and the second storage module to the first computing array in parallel.

[0061] In some embodiments, reading the feature data required by the convolution operation from the second storage module to the first computing array comprises: reading the feature data required by the convolution operation from the cache memory to the first computing array, the cache memory being configured to cache the feature data required by the convolution operation read from the second storage module.

[0062] In some embodiments, reading the feature data required by the convolution operation from the cache memory to the first computing array comprises: determining whether the cache memory has the feature data required by the first computing array to perform the convolution operation; if the determination result is yes, reading the feature data required by the convolution operation from the cache memory to the first computing array; if the determination result is no, reading the required feature data from the second storage module to the cache memory, and reading the feature data required by the convolution operation from the cache memory to the first computing array.

[0063] In some embodiments, the distance from the cache memory to the first computing array is less than the distance from the second storage module to the first computing array.

[0064] In some embodiments, the method further comprises: reading the data required by the first type of neural network operation from a third storage module to the first computing array, and reading the data required by the second type of neural network operation from a fourth storage module to the second computing array, the fourth storage module being arranged separately from the third storage module.

[0065] Optionally, the distance from the third storage module to the first computing array is less than the distance from the fourth storage module to the first computing array; and / or the distance from the fourth storage module to the second computing array is less than the distance from the third storage module to the second computing array.

[0066] In the fifth aspect, a method for processing a neural network is provided. The method comprises: reading weights required for a convolution operation from a first storage module to a first computing array; reading feature data required for the convolution operation from a second storage module to the first computing array, the second storage module being separately arranged from the first storage module; and performing the convolution operation on the feature data and the weights by the first computing array.

[0067] The method can be performed by the apparatus for processing a neural network provided in the second aspect.

[0068] In some implementations, the reading of the weights required for the convolution operation from the first storage module to the first computing array and the reading of the feature data required for the convolution operation from the second storage module to the first computing array comprises: reading the weights and the feature data required for the convolution operation from the first storage module and the second storage module to the first computing array in parallel.

[0069] In some implementations, the reading of the feature data required for the convolution operation from the second storage module to the first computing array comprises: reading the feature data required for the convolution operation from a cache memory to the first computing array, the cache memory being configured to cache the feature data required for the convolution operation read from the second storage module.

[0070] In some implementations, the reading of the feature data required for the convolution operation from the cache memory to the first computing array comprises: determining whether the cache memory has the feature data required for the convolution operation by the first computing array; if the determination result is positive, reading the feature data required for the convolution operation from the cache memory to the first computing array; and if the determination result is negative, reading the required feature data from the second storage module to the cache memory and reading the feature data required for the convolution operation from the cache memory to the first computing array.

[0071] In some implementations, the cache memory is closer to the first computing array than the second storage module is to the first computing array.

[0072] In the sixth aspect, a method for processing a neural network is provided. The method comprises: reading data required for a neural network operation from a storage unit to one or more processing units, the processing units being the apparatus for processing a neural network provided in the first aspect or the second aspect; and performing the neural network operation by the one or more processing units respectively.

[0073] The method can be performed by the system for processing a neural network provided in the third aspect.

[0074] In some implementations, the performing of the neural network operation by the one or more processing units respectively comprises: performing a plurality of neural network operations in parallel by the plurality of processing units.

[0075] In some implementations, in a case where multiple processing units are used to perform operations of the same neural network, reading data required by the neural network operations from the storage unit to the one or more processing units includes broadcasting data stored in the storage unit that is shared by the multiple processing units to the multiple processing units.

[0076] In some implementations, different processing units of the multiple processing units are used to perform operations of different neural networks.

[0077] Based on the above description, the apparatus, system and method for neural network processing provided by the embodiments of the present disclosure can accelerate multiple types of operations in the neural network, thereby improving the computing efficiency of the deep neural network. BRIEF DESCRIPTION OF DRAWINGS

[0078] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, which are schematic and not intended to be limiting of the embodiments, and in which like reference numerals designate similar items in the figures and wherein:

[0079] Figure 1 A schematic block diagram of the apparatus for neural network processing provided by the embodiments of the present disclosure.

[0080] Figure 2 A schematic diagram of performing convolution operations for the first computing array in the embodiments of the present disclosure.

[0081] Figure 3 A schematic diagram of performing neural network operations for the second computing array in the embodiments of the present disclosure.

[0082] Figure 4 Another schematic block diagram of the apparatus for neural network processing provided by the embodiments of the present disclosure.

[0083] Figure 5 Still another schematic block diagram of the apparatus for neural network processing provided by the embodiments of the present disclosure.

[0084] Figure 6 Still another schematic block diagram of the apparatus for neural network processing provided by the embodiments of the present disclosure.

[0085] Figure 7 Still another schematic block diagram of the apparatus for neural network processing provided by the embodiments of the present disclosure.

[0086] Figure 8 A schematic diagram of the multiple second computing arrays included in the apparatus for neural network processing provided by the embodiments of the present disclosure.

[0087] Figure 9Another schematic diagram of a plurality of second compute arrays included in an apparatus for neural network processing provided by embodiments of the present disclosure.

[0088] Figure 10 Yet another schematic block diagram of an apparatus for neural network processing provided by embodiments of the present disclosure.

[0089] Figure 11 A schematic block diagram of an apparatus for neural network processing provided by another embodiment of the present disclosure.

[0090] Figure 12 Another schematic block diagram of an apparatus for neural network processing provided by another embodiment of the present disclosure.

[0091] Figure 13 A schematic block diagram of a system for neural network processing provided by embodiments of the present disclosure.

[0092] Figure 14 A schematic flow diagram of a method for neural network processing provided by embodiments of the present disclosure.

[0093] Figure 15 Another schematic flow diagram of a method for neural network processing provided by embodiments of the present disclosure.

[0094] Figure 16 Another schematic flow diagram of a method for neural network processing provided by embodiments of the present disclosure. DETAILED DESCRIPTION

[0095] In order to enable every detailed understanding of the characteristics and technical contents of the embodiments of the present disclosure, the implementation of the embodiments of the present disclosure will be described in detail below in conjunction with the drawings for reference only, and not to limit the embodiments of the present disclosure. In the following technical description, in order to facilitate explanation, a plurality of details are provided to provide a full understanding of the disclosed embodiments. However, one or more embodiments can still be implemented without these details. In other cases, in order to simplify the drawings, well-known structures and devices can be simplified.

[0096] In order to better understand the technical solutions of the embodiments of the present disclosure, the concepts related to the embodiments of the present disclosure are described first below.

[0097] 1. Deep learning

[0098] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. Machine learning (ML) is the core of artificial intelligence, and the goal is to enable machines (general-purpose computers) to obtain human-like intelligence through learning. Deep learning (DL) is a machine learning algorithm. Deep learning has good application prospects in the fields of image processing, pattern recognition, speech recognition, natural language, etc.

[0099] Deep learning is essentially a deep neural network. The neural network of deep learning can be referred to as a deep neural network. The deep neural network includes an input layer, a plurality of hidden layers and an output layer. The input layer is used to input input values (input feature values) to the neural network. The output of each hidden layer is the sum (i.e., multiplication and accumulation) of the product of a set of weight values and its corresponding input feature values. The output of each hidden layer can be referred to as an output feature value, which serves as the input feature value of the next hidden layer or the output layer.

[0100] Typical deep learning neural network models include deep convolutional neural network (DCNN), deep belief network (DBN) and stacked auto-encoder network models, etc. Among them, the deep convolutional neural network is a deep neural network whose operation of at least one hidden layer is convolution operation.

[0101] Taking the deep convolutional neural network as an example. The input value (input by the input layer) of the deep convolutional neural network is subjected to any one or more operations of the following operation types by the hidden layer, and then the output value (output by the output layer) is obtained:

[0102] Convolution, transposed convolution or deconvolution, batch normalization (BN), scaling, fully connected, concatenation, pooling, element-wise addition and activation, etc.

[0103] In the embodiments of the present disclosure, the operations that the hidden layer of the neural network can involve are not limited.

[0104] For example, the operations involved in the hidden layers of a deep neural network can also include depthwise (DW) convolution and pointwise (PW) convolution, which are two steps of decomposing a full convolution operation. Such a two-step decomposed convolution operation can also be referred to as depthwise separable convolution.

[0105] The operation of depthwise (DW) convolution is different from the operation of a regular convolution. In DW convolution, one convolution kernel is responsible for one channel, and one channel is only convolved by one convolution kernel. In the operation of a regular convolution, each convolution kernel is operated on each channel of an input image. The number of output feature maps obtained by DW convolution is the same as the number of channels of an input layer.

[0106] Pointwise (PW) convolution is performed on the output feature maps obtained by DW convolution. The operation of PW convolution is similar to that of a regular convolution. The size of the convolution kernel of PW convolution is 1x1xM, where M represents the number of channels of the previous layer.

[0107] In a deep neural network, each layer is usually named according to the function it implements, for example, a layer implementing convolution operation is referred to as a convolution layer, and a layer implementing pooling operation is referred to as a pooling layer. For example, the hidden layers of a deep convolutional neural network can include a transposed convolution layer, a normalization layer, a scaling layer, a fully connected layer, a concatenation layer, an element-wise addition layer, and an activation layer, etc. The specific operation process of each layer can refer to existing technologies, and will not be described herein.

[0108] 2. Neural network accelerator

[0109] As described above, deep learning needs to process a large amount of data. With the development of deep learning, the computing requirements of deep learning are getting higher and higher, and traditional processors are already difficult to meet the computing requirements of deep learning.

[0110] In view of the computing requirements of deep learning, the industry proposes a neural network dedicated processor, including a tensor processing unit (TPU). The TPU in the prior art is a neural network dedicated processor based on a systolic array. The systolic array can well accelerate convolution operations.

[0111] As described above, deep learning involves multiple types of operations, in addition to convolution operations, other types of operations such as pooling operations and fully connected operations, so it is necessary to accelerate the computation of multiple types of operations in deep neural networks.

[0112] The embodiments of the present disclosure provide a method, device and system for neural network processing, which can realize calculation acceleration for various types of operations in a neural network. In other words, the embodiments of the present disclosure provide a neural network processing scheme, which can realize further calculation acceleration compared with the existing TPU.

[0113] The input feature data involved in the embodiments of the present disclosure represents data in an input feature map (IFMAP).

[0114] In other words, the input feature data represents data in an image matrix corresponding to an image to be processed by a neural network. The image matrix corresponding to the image to be processed by the neural network can also be referred to as an input feature matrix. The input feature matrix can be a two-dimensional matrix, for example, the input feature matrix is a matrix with a size of HxW. The input feature matrix can also be a multi-dimensional matrix, for example, the input feature matrix is a matrix with a size of HxWxR, which can be understood as an HxW two-dimensional matrix with R channels. When the input feature matrix is a multi-dimensional matrix, it can also be understood that the input of the neural network includes multiple feature maps. For example, the feature matrix corresponding to a color image is HxWx3, that is, an HxW two-dimensional matrix with 3 channels, and the 3 matrices correspond to the three primary colors RGB of the image respectively. H is referred to as the height of the input feature matrix, W is referred to as the width of the input feature matrix, and R is referred to as the depth of the input feature matrix. The input feature data can also be referred to as an input feature value.

[0115] The output feature data involved in the embodiments of the present disclosure represents data in an output feature map (OFMAP).

[0116] Or, the output feature data represents data in an output feature matrix. The output feature matrix represents a matrix obtained by performing neural network operation on the input feature matrix. Taking a convolution layer as an example, the output feature matrix represents a matrix obtained by performing convolution operation on the input feature matrix and a filter matrix. The filter matrix represents a matrix composed of weight values used by the convolution layer. The filter matrix can be a two-dimensional matrix, for example, the filter matrix is a matrix with a size of HxW. The filter matrix can also be a multi-dimensional matrix, for example, the filter matrix is a matrix with a size of HxWxR, which can be understood as R two-dimensional HxW matrices. For example, for a color image, the corresponding filter matrix should also be a three-dimensional matrix HxWx3, that is, three two-dimensional HxW matrices, which correspond to the three primary colors RGB of the image. Among them, H is referred to as the height of the filter matrix, W is referred to as the width of the filter matrix, and R is referred to as the depth of the filter matrix. The output feature data can also be referred to as the output feature value. Similarly, the output feature matrix can be a two-dimensional matrix, for example, the output feature matrix is a matrix with a size of HxW. The output feature matrix can also be a multi-dimensional matrix, for example, the output feature matrix is a matrix with a size of HxWxR, where H is referred to as the height of the output feature matrix, W is referred to as the width of the output feature matrix, and R is referred to as the depth of the output feature matrix. It should be understood that the depth of the output feature matrix is consistent with the depth of the filter matrix.

[0117] Figure 1 A schematic block diagram of the apparatus 100 for neural network processing is provided for the embodiments of the present disclosure. As shown in the figure, the apparatus 100 includes a first computing array 10, a second computing array 20, and a control module 30. Figure 1 The first computing array 10 is configured to perform a first type of neural network operation, and the second computing array 20 is configured to perform a second type of neural network operation different from the first type of neural network operation.

[0118] The control module 30 is configured to control the first computing array 10 to perform the first type of neural network operation, and control the second computing array 20 to perform the second type of neural network operation.

[0119] The control module 30 can control the first computing array 10 to perform the first type of neural network operation by sending a command (or instruction) indicating the first type of neural network operation to the first computing array 10. The control module 30 can control the second computing array 20 to perform the second type of neural network operation by sending a command (or instruction) indicating the second type of neural network operation to the second computing array 20.

[0120] The control module 30 can control the first computing array 10 to perform the first type of neural network operation by sending a command (or instruction) indicating the first type of neural network operation to the first computing array 10. The control module 30 can control the second computing array 20 to perform the second type of neural network operation by sending a command (or instruction) indicating the second type of neural network operation to the second computing array 20.

[0121] For example, the first type of neural network operation and the second type of neural network operation can be any one of the following operation types, wherein the second type of neural network operation is different from the first type of neural network operation:

[0122] Convolution, transposed convolution or deconvolution, batch normalization (BN), scale, fully connected, concatenation, pooling, element-wise addition, activation, depthwise (DW) convolution, and pointwise (PW) convolution.

[0123] For example, when the first type of neural network operation is convolution operation, the second type of neural network operation is any one of the above operation types other than convolution operation, for example, pooling operation.

[0124] It should be noted that when the first type of neural network operation is conventional convolution operation, the second type of neural network operation can be depthwise (DW) convolution operation or pointwise (PW) convolution operation.

[0125] The second type of neural network operation can include vector operation. The vector operation means operation of multi-dimensional data and multi-dimensional data.

[0126] The second type of neural network operation can also include scalar operation. The scalar operation means operation of single data and single data.

[0127] It should be understood that the apparatus 100 provided by the embodiments of the present disclosure includes the first computing array 10 and the second computing array 20 for performing different types of operations in the neural network, wherein the first computing array 10 and the second computing array 20 can be used to perform a plurality of different types of operations, and therefore, the apparatus 100 provided by the embodiments of the present disclosure can accelerate a plurality of types of operations.

[0128] It should also be understood that, as described above, the operation types involved in the hidden layer of the deep neural network can include a plurality of the above-mentioned operation types, and it is necessary to accelerate a plurality of operation types. As known from the above, the apparatus 100 provided by the embodiments of the present disclosure can solve this problem.

[0129] As an example, the first computing array 10 is configured to perform convolution operation of a convolution layer of a deep neural network, and the second computing array 20 is configured to perform pooling operation of a pooling layer, then the apparatus 100 provided by the embodiments of the present disclosure can accelerate the convolution layer and the pooling layer of the deep neural network (assuming that the deep neural network includes the convolution layer and the pooling layer).

[0130] As another example, the first computing array 10 is configured to perform convolution operation of a convolution layer of a deep neural network, and the second computing array 20 is configured to perform full connection operation of a full connection layer, then the apparatus 100 provided by the embodiments of the present disclosure can accelerate the convolution layer and the full connection layer of the deep neural network (assuming that the deep neural network includes the convolution layer and the full connection layer).

[0131] It should also be understood that the above is only an example and is not limiting, and in actual applications, the type of operation that can be supported by the first computing array 10 and the second computing array 20 included in the apparatus 100 provided by the embodiments of the present disclosure can be determined according to actual needs.

[0132] For example, the apparatus 100 provided by the embodiments of the present disclosure can be regarded as having a heterogeneous acceleration structure of “first computing array + second computing array”. By adopting such a heterogeneous acceleration structure, the operation on multiple layers involved in the deep neural network can be accelerated, and the neural network operation efficiency can be effectively improved.

[0133] Therefore, in the apparatus 100 provided by the embodiments of the present disclosure, the first computing array 10 and the second computing array 20 for performing different types of operations in the neural network are included, so that the apparatus 100 can accelerate multiple types of operations in the deep neural network.

[0134] It should be noted that, Figure 1 This is only an example and is not limiting. For example, the apparatus 100 provided by the embodiments of the present disclosure can include more than two computing arrays. As described below in combination with Figure 8 and Figure 9 the description, the apparatus 100 provided by the embodiments of the present disclosure can include multiple second computing arrays 20. This will be described below.

[0135] The control module 30 is further configured to instruct the parsing.

[0136] For example, the control module 30 is configured to receive a previous instruction and parse the instruction to determine the computing task to be performed by the apparatus 100.

[0137] The control module 30 is further configured to control the computing process of the first computing array 10 and the second computing array 20 and transmit the operation number.

[0138] As an example, the control module 30 can control the first computing array 10 to perform a first type of neural network operation by performing the following operations: reading the data required for the first type of neural network operation and inputting it into the first computing array 10; sending a command (or instruction) to the first computing array 10 to instruct it to perform the first type of neural network operation, so that the first computing array 10 performs the first type of neural network operation on the input data.

[0139] Similarly, the control module 30 can control the second computing array 20 to perform the second type of neural network operation by performing the following operations: reading the data required for the second type of neural network operation and inputting it into the second computing array 20; sending a command (or instruction) to the second computing array 20 to instruct it to perform the second type of neural network operation, so that the second computing array 20 performs the second type of neural network operation on the input data.

[0140] Optionally, the control module 30 can also be used to store the data (output results or intermediate results) processed by the first computing array 10 and the second computing array 20 into an on-chip storage module. The on-chip storage module refers to the storage module on the device 100.

[0141] As an example, when the first type of neural network operation is a convolution operation, the computational units in the first computational array 10 have multiplication and accumulation functions. A schematic diagram of the control module 30 controlling the first computational array 10 to perform the convolution operation is shown below. Figure 2 As shown. The control module 30 inputs the input feature data and weights required for the convolution operation into the first computing array 10. The first computing array 10 performs a convolution operation on the input feature data and weights to obtain output feature data. The input feature data and weights required for the convolution operation can be read from the on-chip storage module and input into the first computing array 10 by the control module 30. The on-chip storage module refers to the storage module on the device 100.

[0142] The operation of reading data from an external (off-chip) storage module to an on-chip storage module can be performed by other related modules, for example, by the following: Figure 13 The intermediate unit 1040 (GDMAx) shown is executed, but this disclosure does not limit the scope of the embodiments.

[0143] The first computing array 10 can perform convolution operations on the input feature data and weights by receiving commands from the control module 30.

[0144] The process of the control module 30 controlling the first computing array 10 to perform convolution operation can also be described as follows: the input feature map (IPMAP) and weights required for the convolution operation are input into the first computing array 10; the first computing array 10 performs convolution operation on the input feature map and weights to obtain the output feature map (OPMAP).

[0145] As an example, when the second type of neural network operation is a pooling operation, a schematic diagram of the control module 30 controlling the second computing array 20 to perform the pooling operation is shown in FIG. 3B. Figure 3 The control module 30 inputs the input feature data required by the pooling operation to the second computing array 20; the second computing array 20 performs the pooling operation on the input input feature data according to the pooling operation mode and the pooling frame resolution, and obtains the output feature data.

[0146] For example, the pooling operation mode includes finding the maximum value or finding the average value. For example, when the pooling operation mode is to find the maximum value, the calculation unit in the second computing array 20 has the function of finding the maximum value, and when the pooling operation mode is to find the average value, the calculation unit in the second computing array 20 has the function of finding the average value. The pooling frame resolution indicates that the input feature data of how large size on the input feature map is pooled (i.e. down-sampled). It should be understood that the pooling operation mode and the pooling frame resolution can be pre-set. For example, the control module 30 can obtain the pooling operation mode and the pooling frame resolution by parsing the instruction.

[0147] The input feature data required by the pooling operation can be read by the control module 30 from the on-chip storage module and input to the second computing array 20.

[0148] The second computing array 20 can obtain the pooling operation mode and the pooling frame resolution by receiving the command of the control module 30.

[0149] The flow of the control module 30 controlling the second computing array 20 to perform the pooling operation can also be described as: the input feature map (IPMAP) required by the pooling operation is input to the second computing array 20; the second computing array 20 performs the pooling operation on the input input feature map and obtains the output feature map (OPMAP).

[0150] The implementation method of the computing array for performing the pooling operation can refer to the existing technology, which will not be described herein.

[0151] Optionally, in the embodiment in which the first type of neural network operation is a convolution operation, the first computing array 10 can be a three-dimensional (3D) multiply-accumulate (MAC) array, as shown in FIG. 3A. Figure 7

[0152] The three dimensions of the 3D MAC array respectively represent the number of channels of the input feature map (IFMAP) that can be calculated, the number of channels of the output feature map (OFMAP), and the number of HWS of the OFMAP, wherein the number of HWS of the OFMAP represents the number of convolution windows that can be processed at the same time. ​

[0153] For example, assuming the maximum values of the three dimensions are denoted as IC, OC, and OHW respectively, the number of MACs included in the first computing array 10 is IC x OC x OHW. The values of the three parameters IC, OC, and OHW can be set according to different application scenarios and requirements of the chip, so that the device can have a higher utilization rate.

[0154] Optionally, in the embodiment in which the first type of neural network operation is convolution operation, the first computing array 10 can be a systolic array.

[0155] For example, the weights flow into the systolic array from top to bottom, and flow in the systolic array from top to bottom; the input feature data flow into the systolic array from left to right, and flow in the systolic array from left to right.

[0156] The bottom of the systolic array can be connected to an accumulation unit, which can be used to save partial results in the case that the input feature matrix or the weight matrix exceeds the range of the cells of the systolic array.

[0157] The principle and operation process of the systolic array can refer to the existing technology, and will not be described herein.

[0158] It should be understood that, in the above embodiment, the second type of neural network operation is taken as the pooling operation for description, but the embodiments of the present disclosure are not limited thereto. For example, in the case where the first type of neural network operation is convolution operation, the second type of neural network operation can be other types of operations other than convolution operation. It should also be understood that the structure and function of the second computing array 20 and the computing cells therein are different corresponding to different second type of neural network operations. In other words, the second computing array 20 can implement different types of operations in the neural network.

[0159] Therefore, by including multiple computing arrays that can perform different types of operations in the neural network, the versatility of the device for processing the neural network provided by the embodiments of the present disclosure can be improved.

[0160] Optionally, the control module 30 is configured to control the first computing array 10 to perform the first type of neural network operation, and control the second computing array 20 to perform the second type of neural network operation, so that the first type of neural network operation and the second type of neural network operation are performed in parallel.

[0161] For example, in the device 100 for processing the neural network provided by the embodiments of the present disclosure, with the cooperation of software, two instructions indicating different types of neural network operations can be run simultaneously, for example, the instruction indicating the convolution operation and the instruction indicating the pooling operation can be run simultaneously.

[0162] It should be understood that, by performing multiple types of operations in the neural network in parallel, the computing efficiency of the deep neural network can be effectively improved.

[0163] Optionally, the apparatus 100 can further comprise an on-chip storage module, such as the storage module 40 shown in Figure 1 , which is configured to store data to be processed by the first computing array 10 and the second computing array 20.

[0164] For example, the control module 30 is configured to read out data required by the first type of neural network from the storage module 40 and input the data into the first computing array 10, and read out data required by the second type of neural network from the storage module 40 and input the data into the second computing array 20.

[0165] For the operation of reading out data from an external (off-chip) storage module into the on-chip storage module, it can be performed by other related modules, for example, it can be performed by the intermediate unit 1040 (GDMAx) shown in Figure 13 , which is not limited in the embodiments of the present disclosure.

[0166] Optionally, the storage module 40 can also be configured to store operation results (final calculation results or intermediate calculation results) of the first computing array 10 and the second computing array 20.

[0167] Optionally, the storage module 40 can be logically divided into multiple storage modules.

[0168] For example, according to the logical division, the storage module 40 can be divided into two storage modules respectively configured to store data for the first computing array 10 and the second computing array 20. For another example, according to the logical division, the storage module 40 can be divided into multiple storage modules respectively configured to store different types of data.

[0169] Optionally, the storage module 40 can be physically divided into multiple storage modules. In other words, the apparatus 10 can comprise multiple physically independent storage modules, in which case the storage module 40 can be regarded as a general term of the multiple physically independent storage modules.

[0170] For example, the apparatus 100 comprises two storage modules respectively configured to store data for the first computing array 10 and the second computing array 20, and the two storage modules belong to two different storage modules in a physical sense. For another example, the apparatus 100 comprises multiple storage modules respectively configured to store different types of data, and part or all of the multiple storage modules belong to different storage modules in a physical sense.

[0171] In the embodiments of the present disclosure, different types of data can be stored separately, and data processed by different computing arrays can also be stored separately. This will be described below.

[0172] Optionally, as shown in Figure 4In the embodiment shown in FIG. 1, the device 100 further comprises a first storage module 41 and a second storage module 42. The first storage module 41 is configured to store the weights required for the convolution operation, and the second storage module 42 is configured to store the feature data required for the convolution operation. The control module 30 is configured to read the weights required for the convolution operation from the first storage module 41 to the first computing array 10, and read the feature data required for the convolution operation from the second storage module 42 to the first computing array 10.

[0173] For example, the first storage module 41 can be referred to as L1 weight Mem, and the second storage module 42 can be referred to as L1 Data Mem.

[0174] In the embodiment shown in FIG. 1, the first storage module 41 and the second storage module 42 are physically separated. In other words, the first storage module 41 and the second storage module 42 are physically two different storage modules. Figure 4 The first storage module 41 and the second storage module 42 are on-chip storage modules of the device 100.

[0175] For example, the weights required for the convolution operation can be read out from the external memory and written into the first storage module 41 by other related modules, such as the intermediate unit 1040 (GDMAx) shown in FIG. 10 below, and the input feature data required for the convolution operation can be read out from the external memory and written into the second storage module 42, which is not limited in the embodiments of the present disclosure.

[0176] Figure 13 It should be understood that, by storing the feature data and the weights required for the convolution operation in the first storage module 41 and the second storage module 42 respectively, the feature data and the weights required for the convolution operation can be read from two read ports respectively, which can improve the efficiency of data reading.

[0177] Optionally, in the embodiment shown in FIG. 1, the control module 30 is configured to read the weights and the feature data required for the convolution operation from the first storage module 41 and the second storage module 42 in parallel.

[0178] It should be understood that, by reading the feature data and the weights required for the convolution operation in parallel, the efficiency of data reading in the convolution neural network processing process can be improved, and thus the overall efficiency of the convolution neural network processing can be improved. Figure 4

[0179] It should be understood that, by reading the feature data and the weights required for the convolution operation in parallel, the efficiency of data reading in the convolution neural network processing process can be improved, and thus the overall efficiency of the convolution neural network processing can be improved.

[0180] ​​Therefore, the apparatus 100 for processing neural network provided by the embodiments of the present disclosure can improve the efficiency of data reading, and thus improve the overall efficiency of processing of the convolutional neural network, by using two storage modules to separately store the feature data and the weight required for the convolution operation, so that the feature data and the weight required for the convolution operation can be read from two reading ports respectively.

[0181] Optionally, in the embodiments shown in Figure 4 The apparatus 100 can further include a storage module (denoted as storage module x) dedicated to storing data required for the second type of neural network operation (not shown in the figure). Figure 4 The control module 30 is configured to read the data required for the second type of neural network operation from the storage module x to the second computing array 20.

[0182] Optionally, in the embodiments shown in Figure 4 The first computing array 10 and the second computing array 20 can share one data storage module, as shown in Figure 4 The second storage module 42 can also be configured to store data required for the second type of neural network operation; and the control module 30 is further configured to read the data required for the second type of neural network operation from the second storage module 42 to the second computing array 20.

[0183] For example, the data stored in the second storage module 42 can be uniformly addressed. Before reading the data from the second storage module 42, the data storage address corresponding to the computing task to be performed is determined, and then the corresponding data is read from the second storage module 42 according to the data storage address.

[0184] For example, for the convolution operation instruction, the storage address of the data required for the convolution operation in the second storage module 42 is determined, and then the data at the storage address is read from the second storage module 42 and transmitted to the first computing array 10. For another example, the second type of neural network operation is the pooling operation, and the storage address of the data required for the pooling operation in the second storage module 42 is determined, and then the data at the storage address is read from the second storage module 42 and transmitted to the second computing array 20.

[0185] It should be understood that by sharing one data storage module by the first computing array 10 and the second computing array 20, the occupied space of the storage module on the apparatus 100 can be reduced.

[0186] Optionally, as another embodiment, as shown in Figure 5As shown, the apparatus 100 comprises a third storage module 43 and a fourth storage module 44 which are separately arranged. The third storage module 43 is configured to store data required for the first type of neural network operation, and the fourth storage module 44 is configured to store data required for the second type of neural network operation. The control module 30 is configured to read the data required for the first type of neural network operation from the third storage module 43 to the first computing array 10, and read the data required for the second type of neural network operation from the fourth storage module 44 to the second computing array 20.

[0187] In Figure 5 In the embodiment shown, the third storage module 43 and the fourth storage module 44 are physically separately arranged. In other words, the third storage module 43 and the fourth storage module 44 are physically two different storage modules.

[0188] The third storage module 43 and the fourth storage module 44 are on-chip storage modules of the apparatus 100.

[0189] For example, the data required for the first type of neural network operation can be read out from an external memory and written into the third storage module 43 by other related modules, such as the intermediate unit 1040 (GDMAx) shown in the following Figure 13 The data required for the second type of neural network operation can be read out from an external memory and written into the fourth storage module 44 by other related modules, such as the intermediate unit 1040 (GDMAx) shown in the following

[0190] As an example, the first type of neural network operation is a convolution operation, and the second type of neural network operation is a pooling operation. The third storage module 43 is configured to store feature data and weights required for the convolution operation, and the fourth storage module 44 is configured to store feature data required for the pooling operation.

[0191] It should be understood that the third storage module 43 and the fourth storage module 44 are used to separately store the data required for the first type of neural network operation and the data required for the second type of neural network operation.

[0192] It should also be understood that the third storage module 43 and the fourth storage module 44 are respectively used to store the data required for the first type of neural network operation and the data required for the second type of neural network operation, so that the data required for the first type of neural network operation and the data required for the second type of neural network operation can be read from two read ports respectively, and the efficiency of data reading can be improved.

[0193] Optionally, in Figure 5 In the embodiment shown, the control module 30 is configured to read the data required for the first type of neural network operation and the data required for the second type of neural network operation from the third storage module 43 and the fourth storage module 44 in parallel.

[0194] Therefore, the device for processing neural network provided by the embodiments of the present disclosure can improve the data reading efficiency, and further improve the calculation efficiency of the neural network.

[0195] Optionally, in the embodiments as shown in Figure 5 In the embodiments as shown in the case where the first neural network operation is a convolution operation, the third storage module 43 can be logically or physically divided into two sub-storage units, one of which is used to store feature data required by the convolution operation, and the other is used to store weights required by the convolution operation.

[0196] As can be seen above, the device 100 for processing neural network provided by the embodiments of the present disclosure includes a storage module having a plurality of implementation manners.

[0197] For example, the device 100 includes a storage module 40 as shown in Figure 1 For another example, the device 100 includes a first storage module 41 and a second storage module 42 as shown in Figure 4 For another example, the device 100 includes a third storage module 43 and a fourth storage module 44 as shown in Figure 5

[0198] Optionally, the device 100 mentioned in the embodiments of the present disclosure includes the first storage module 41 and the second storage module 42, which means that the device 100 includes a storage module including two read ports, and the two read ports correspond to the first storage module 41 and the second storage module 42 respectively.

[0199] Optionally, the device 100 mentioned in the embodiments of the present disclosure includes the third storage module 43 and the fourth storage module 44, which means that the device 100 includes a storage module including two read ports, and the two read ports correspond to the third storage module 43 and the fourth storage module 44 respectively.

[0200] Continuing to refer to Figure 4 Optionally, in the embodiments as shown in Figure 4 The distance from the first storage module 41 to the first calculation array 10 is less than the distance from the second storage module 42 to the first calculation array 10.

[0201] Because the distance from the first storage module 41 to the first calculation array 10 is relatively short, the time for the first calculation array 10 to read the weights from the first storage module 41 is reduced, thereby improving the efficiency of reading the weights in the convolution operation, and reducing the power consumption of reading the weights in the convolution operation.

[0202] ​It should be understood that, in the case that the second storage module 42 is also used to store data required by the second type of neural network operation, the second storage module 42 can be farther away from the first computing array 10.

[0203] Continuing to refer to Figure 5 Optionally, in the embodiment as shown in Figure 5 , the distance from the third storage module 43 to the first computing array 10 is less than the distance from the fourth storage module 44 to the first computing array 10; and / or, the distance from the fourth storage module 44 to the second computing array 20 is less than the distance from the third storage module 43 to the second computing array 20.

[0204] It should be understood that, because the third storage module 43 is closer to the first computing array 10, the time for the first computing array 10 to read data required by the first type of neural network operation from the third storage module 43 can be reduced, and because the fourth storage module 44 is closer to the second computing array 20, the time for the second computing array 20 to read data required by the second type of neural network operation from the fourth storage module 44 can be reduced, so that the efficiency of data reading can be improved and the power consumption of data reading can be reduced.

[0205] Optionally, in the embodiment as shown in Figure 4 , the apparatus 100 can further include a cache 50, as shown in Figure 6 , wherein the control module 30 is configured to read the feature data required by the convolution operation from the cache to the first computing array 10.

[0206] The embodiment can be applied to the case that the first type of neural network operation is a convolution operation.

[0207] It should be understood that the input feature data of the convolution operation has good locality, so the input feature data of the convolution operation can be read out from the second storage module 42 and cached to the cache 50 by using the good locality.

[0208] The access speed of the cache is fast, so that by using the cache to cache the feature data required by the convolution operation, the reading efficiency of the feature data required by the convolution operation can be improved, and the power consumption of data reading can be reduced.

[0209] In addition, if the second storage module 42 is farther away from the first computing array 10, the first computing array 10 takes a long time and consumes high power to read data from the second storage module 42. In the embodiment of the present disclosure, by using the cache to cache the data read from the first storage module 41, the first computing array 10 reads data from the cache, the reading frequency of the second storage module 42 can be reduced, so that the time of data reading can be reduced, and the power consumption of the convolution operation can be reduced.

[0210] The control module 30 is configured to read the feature data required for the convolution operation of the first computing array 10 from the second storage module 42 to the cache memory 50 in a case where the cache memory 50 does not include the feature data required for the convolution operation of the first computing array 10, and directly read the feature data from the second storage module 42 to the first computing array 10 in a case where the cache memory 50 already includes the feature data required for the convolution operation of the first computing array 10.

[0211] For example, the control module 30 can read the required feature data from the second storage module 42 to the cache memory 50 in advance, and directly read the feature data required for the convolution operation from the cache memory to the first computing array 10 in the process of performing the convolution operation by the first computing array 10.

[0212] For another example, the control module 30 can also read the required feature data from the second storage module 42 to the cache memory 50 in real time according to actual conditions. For example, the control module 30 is configured to perform the following steps.

[0213] 1) determining whether the cache memory 50 has the feature data required for the convolution operation of the first computing array 10;

[0214] 2) if the result of the determination is yes, reading the feature data required for the convolution operation from the cache memory 50 to the first computing array 10;

[0215] 3) if the result of the determination is no, reading the required feature data from the second storage module 42 to the cache memory 50, and reading the feature data required for the convolution operation from the cache memory 50 to the first computing array 10.

[0216] In the embodiment shown in FIG. 1, the distance from the cache memory 50 to the first computing array 10 is less than the distance from the second storage module 42 to the first computing array 10. Figure 6 Therefore, in the embodiment of the present disclosure, by adopting the cache memory to cache the data read from the first storage module 41, the first computing array 10 reads the data from the cache memory, which can reduce the number of reading from the second storage module 42, thereby reducing the time of data reading and further reducing the power consumption of the convolution operation. In addition, the efficiency of data reading in the convolution operation can also be improved.

[0217] Optionally, in the embodiment shown in FIG. 1, the distance from the first storage module 41 to the first computing array 10 is less than the distance from the second storage module 42 to the first computing array 10.

[0218] Figure 6 Therefore, in the embodiment of the present disclosure, by adopting the cache memory to cache the data read from the first storage module 41, the first computing array 10 reads the data from the cache memory, which can reduce the number of reading from the second storage module 42, thereby reducing the time of data reading and further reducing the power consumption of the convolution operation. In addition, the efficiency of data reading in the convolution operation can also be improved.

[0219] ​It should be understood that the first storage module 41 is relatively close to the first computing array 10, which can reduce the time for the weights stored in the first storage module 41 to be transmitted to the first computing array 10, thereby reducing the overhead of the first computing array 10 reading the weights from the first storage module 41.

[0220] Therefore, in this embodiment, both the efficiency of reading the feature data required for convolution operations and the efficiency of reading the weights required for convolution operations can be improved, thereby improving the efficiency of convolutional neural network operations. Simultaneously, in this embodiment, because the efficiency of reading the feature data and weights required for convolution operations is improved, power consumption can be reduced.

[0221] The device 100 may include a control module, namely, control module 30 represents a control unit.

[0222] Optionally, in some embodiments, the control module 30 in the device 100 may include multiple control units, each used to perform different control operations. Handshake signals may exist between the multiple control units.

[0223] like Figure 7 As shown, the first column of neural network operations is a convolution operation, that is, the first computing array 10 is used to perform convolution operations, and the control module 30 includes a first control unit 31 and a second control unit 32.

[0224] The first control unit 31 is used to control the process of convolution operation and the transfer of weights required during the convolution operation. Here, "transfer" means reading the weights into the first computing array 10.

[0225] The second control unit 32 is used to control the overall computation process of the first computing array 10 and the second computing array 20. For example, the second control unit 32 is responsible for transporting the input feature data required during the product operation, and also for transporting the data required for the second type of neural network operation performed by the second computing array 20. The second control unit 32 can also be used to store the computation results of the first computing array 10 and the second computing array 20 into the second storage module 42.

[0226] The first control unit 31 and the second control unit 32 may have a handshake signal to cooperate in transmitting the input feature data and weights required for convolution operations.

[0227] exist Figure 7 In the example, the first control unit 31 is used to read the weights required for the convolution operation from the first storage module 41 into the first computing array 10.

[0228] The second control unit 32 is used to read the feature data required for the convolution operation from the second storage module 42 and write it into the cache memory 50. When the first computing array 10 is to perform a convolution operation, the second control unit 32 is used to read the feature data required for the convolution operation from the cache memory 50 and write it into the first computing array 10.

[0229] For example, the second control unit 32 is used to perform the following steps.

[0230] 1) Determine whether the cache memory 50 contains the feature data required for the first computing array 10 to perform convolution operations;

[0231] 2) If the judgment result is yes, read the feature data required for the convolution operation from the high-speed cache memory 50 and put it into the first computing array 10;

[0232] 3) If the judgment result is negative, the required feature data is read from the second storage module 42 into the high-speed cache memory 50, and the feature data required for convolution operation is read from the high-speed cache memory 50 into the first computing array 10.

[0233] The second control unit 32 is also used to read the data required for the second neural network operation from the second storage module 42 to the second computing array 20.

[0234] The second control unit 32 can communicate with the first control unit 31. For example, the second control unit 32 can send a command to the first control unit 31, such as instructing the first control unit 31 to read the weights in the first storage module 41 into the first computing array 10.

[0235] For example, in Figure 5 In the illustrated embodiment, the control module 30 includes a first control unit and a second control unit. The first control unit is used to read data required for a first type of neural network operation from the third storage module 43 and store it in the first computing array 10; the second control unit is used to read data required for a second type of neural network operation from the fourth storage module 44 and store it in the second computing array 20.

[0236] Optionally, the neural network processing apparatus 100 provided in this embodiment of the disclosure may include a plurality of second computing arrays 20.

[0237] Each second computing array 20 can support one or more types of operations.

[0238] For example, each second computing array 20 can support operation instructions for any one or more of the following operation types:

[0239] transposed convolution or deconvolution, batch normalization (BN), scale, fully connected, concatenation, pooling, element-wise addition, activation, depthwise (DW) convolution, and pointwise (PW) convolution.

[0240] Optionally, in the embodiment where the apparatus 100 comprises a plurality of second computing arrays 20, the plurality of second computing arrays 20 can be configured to perform the same type of operation in the neural network.

[0241] As an example, the apparatus 100 comprises a plurality of second computing arrays 20 configured to perform a pooling operation.

[0242] It should be understood that by performing the pooling operation through the plurality of computing arrays, the computing efficiency of the pooling operation can be effectively improved to accelerate the pooling operation.

[0243] As another example, the apparatus 100 comprises a plurality of second computing arrays 20 configured to perform a fully connected operation.

[0244] It should be understood that by performing the fully connected operation through the plurality of computing arrays, the computing efficiency of the fully connected operation can be effectively improved to accelerate the fully connected operation.

[0245] It should also be understood that the above is only an example and not a limitation, and in the embodiment where the apparatus 100 comprises a plurality of second computing arrays 20, the plurality of second computing arrays 20 can also be configured to perform the same type of other operation, which is not enumerated here.

[0246] Therefore, in the apparatus 100 provided by the embodiments of the present disclosure, by adopting the plurality of second computing arrays 20 to perform the same type of operation in the neural network, the computing efficiency of the corresponding type of operation can be effectively improved.

[0247] It should be noted that in actual applications, the type of operation that can be supported by the plurality of second computing arrays 20 included in the apparatus 100 provided by the embodiments of the present disclosure can be determined according to actual needs.

[0248] It should also be noted that the description of the second computing array 20 in the above embodiments is applicable to each of the plurality of second computing arrays 20 in the present embodiment, and details are described above and will not be repeated here.

[0249] As an example, Figure 8 is a schematic diagram of a plurality of second computing arrays 20 included in the apparatus 100. In Figure 8 , the apparatus 100 includes n second computing arrays 20, n being a positive integer. In Figure 8 , each of the second computing arrays 20 corresponds to a memory unit (Mem). The entirety of each of the second computing arrays 20 and the memory unit (Mem) therein can be denoted as a block, as Figure 8 indicated. For example, the block can also be denoted as a core, a block, or a Lane. As Figure 8 indicated, the apparatus 100 includes n blocks (Block 0 to Block (n-1)). For example, Figure 8 , the memory units (Mem) in Block 0 as indicated in indicate memory units for storing data required by the second computing arrays 20 in Block 0 to perform neural network operations.

[0250] It should be noted that, Figure 8 , the n memory units as indicated in are physically separate and logically the same memory unit.

[0251] For example, in Figure 4 , the n memory units as indicated in Figure 8 are logically the same memory unit: the second memory module 42.

[0252] For another example, in Figure 5 , the n memory units as indicated in Figure 8 are logically the same memory unit: the fourth memory module 44.

[0253] Continuing to refer to Figure 8 , the n second computing arrays 20 included in the apparatus 100 can be regarded as a group, which can be denoted as Array.

[0254] Optionally, as Figure 8 indicated, the apparatus 100 can further include an array interconnection structure 60 for being responsible for the transmission of commands and data between the control module (such as the second control unit 32 as indicated in Figure 8 ) and each of the second computing arrays 20 in the group of n second computing arrays 20.

[0255] For example, the array interconnection structure 60 can be a bus interconnection structure, or other forms of connection structures.

[0256] For example, the array interconnection structure 60 can be denoted as Array Fabx.

[0257] The array interconnection structure 60 can be in communication with the control module of the apparatus 100. For example, in Figure 7 , the array interconnection structure 60 is in communication with the second control unit 32.In the illustrated embodiment, the array interconnect structure 60 can communicate with the second control unit 32, such as... Figure 8 As shown.

[0258] Optionally, such as Figure 9 As shown, the device 100 may include m units. Figure 8 The groups shown (such as) Figure 9 The groups shown (0 to (m-1)) comprise n×m second computation arrays 20, where m is a positive integer. Each group includes an array interconnection structure 60. The array interconnection structure 60 of each group (Array) can be called Array Fab i (i is 0 to m-1).

[0259] For example, Figure 9 The entirety of m groups (Arrays) can be denoted as a lane array.

[0260] exist Figure 4 In the embodiments, Figure 9 The storage units (Mem) in each block (Lane) can be logical partitioning units in the second storage module 42. The data stored in the second storage module 42 can be uniformly addressed, and the storage units (Mem) in each block (Lane) can correspond to a segment of storage address in the second storage module 42.

[0261] For example, to perform neural network operations using the second computing array 20 in a certain block (Lane), first determine the storage address of the storage unit (Mem) in the second storage module 42 in this block (Lane), then read the data at that storage address from the second storage module 42 and transmit it to the second computing array 20 in this block (Lane).

[0262] For example, in Figure 9 In the illustrated embodiment, if device 100 needs to perform a pooling operation task, the execution process is as follows.

[0263] 1) The control module 30 parses the system instructions, learns that a pooling operation is to be performed, and determines the identifier (ID) of the block (Lane) used to perform the pooling operation. Assuming the identifier is 0_1, that is, the second computing array 20 in block 1 of group 0 is used to perform the pooling operation.

[0264] It should be understood that the blocks used to perform pooling operations can include one or more. Here, we describe one block as an example only and not a limitation.

[0265] 2) Control module 30 sends instructions of reading input feature map (IFMAP) and reading pooling kernel to the storage unit (Mem) in block 1 of group 0 through two reading channels (for example, which can be called r0 and r1). Among them, r0 channel is responsible for reading input feature map, and r1 is responsible for reading pooling kernel. The pooling kernel mentioned here corresponds to the pooling frame size mentioned above.

[0266] After the input feature map (IFMAP) and the pooling kernel are read out from the corresponding storage unit of block 1 of group 0, they are sent into the second computing array 20 in block 1 of group 0 for pooling operation.

[0267] Reading input feature map (IFMAP) and pooling kernel from the storage unit (Mem) in block 1 of group 0 is actually reading input feature map (IFMAP) and pooling kernel from the corresponding address of the storage unit (Mem) in block 1 of group 0 in the second storage module 42.

[0268] 3) After completing the pooling operation, the control module 30 sends a write command through the write port to write the pooling result obtained by the second computing array 20 in block 1 of group 0 into the corresponding storage unit of block 1 of group 0.

[0269] It should be noted that in actual application, the number of second computing arrays 20 included in the device 100 can be determined according to actual needs.

[0270] In the embodiment shown in Figure 4 , the second computing array 20 and the module or area where the second storage module 42 is located can be recorded as an array lane, as shown in Figure 9 .

[0271] In the embodiment shown in Figure 4 , the first computing array 10 and the module or area where the first storage module 41 is located can be recorded as a convolution acceleration unit (also called Cube Top), as shown in Figure 7 .

[0272] As shown in Figure 10 , the device 100 for neural network processing provided by the embodiment of the present disclosure can be represented as being composed of a convolution acceleration unit (Cube Top), an array lane (Array Lane) and a control module 30.

[0273] Figure 11A schematic block diagram of the apparatus 200 for neural network processing is provided for another embodiment of the present disclosure. The apparatus 200 comprises a first computing array 10, a first storage module 41, a second storage module 42, and a control module 30. The first storage module 41 and the second storage module 42 are physically separated.

[0274] The first computing array 10 is configured to perform a first type of neural network operation, and the first computing array is a 3D MAC array, and the first type of neural network operation is a convolution operation.

[0275] The first storage module 41 is configured to store weights required for the convolution operation.

[0276] The second storage module 42 is configured to store feature data required for the convolution operation.

[0277] The control module 30 is configured to read the weights required for the convolution operation from the first storage module 41 to the first computing array 10, and read the feature data required for the convolution operation from the second storage module 42 to the first computing array 10.

[0278] Figure 11 The first computing array 10 in the illustrated embodiment is the same as the first computing array 10 for performing the convolution operation in the above embodiment, and the specific description is referred to the above, which will not be repeated here.

[0279] Figure 11 The control module 30 in the illustrated embodiment is the same as the control module 30 in the above embodiment, and the specific description is referred to the above, which will not be repeated here.

[0280] Figure 11 The first storage module 41 and the second storage module 42 in the illustrated embodiment are respectively the same as the first storage module 41 and the second storage module 42 in the above embodiment, and the specific description is referred to the above, which will not be repeated here.

[0281] It should be understood that in the apparatus 200 for neural network processing provided by the embodiment of the present disclosure, the feature data and the weights required for the convolution operation are stored separately, so that the feature data and the weights required for the convolution operation can be read from two read ports respectively, thereby improving the data reading efficiency.

[0282] In Figure 11 In the illustrated embodiment, whether the apparatus 200 comprises the second computing array 20 in the above embodiment is not limited.

[0283] Optionally, in Figure 11 In the illustrated embodiment, the control module 30 is configured to read the weights and the feature data required for the convolution operation from the first storage module 41 and the second storage module 42 to the first computing array 10 in parallel.

[0284] It should be understood that by reading the feature data and weights required for convolution operations in parallel, the data reading efficiency in the convolutional neural network processing process can be improved, thereby improving the overall efficiency of convolutional neural network processing.

[0285] Optionally, such as Figure 12 As shown, in Figure 10 In the illustrated embodiment, the device 200 further includes a cache memory 50 for caching feature data required for convolution operations read from the second storage module 42. The control module 30 is configured to read the feature data required for convolution operations from the cache memory 50 into the first computing array 10.

[0286] Figure 12 The cache memory 50 in the illustrated embodiment is the same as the cache memory 50 in the embodiment above. For related descriptions, please refer to the above text, and they will not be repeated here.

[0287] exist Figure 12 In the illustrated embodiment, the control module 30 is configured to read feature data from the second storage module 42 into the cache memory 50 when the cache memory 50 does not contain the feature data required for the first computing array 10 to perform convolution operations; and to directly read feature data from the second storage module 42 into the first computing array 10 when the cache memory 50 already contains the feature data required for the first computing array 10 to perform convolution operations.

[0288] For example, the control module 30 can read the required feature data from the second storage module 42 into the cache memory 50 in advance, and directly read the feature data required for the convolution operation from the cache memory into the first computing array 10 during the convolution operation of the first computing array 10.

[0289] For example, the control module 30 can also read the required feature data from the second storage module 42 into the cache memory 50 in real time, based on the actual situation. See the relevant description above for details, which will not be repeated here.

[0290] It should be understood that cache memory has a faster access speed. Therefore, by using cache memory to cache the feature data required for convolution operations, the reading efficiency of the feature data required for convolution operations can be improved, and the power consumption of data reading can also be reduced.

[0291] Optionally, in Figure 12 In the illustrated embodiment, the distance from the cache memory 50 to the first computing array 10 is less than the distance from the second storage module 42 to the first computing array 10.

[0292] For example, in Figure 11 and Figure 12In the illustrated embodiment, the first computing array 10 performs a convolution operation, as shown in the schematic diagram of FIG. 1. Figure 7 As shown, the relevant description is detailed above and will not be repeated here.

[0293] It should be understood that the embodiment can further improve the reading efficiency of the feature data required for the convolution operation, and also reduce the power consumption of data reading.

[0294] Optionally, the apparatus 100 or 200 for neural network processing provided by the embodiment of the present disclosure can be a neural network dedicated chip. For example, the apparatus 100 or 200 can be referred to as a TPU.

[0295] Figure 13 A schematic block diagram of the system 1000 for neural network processing provided by the embodiment of the present disclosure is shown.

[0296] The system 1000 includes one or more processing units 1010. In Figure 13 For example, the system 1000 includes multiple processing units 1010.

[0297] Each processing unit 1010 is the apparatus 100 or the apparatus 200 for neural network processing in the above embodiment.

[0298] For example, the processing unit 1010 is the apparatus 100 as shown, i.e., the processing unit 1010 is composed of a convolution computing array region (Cube Top), an array lane, and a control module. Figure 10

[0299] In the case where the system 1000 includes multiple processing units 1010, the multiple processing units 1010 can include the apparatus 100 and / or the apparatus 200.

[0300] If the apparatus 100 or the apparatus 200 is referred to as a TPU core, the system 1000 for neural network processing provided by the embodiment of the present disclosure includes one or more TPU cores.

[0301] Each processing unit 1010 in the system 1000 can perform the operation of one neural network (e.g., a deep neural network). It should be understood that when the system 1000 includes multiple processing units 1010, the system 1000 can perform the operation of multiple neural networks.

[0302] In the case where the system 1000 includes multiple processing units 1010, the multiple processing units 1010 can run the same neural network simultaneously, or run different neural networks.

[0303] ​Multiple processing units 1010 running the same neural network simultaneously means that multiple processing units 1010 can execute the operations of the same neural network at the same time. Multiple processing units 1010 running different neural networks means that different processing units 1010 among the multiple processing units 1010 execute the operations of different neural networks.

[0304] System 1000 also includes a storage unit 1020 for storing data required by one or more processing units 1010 to perform neural network operations.

[0305] For example, in the case where system 1000 includes multiple processing units 1010, the multiple processing units 1010 share the bandwidth of storage unit 1020.

[0306] For example, the storage unit 1020 includes one or more of the following: a first memory and a second memory, wherein the read speed of the second memory is greater than the read speed of the first memory.

[0307] For example, the first memory is RAM, such as Double Data Rate (DDR) RAM, and the second memory is L2 Shared Memory. The read latency of L2 Shared Memory is less than that of RAM.

[0308] In other words, storage unit 1020 is DDR memory, or storage unit 1020 includes DDR memory and L2 shared memory.

[0309] The system 1000 also includes a data transport unit for reading data required for neural network operations stored in the storage unit 1020 to one or more processing units 1010. The data transport unit includes an intermediate unit 1040 and / or a broadcast unit 1050, as described below.

[0310] Optionally, such as Figure 13 As shown, the system 1000 also includes one or more intermediate units 1040, each corresponding to one or more processing units 1010. Each intermediate unit 1040 is responsible for transferring data between its corresponding processing unit 1010 and storage unit 1020. This transfer includes reading data from storage unit 1020 to processing unit 1010, or writing the processing result from processing unit 1010 into storage unit 1020.

[0311] Each processing unit 1010 can access the data on the storage unit 1020 through its respective intermediate unit 1040.

[0312] Optionally, such as Figure 13As shown, the system 1000 further includes a broadcast unit 1050 configured to broadcast data stored in the storage unit 1020 to the plurality of processing units 1010.

[0313] For example, in a case where the plurality of processing units 1010 are configured to run the same convolutional neural network simultaneously, the broadcast unit 1050 can read out and broadcast the weights stored in the storage unit 1020 to the plurality of processing units 1010. For example, in a case where the processing units 1010 are configured to run the same convolutional neural network simultaneously, the broadcast unit 1050 can read out and broadcast the weights stored in the storage unit 1020 to the first storage modules 41 (L1 weight Mem) of the plurality of processing units 1010. Figure 7 For example, in a case where the plurality of processing units 1010 are configured to run the same convolutional neural network simultaneously, the broadcast unit 1050 can read out and broadcast the weights stored in the storage unit 1020 to the plurality of processing units 1010. For example, in a case where the processing units 1010 are configured to run the same convolutional neural network simultaneously, the broadcast unit 1050 can read out and broadcast the weights stored in the storage unit 1020 to the first storage modules 41 (L1 weight Mem) of the plurality of processing units 1010.

[0314] For example, the intermediate unit 1040 corresponding to each processing unit 1010 can be referred to as GDMAx, and the broadcast unit 1050 can be referred to as Broadcast GDMA.

[0315] The system 1000 further includes an interconnection structure 1030 configured to transmit data or commands between the processing units 1010 and the storage unit 1020.

[0316] The interconnection structure 1030 can be a bus interconnection structure for command and data distribution.

[0317] Alternatively, the interconnection structure 1030 can also be other forms of interconnection structure for command and data distribution.

[0318] For example, the transmission of data or commands between the plurality of processing units 1010, the storage unit 1020, and the broadcast unit 1050 can be performed through a bus interconnection structure, or through other forms of interconnection structure.

[0319] The interconnection structure 1030 can be referred to as Fabric.

[0320] It should be understood that, in a case where the system 1000 includes a plurality of processing units 1010, the system 1000 provided by the embodiments of the present disclosure can support a plurality of neural networks, and has good versatility.

[0321] It should be understood that the system 1000 provided by the embodiments of the present disclosure can support a plurality of neural networks, and has good versatility. In addition, the system 1000 can perform the operation of a plurality of neural networks in parallel, which can improve the calculation efficiency of the neural networks.

[0322] The system 1000 can be referred to as a neural network dedicated chip architecture.

[0323] If the apparatus 100 or 200 provided by the embodiments of the present disclosure is referred to as a TPU, the processing unit 1010 included in the system 1000 can be referred to as a TPU core. The TPU core in the system 1000 has scalability.

[0324] It should be understood that when the system 1000 includes multiple TPU cores, the computing power of the entire neural network processing chip can be improved.

[0325] The system 1000 provided by the embodiments of the present disclosure includes multiple TPU cores, which can run different neural networks or simultaneously run the same neural network. Such flexible configuration can improve the utilization rate of each TPU core and the overall computing power of the system 1000.

[0326] Based on the above description, the apparatus and system for neural network processing provided by the embodiments of the present disclosure can accelerate the computing speed of the neural network and reduce the computing power consumption.

[0327] The above describes the apparatus embodiments, and the following describes the method embodiments. It should be understood that the description of the method embodiments corresponds to the description of the apparatus embodiments, and therefore, the content not described in detail can be referred to the above apparatus embodiments, which will not be described here for brevity.

[0328] As shown in Figure 14 The embodiments of the present disclosure provide a method for neural network processing, which is applied to the apparatus 100 for neural network processing provided by the above embodiments, the apparatus 100 including a first computing array and a second computing array, and the method including the following steps.

[0329] S1410, performing a first type of neural network operation through the first computing array.

[0330] S1420, performing a second type of neural network operation through the second computing array, the second type of neural network operation being different from the first type of neural network operation.

[0331] For example, Figure 14 The method in the embodiments shown can be performed by the apparatus 100 for neural network processing in the above embodiments, for example, by the control module 30 in the apparatus 100.

[0332] In Figure 14 The first computing array in the embodiments shown is the same as the first computing array 10 in the apparatus 100 in the above embodiments, the second computing array is the same as the second computing array 20 in the apparatus 100 in the above embodiments, the first type of neural network operation is the same as the first type of neural network operation in the above embodiments, and the second type of neural network operation is the same as the second type of neural network operation in the above embodiments. For related content, please refer to the above description, which will not be described here for brevity.

[0333] Optionally, in Figure 14 In the illustrated embodiment, the second type of neural network operation is performed in parallel with the first type of neural network operation.

[0334] Optionally, in Figure 14 In the illustrated embodiment, the method further comprises: reading the weights required by the convolution operation from the first storage module to the first computing array, and reading the feature data required by the convolution operation from the second storage module to the first computing array, the second storage module being arranged separately from the first storage module.

[0335] Optionally, in Figure 14 In the illustrated embodiment, the method further comprises: reading the data required by the second type of neural network operation from the second storage module to the second computing array.

[0336] Optionally, in Figure 14 In the illustrated embodiment, reading the weights required by the convolution operation from the first storage module to the first computing array, and reading the feature data required by the convolution operation from the second storage module to the first computing array, comprises: reading the weights and the feature data required by the convolution operation from the first storage module and the second storage module in parallel to the first computing array.

[0337] Optionally, in Figure 14 In the illustrated embodiment, reading the feature data required by the convolution operation from the second storage module to the first computing array comprises: reading the feature data required by the convolution operation from the cache memory to the first computing array, the cache memory being used to cache the feature data required by the convolution operation stored in the second storage module.

[0338] Optionally, in Figure 14 In the illustrated embodiment, reading the feature data required by the convolution operation from the cache memory to the first computing array comprises: determining whether the cache memory has the feature data required by the convolution operation performed by the first computing array; if the determination result is yes, reading the feature data required by the convolution operation from the cache memory to the first computing array; if the determination result is no, reading the required feature data from the second storage module to the cache memory, and reading the feature data required by the convolution operation from the cache memory to the first computing array.

[0339] Optionally, in Figure 14 In the illustrated embodiment, the distance from the cache memory to the first computing array is less than the distance from the second storage module to the first computing array.

[0340] In Figure 14In the illustrated embodiment, the first storage module is the same as the first storage module 41 in the device 100 in the above embodiment, the second storage module is the same as the second storage module 42 in the device 100 in the above embodiment, and the cache is the same as the cache 50 in the device 100 in the above embodiment. For details, refer to the above description, which will not be repeated here.

[0341] Optionally, in the method, Figure 14 In the illustrated embodiment, the method further includes: reading data required for the first type of neural network operation from a third storage module into the first computing array, and reading data required for the second type of neural network operation from a fourth storage module into the second computing array, the fourth storage module being arranged separately from the third storage module.

[0342] Optionally, in the method, Figure 14 In the illustrated embodiment, the distance from the third storage module to the first computing array is less than the distance from the fourth storage module to the first computing array; and / or the distance from the fourth storage module to the second computing array is less than the distance from the third storage module to the second computing array.

[0343] In the method, Figure 14 In the illustrated embodiment, the third storage module is the same as the third storage module 43 in the device 100 in the above embodiment, and the fourth storage module is the same as the fourth storage module 44 in the device 100 in the above embodiment. For details, refer to the above description, which will not be repeated here.

[0344] Optionally, in the method, Figure 15 In the illustrated embodiment, the step S1420 includes: performing the second type of neural network operation by the plurality of second computing arrays respectively, i.e., performing the same type of operation in the neural network by the plurality of second computing arrays.

[0345] As Figure 15 The present disclosure provides another embodiment of a method for processing a neural network, which includes the following steps.

[0346] S1510, reading weights required for convolution operation from a first storage module into a first computing array, and reading feature data required for convolution operation from a second storage module into the first computing array, the second storage module being arranged separately from the first storage module.

[0347] S1520, performing convolution operation on the feature data and the weights by the first computing array.

[0348] The method can be performed by the device 200 for processing a neural network in the above embodiment, e.g., by the control module 30 in the device 200.

[0349] In the method, Figure 15In the illustrated embodiment, the first computing array is the same as the first computing array 10 in the device 200 in the above embodiment, the first storage module is the same as the first storage module 41 in the device 200 in the above embodiment, and the second storage module is the same as the second storage module 42 in the device 200 in the above embodiment. For related details, please refer to the above description, which will not be repeated here.

[0350] Optionally, in Figure 15 In the embodiment shown, step S1510 includes: reading the weights and feature data required for convolution operations from the first storage module and the second storage module in parallel into the first computing array.

[0351] Optionally, in Figure 15 In the embodiment shown, step S1510 includes: reading the feature data required for the convolution operation from the cache memory into the first computing array. The cache memory is used to cache the feature data required for the convolution operation stored in the second storage module.

[0352] Optionally, in Figure 15 In the embodiment shown, step S1510 includes: determining whether the cache memory contains the feature data required for the first computing array to perform convolution operation; if the determination result is yes, reading the feature data required for the convolution operation from the cache memory into the first computing array; if the determination result is no, reading the required feature data from the second storage module into the cache memory, and reading the feature data required for the convolution operation from the cache memory into the first computing array.

[0353] Optionally, in Figure 15 In the illustrated embodiment, the distance from the cache memory to the first computing array is less than the distance from the second storage module to the first computing array.

[0354] exist Figure 16 In the embodiment shown, the cache memory is the same as the cache memory 50 in the device 200 in the above embodiment. For relevant details, please refer to the above description and we will not repeat them here.

[0355] Figure 16 A method for neural network processing, as provided in another embodiment of this disclosure, includes the following steps.

[0356] S1610, read the data required for neural network operation from the storage unit to one or more processing units, the processing unit being the neural network processing device 100 or 200 in the above embodiment.

[0357] S1620 performs neural network operations through one or more processing units.

[0358] For example, Figure 16The method in the illustrated embodiment can be performed by the intermediate unit 1040 or the broadcast unit 1050 in the system 1000 of the above embodiment.

[0359] Optionally, in Figure 16 In the illustrated embodiment, the step S1610 comprises: performing, by the plurality of processing units, a plurality of neural network operations in parallel.

[0360] Optionally, in Figure 16 In the illustrated embodiment, in a case where the plurality of processing units are used to perform operations of the same neural network, the step S1610 comprises: broadcasting, to the plurality of processing units, data stored in the storage unit and shared by the plurality of processing units.

[0361] Optionally, in Figure 7 In the illustrated embodiment, different processing units in the plurality of processing units are used to perform operations of different neural networks.

[0362] By way of example, and without limitation, below, reference is made to Figure 9 With Figure 7 Two examples are given.

[0363] Reference is made to Figure 7 The components involved in the convolution operation include the following.

[0364] 1) Second control unit 32 (Control Logic), responsible for instruction parsing, control of the calculation process and transmission of operands. The second control unit 32 is located outside the convolution acceleration unit (Cube Top).

[0365] For example, for a convolution instruction, the second control unit 32 is mainly responsible for reading the input feature map (IFMAP) from the second storage module 42 (L1 Data Mem) to the first calculation array 10 (Cube) for convolution calculation, and caching the final calculation result of the first calculation array 10 to the second storage module 42.

[0366] 2) Second storage module 42 (L1 Data Mem), mainly used to store data required in the instruction calculation process, and calculation results. The second storage module 42 is located outside the Cube Top.

[0367] For example, for a convolution instruction, the L1 Data Mem stores the input feature map (IFMAP) and the output feature map (DFMAP) obtained after the first calculation array 10 calculates.

[0368] 3) First control unit 31 (also can be called Cube Control Logic), mainly responsible for the control of the convolution calculation process, and the transport of the weight (or called kernel data) required by the convolution process. The first control unit 31 is located in the Cube Top.

[0369] For example, the second control unit 32 located outside the Cube Top and the first control unit 31 located inside the Cube Top can have some handshake signals to cooperate to complete the transmission of the input feature map (IFMAP) and the weight (Weight).

[0370] 4) First storage module 41 (L1 Weight Mem), used to store the weight (or called kernel data) required by the convolution instruction. The first storage module 41 is located in the Cube Top.

[0371] 5) Cache 50 (IFMAP Cache), used to cache the IFMAP read from the L1 Data Mem. The cache 50 is located in the Cube Top.

[0372] 6) First calculation array 10 (Cube), used to perform convolution operation.

[0373] The first calculation array 10 is a 3D MAC array. Its three dimensions respectively represent the number of channels of the IFMAP that can be calculated, the number of channels of the OFMAP, and the number of OFMAP HWs.

[0374] Assuming that the maximum values of the three dimensions are IC, OC, and OHW respectively, the number of MACs contained in the first calculation array 10 is IC*OC*OHW. The specific settings of the three parameters can refer to different application scenarios of the chip, so that the first calculation array 10 can be used to a higher utilization.

[0375] Because the second storage module 42 is far away from the first calculation array 10, the time required by the first calculation array 10 to read the IFMAP is longer, resulting in higher power consumption.

[0376] Because the IFMAP data in the convolution calculation has good locality, the locality of the IFMAP data can be used to cache the IFMAP data through the IFMAP Cache, so as to reduce the number of times of reading the second storage module 42 by the first calculation array 10, and further reduce the time and power consumption of the convolution calculation process.

[0377] As an example, the following is used Figure 9The process of convolution operation performed by the illustrated convolution acceleration unit is as follows.

[0378] First, the first control unit 31 reads the weight required for the convolution operation from the first storage module 41 to the first computation array 10.

[0379] Then, the second control unit 32 determines whether the data to be read is already in the cache memory 50 before sending a read command to the second storage module 42. If it is, the data is read from the cache memory 50 directly to the first computation array 10. If not, the second control unit 32 sends a read command to the second storage module 42 so that the IFMAP is read from the second storage module 42 to the cache memory 50 and the first computation array 10.

[0380] Optionally, the operation of the first control unit 31 reading the weight required for the convolution operation can be performed simultaneously with the operation of the second control unit 32 reading the feature data required for the convolution operation.

[0381] After the convolution operation of the first computation array 10 is completed, the second control unit 32 stores the OFMAP obtained by the first computation array 10 in the second storage module 42.

[0382] As shown in Figure 9 The structure for performing vector operation can be composed of a series of vector computation unit (Vector unit) arrays (such as the second computation array 20 as shown in Figure 9 ).

[0383] As shown in Figure 9 A block (Lane) contains a group of second computation arrays 20 and storage units (which can be abbreviated as Mem in Figure 9 ). A group (Array) contains several blocks (Lane), and several groups (Array) form a larger array, which can be called a block group (lane array).

[0384] Referring to Figure 9 , the components involved in vector operation include the following.

[0385] 1) The second control unit 32 (Control Logic) is responsible for instruction parsing, computation flow control, and operand transmission. The second control unit 32 is located outside the block group (lane array).

[0386] 2) The array interconnection structure 60 (Array Fabx) is responsible for the transmission of commands and data between groups (Array) and within groups (Array). The array interconnection structure 60 is located within the block group (lane array).

[0387] 3) Second compute array 20 (vector compute unit array) for performing vector operations.

[0388] 4) Memory (Mem) for storing data required in the instruction computing process and the computing result. The memory (Mem) is located in the lane array.

[0389] Figure 9 The memory (Mem) shown in FIG. 2 is logically a memory module: second memory module 42.

[0390] The memory (Mem) in each lane can be uniformly addressed.

[0391] For example, for a convolution instruction, data corresponding to an address needs to be collected from the lane array to the first compute array 10.

[0392] For example, for a pooling instruction, data corresponding to an address needs to be collected from the lane array to the second compute array 20.

[0393] As an example, the structure shown in FIG. 2 performs a vector (Vector) operation. The process of performing a pooling (Pooling) operation is as follows. ​

[0394] First, the second control unit 32 parses the instruction to obtain the identification (ID) of the lane to be involved in the computation as IDx, and sends the instruction of reading the input feature map (IFMAP) and the pooling frame (Kernel) to the Mem of the lane identified as IDx through two read channels r0 and r1. Among them, the r0 channel is responsible for reading the IFMAP, and the r1 is responsible for reading the pooling frame (Kernel).

[0395] After the second control unit 32 reads the IFMAP and the Kernel from the Mem, the pooling operation is performed in the second compute array 20 of the lane identified as IDx.

[0396] After the second compute array 20 of the lane identified as IDx completes the pooling operation, the second control unit 32 sends a write command through the write port to write the computing result of the pooling operation to the Mem of the lane identified as IDx.

[0397] ​The terminology used in the description herein is for the purpose of describing the embodiments and is not intended to be limiting of the claims. As used in the description of the embodiments and the claims herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Similarly, the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. Additionally, the term "comprise" and variations of the term, such as "comprises" and / or "comprising", as used in this application are used in the sense of "including", but do not preclude the presence of excluded elements, integers, steps, operations, elements, and / or groups thereof.

[0398] Aspects, implementations, or features described in the embodiments can be used individually or in any combination. Aspects in the described embodiments can be implemented by software, hardware, or a combination of software and hardware. The described embodiments can also be embodied in a computer-readable medium storing computer readable code, which can be executed by at least one computing device. The computer-readable medium can be associated with any data storage capable of storing data which can be read by a computer system. The computer-readable medium for example can include read-only memory, random access memory, CD-ROMs, HDDs, DVDs, magnetic tape, optical data storage devices, etc. The computer-readable medium can also be distributed over a network of computer systems so that the computer readable code is stored and executed in a distributed fashion.

[0399] The above description of the technology can be related to the accompanying drawings, which form a part of the detailed description, and which illustrate embodiments in accordance with the described technology. Although the described embodiments are sufficiently detailed to enable one of ordinary skill in the art to practice the embodiments, the embodiments are non-limiting; thus, other embodiments can be used and changes can be made without departing from the scope of the described technology. For example, the order in which operations are described is non-limiting, and thus, the order of the operations described in the flowcharts can be changed, according to several embodiments. As another example, one or more of the operations described in the flowcharts can be optional, or can be deleted, in several embodiments. Additionally, certain steps or functions can be added to the disclosed embodiments, or certain steps or functions can be removed from the disclosed embodiments. All such changes and modifications are intended to be included within the scope of the disclosed embodiments and the claims.

[0400] In addition, the terminology used in the description presented above is intended to be interpreted in a descriptive sense, purely for the purpose of providing a clear and comprehensive understanding of the described embodiments. However, the above description is presented in terms of embodiments that are presented as examples. The described embodiments are presented in terms of a particular sequence of steps; however, this should not be interpreted in a limiting manner. Many modifications and variations to the described embodiments can be made in light of the above teachings. It is, therefore, to be understood that a variety of modifications, substitutions, and changes can be made to the described embodiments without departing from the scope of the disclosure. Although the described embodiments have been presented in terms of specific examples, it is not intended that the described embodiments be limited to the specific examples presented. The described embodiments are meant to be exemplary and are not intended as limiting.

Claims

1. A neural network processing device, characterized in that, include: The first computing array is used to perform a first type of neural network operation, which is a convolution operation, and the first computing array is a three-dimensional multiply-accumulate 3D MAC array. The second computing array is used to perform a second type of neural network operation, which is different from the first type of neural network operation. The first storage module is used to store the weights required for convolution operations; The second storage module, which is separate from the first storage module, is used to store the feature data required for convolution operations and also to store the data required for the second type of neural network operations; A high-speed cache memory is used to cache the feature data required for convolution operations read from the second storage module; A control module is configured to control the first computing array to perform the first type of neural network operation and to control the second computing array to perform the second type of neural network operation. The control module includes: The first control unit is configured to read the weights required for the convolution operation from the first storage module and store them in the first computing array. The second control unit is used to determine whether the high-speed cache memory contains the feature data required by the first computing array to perform the convolution operation when the first computing array is about to perform the convolution operation. If the determination result is yes, the feature data required for the convolution operation is read from the high-speed cache memory and added to the first computing array; If the determination result is negative, the required feature data is read from the second storage module into the cache memory, and the feature data required for the convolution operation is read from the cache memory into the first computing array. The second control unit is also configured to read the data required for the second type of neural network operation from the second storage module and store it in the second computing array; The operation of the first control unit to read the weights required for the convolution operation and the operation of the second control unit to read the feature data required for the convolution operation are performed simultaneously; after the convolution operation of the first computing array is completed, the second control unit stores the calculation result obtained by the first computing array into the second storage module; The second control unit can communicate with the first control unit. The module or region where the first computing array and the first storage module are located is referred to as the convolution acceleration unit. The second control unit is located outside the convolution acceleration unit.

2. The apparatus according to claim 1, characterized in that, The control module is used to control the first computing array and the second computing array, so that the first type of neural network operation and the second type of neural network operation are executed in parallel.

3. The apparatus according to claim 1 or 2, characterized in that, The second type of neural network operation includes vector operations.

4. The apparatus according to claim 1 or 2, characterized in that, The device further includes: The third storage module is used to store the data required for the operation of the first type of neural network. A fourth storage module, separate from the third storage module, is used to store the data required for the second type of neural network operation; The control module is used for: Read the data required for the first type of neural network operation from the third storage module and store it in the first computing array; The data required for the second type of neural network operation is read from the fourth storage module and stored in the second computing array.

5. The apparatus according to claim 4, characterized in that, The distance from the third storage module to the first computing array is less than the distance from the fourth storage module to the first computing array; and / or The distance from the fourth storage module to the second computing array is less than the distance from the third storage module to the second computing array.

6. The apparatus according to claim 1 or 2, characterized in that, The distance from the cache memory to the first computing array is less than the distance from the second storage module to the first computing array.

7. The apparatus according to claim 1 or 2, characterized in that, The device includes a plurality of the second computing arrays.

8. The apparatus according to claim 1 or 2, characterized in that, The device is a tensor processor (TPU).

9. A neural network processing system, characterized in that, include: One or more processing units, wherein the processing units are neural network processing apparatus as described in any one of claims 1 to 8; Storage unit, used to store data required by the one or more processing units to perform neural network operations; A data transfer unit is used to read the data required for neural network operations stored in the storage unit to the one or more processing units.

10. The system according to claim 9, characterized in that, The system includes the plurality of processing units, which are used to perform neural network operations in parallel.

11. The system according to claim 9 or 10, characterized in that, The data transport unit includes: One or more intermediate units, each corresponding to one or more processing units, wherein each intermediate unit is responsible for transferring data between the corresponding processing unit and the storage unit.

12. The system according to claim 9 or 10, characterized in that, The system includes the plurality of processing units; The data transport unit includes: A broadcasting unit is used to broadcast data shared by the multiple processing units stored in the storage unit to the multiple processing units when the multiple processing units are used to perform the operation of the same neural network.

13. The system according to claim 9 or 10, characterized in that, The storage unit includes one or more of the following: a first memory and a second memory, wherein the read speed of the second memory is greater than the read speed of the first memory.

14. The system according to claim 13, characterized in that, The first memory is RAM, and the second memory is L2.

15. The system according to claim 9 or 10, characterized in that, The system includes the plurality of processing units, wherein different processing units are used to perform operations of different neural networks.

16. The system according to claim 9 or 10, characterized in that, The system also includes: An interconnection structure is used for the transmission of data or commands between the one or more processing units and the storage unit.

17. A method for neural network processing, characterized in that, The method is applied to a neural network processing apparatus, which includes a first computing array, a second computing array, and a control module. The control module includes a first control unit and a second control unit, the second control unit being communicable to the first control unit. The module or region containing the first computing array and the first storage module is referred to as a convolution acceleration unit. The second control unit is located outside the convolution acceleration unit and includes: The first type of neural network operation is performed through the first computing array, where the first computing array is a three-dimensional multiply-accumulate 3D MAC array and the first type of neural network operation is a convolution operation. The second type of neural network operation is performed through the second computing array, and the second type of neural network operation is different from the first type of neural network operation. The first control unit reads the weights required for convolution operations from the first storage module into the first computing array, the second control unit reads the feature data required for convolution operations from the second storage module into the first computing array, and the second control unit reads the data required for the second type of neural network operations from the second storage module into the second computing array. The second storage module is set separately from the first storage module. The step of reading the feature data required for the convolution operation from the second storage module and storing it in the first computing array by the second control unit includes: When the first computing array needs to perform a convolution operation, the second control unit reads the feature data required for the convolution operation from the cache memory into the first computing array. The cache memory is used to cache the feature data required for the convolution operation read from the second storage module. Wherein, when the first computing array is to perform a convolution operation, the second control unit reads the feature data required for the convolution operation from the cache memory into the first computing array, including: Determine whether the cache memory contains the feature data required for the first computing array to perform convolution operations; If the determination result is yes, the feature data required for the convolution operation is read from the high-speed cache memory and added to the first computing array; If the determination result is negative, the required feature data is read from the second storage module into the cache memory, and the feature data required for the convolution operation is read from the cache memory into the first computing array. The operation of the first control unit to read the weights required for the convolution operation and the operation of the second control unit to read the feature data required for the convolution operation are performed simultaneously; after the convolution operation of the first computing array is completed, the second control unit stores the calculation result obtained by the first computing array into the second storage module.

18. The method according to claim 17, characterized in that, The second type of neural network operation is executed in parallel with the first type of neural network operation.

19. The method according to claim 17 or 18, characterized in that, The second type of neural network operation includes vector operations.

20. The method according to claim 17 or 18, characterized in that, The method further includes: The third storage module reads the data required for the operation of the first type of neural network into the first computing array, and the fourth storage module reads the data required for the operation of the second type of neural network into the second computing array. The fourth storage module is set separately from the third storage module.

21. The method according to claim 20, characterized in that, The distance from the third storage module to the first computing array is less than the distance from the fourth storage module to the first computing array; and / or The distance from the fourth storage module to the second computing array is less than the distance from the third storage module to the second computing array.

22. The method according to claim 17 or 18, characterized in that, The distance from the cache memory to the first computing array is less than the distance from the second storage module to the first computing array.

23. The method according to claim 17 or 18, characterized in that, The device includes a plurality of the second computing arrays; The step of performing the second type of neural network operation through the second computing array includes: The second type of neural network operation is performed through multiple second computing arrays.

24. A method for neural network processing, characterized in that, include: The data required for neural network operations is read from the storage unit to one or more processing units, wherein the processing unit is a neural network processing apparatus as described in any one of claims 1 to 8; Neural network operations are performed through the one or more processing units respectively.

25. The method according to claim 24, characterized in that, The execution of neural network operations through the one or more processing units includes: Multiple neural network operations are executed in parallel through the aforementioned processing units.

26. The method according to claim 24 or 25, characterized in that, When the multiple processing units are used to perform the operation of the same neural network, reading the data required for the neural network operation from the storage unit to one or more processing units includes: The data shared by the plurality of processing units stored in the storage unit is broadcast to the plurality of processing units.

27. The method according to claim 24 or 25, characterized in that, Different processing units among the multiple processing units are used to perform operations on different neural networks.

Images