Method implemented by processing unit, readable storage medium and processing unit

Abstract

The present disclosure relates to a method implemented by a processing unit, a readable storage medium, and a processing unit. The processing unit includes a command parser and at least one core, the command parser is used to assign commands and computing tasks, and the at least one core is coupled to the command parser for communication and is used to process the assigned computing tasks. Each core may include a convolution unit, a pooling unit, at least one operation unit, and a sequencer coupled to the convolution unit, the pooling unit, and the at least one operation unit for communication and for assigning instructions of the assigned computing tasks to the convolution unit, the pooling unit, and the at least one operation unit for execution. The method may include: reading data from a local memory of at least one operation unit by the convolution unit; performing a convolution operation on the data by the convolution unit to generate a feature map; and performing a pooling operation on the feature map by the pooling unit.

Description

[0001] This disclosure claims priority to U.S. Provisional Application No. 62 / 904,953, filed on September 24, 2019, which is incorporated herein by reference in its entirety. Technical Field

[0002] The embodiments of the present disclosure relate to the field of neural networks, and in particular to a method implemented by a processing unit, a readable storage medium, and a processing unit. Background Art

[0003] In terms of machine learning (ML) or deep learning (DL), a neural network (NN) is a very powerful architecture that can roughly mimic the way the human brain learns. Deep neural network (DNN) is a classification of neural networks. Over the years, DNN has achieved great success in fields such as computer vision and natural language processing. A typical DNN model may have millions of parameters, which requires a lot of computing and storage resources to train and deploy the model. The development of modern large-scale parallel processing devices has provided opportunities for the implementation of DNN technology in various applications.

[0004] Ten years ago, general-purpose graphics processing units (GPGPUs) were developed to accelerate scientific computing. Currently, GPUs are widely used to implement DNN technology. Although the resource utilization of GPUs continues to increase with the computational demands of DNNs, it is still largely suboptimal for many reasons. For example, the memory hierarchy of GPUs limits fast storage on the chip, while DNNs require fast access to massive amounts of data. In addition, GPUs maintain a comprehensive general-purpose instruction set, which requires additional resources, while DNNs only require a small number of dedicated programmable operations. Summary of the invention

[0005] In some embodiments, an exemplary method implemented by a processing unit is provided, wherein the processing unit includes a command parser and at least one core, the command parser is used to assign commands and computing tasks, the at least one core is coupled to the command parser for communication and is used to process the assigned computing tasks, each core includes a convolution unit, a pooling unit, at least one operation unit and a sequencer, the sequencer is coupled to the convolution unit, the pooling unit and the at least one operation unit for communication and is used to assign instructions of the assigned computing tasks to the convolution unit, the pooling unit and the at least one operation unit for execution, the method including: reading data from a local memory of at least one operation unit by the convolution unit; performing a convolution operation on the data by the convolution unit to generate a feature map; and performing a pooling operation on the feature map by the pooling unit.

[0006] In some embodiments, an exemplary non-temporary computer-readable storage medium is provided, the readable storage medium storing a set of instructions that can be executed by at least one processing unit to enable a computer to execute a method, the processing unit including a command parser and at least one core, the command parser being used to assign commands and computing tasks, at least one core being coupled and communicated with the command parser and being used to process the assigned computing tasks, each core including a convolution unit, a pooling unit, at least one operation unit and a sequencer, the sequencer being coupled and communicated with the convolution unit, the pooling unit and the at least one operation unit and being used to assign instructions of the assigned computing tasks to the convolution unit, the pooling unit and the at least one operation unit for execution, the method including: reading data from a local memory of at least one operation unit by the convolution unit; performing a convolution operation on the data by the convolution unit to generate a feature map; and performing a pooling operation on the feature map by the pooling unit.

[0007] In some embodiments, an exemplary processing unit is provided, which includes: a command parser for assigning commands and computing tasks; and at least one core coupled and communicated with the command parser, for processing the assigned computing tasks, each core including: a convolution unit having a circuit for performing a convolution operation through a convolution instruction to generate a feature map; a pooling unit having a circuit for performing a pooling operation on the feature map through a pooling instruction; at least one operation unit having a circuit for processing data; and a sequencer coupled and communicated with the convolution unit, the pooling unit and the at least one operation unit, and having a circuit for assigning instructions of the assigned computing tasks to the convolution unit, the pooling unit and the at least one operation unit for execution.

[0008] Part of the additional features and advantages of the present disclosure will be described in detail below, and part of it will be apparent from the detailed description, or can be understood through the practice of the present disclosure. The features and advantages of the present disclosure will be realized and obtained through the elements and combinations particularly pointed out in the appended claims.

[0009] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings, which constitute a part of this specification, illustrate several embodiments and, together with the description, are used to explain the principles and features of the various embodiments of the present disclosure. In the drawings:

[0011] Figure 1 is a schematic diagram of a neural network according to some embodiments of the present disclosure.

[0012] Figure 2is a schematic diagram of an exemplary pipeline workflow for reasoning based on a neural network according to some embodiments of the present disclosure;

[0013] Figure 3A is a schematic diagram of a fragment of a building block in an exemplary convolutional neural network (CNN) according to some embodiments of the present disclosure;

[0014] Figure 3B is a schematic diagram of a fragment of a building block in another exemplary convolutional neural network (CNN) according to some embodiments of the present disclosure;

[0015] Figure 4 is a schematic diagram of an exemplary neural network processing unit (NPU) according to some embodiments of the present disclosure.

[0016] Figure 5A is a schematic diagram of an exemplary machine learning system according to some embodiments of the present disclosure;

[0017] Figure 5B is a schematic diagram of a multi-layer software architecture according to some embodiments of the present disclosure;

[0018] Figure 5C is a schematic diagram of an exemplary cloud system including an NPU according to some embodiments of the present disclosure;

[0019] Fig. 6A is a schematic diagram of an exemplary inference workflow of a core of an NPU according to some embodiments of the present disclosure;

[0020] Figure 6B is a schematic diagram of an exemplary inference workflow of a core of an NPU according to some embodiments of the present disclosure;

[0021] Figure 7 is a schematic diagram of an exemplary neural network workflow according to some embodiments of the present disclosure;

[0022] Figure 8 is a schematic diagram of exemplary data movement in a core of an NPU according to some embodiments of the present disclosure;

[0023] Fig. 9 is a schematic diagram of the workflow between various processing units in the core of an NPU according to some embodiments of the present disclosure;

[0024] Fig.10 shows exemplary instructions of an NPU according to some embodiments of the present disclosure;

[0025] Fig.11 is a flow chart of an exemplary method implemented by a processing unit according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0026] Exemplary embodiments will be described in detail below, and exemplary drawings thereof are shown in the accompanying drawings. The following description will refer to the accompanying drawings, in which the same numbers in different drawings represent the same or similar elements, unless otherwise indicated. The implementations set forth below in the description of the exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices, systems, and methods consistent with various aspects related to the present invention as described in the appended claims.

[0027] The devices and systems disclosed herein can be used in various neural network-based architectures, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), etc., and can be configured as architectures such as neural network processing units (NPUs), etc.

[0028] Figure 1 is a schematic diagram of a neural network (NN) 100. Figure 1 As shown, neural network 100 may include input layer 120 for receiving inputs such as input 110-1 to input 110-m. The inputs may include images, text, or any other structured or unstructured data processed by neural network 100. In some embodiments, neural network 100 may accept multiple inputs simultaneously. For example, in Figure 1 In the example of FIG. 1 , neural network 100 may receive up to m inputs simultaneously. Additionally or alternatively, input layer 120 may receive up to m inputs in rapid succession, e.g., input 110-1 is an input received by input layer 120 in one cycle, a second input is an input received by input layer 120 in a second cycle, in which input layer 120 pushes data from input 110-1 to a first hidden layer, and so on. Any number of inputs may be simultaneous inputs, rapid and continuous inputs, and so on.

[0029] The input layer 120 may include one or more nodes, for example, node 120-1, node 120-2, ..., node 120-a. Each node may apply an activation function to a corresponding input (e.g., one or more of input 110-1 to input 110-m) and weight the output of the activation function by a specific weight associated with the node. The activation function may include a Heaviside step function, a Gaussian function, a multiquadratic function, an inverse multiquadratic function, a sigmoidal function, a ReLU function, a Leaky ReLU function, a Tanh function, and the like. The weight may include a positive value between 0.0 and 1.0, or any other value that allows the output of some nodes in a layer to be larger or smaller than the output of other nodes in the layer.

[0030] like Figure 1 As further described, the neural network 100 may include one or more hidden layers, such as hidden layers 130-1 to 130-n. Each hidden layer may include one or more nodes. Figure 1 , hidden layer 130-1 includes node 130-1-1, node 130-1-2, node 130-1-3, ..., node 130-1-b, and hidden layer 130-n includes node 130-n-1, node 130-n-2, node 130-n-3, ..., node 130-nc. Similar to the nodes of input layer 120, the nodes of the hidden layer can apply an activation function to the output of the connected nodes of the previous layer and weight the output of the activation function by a specific weight associated with the node.

[0031] like Figure 1 As further described in , the neural network 100 may include an output layer 140 for completing outputs (e.g., output 150-1, output 150-2, ..., output 150-d). The output layer 140 may include one or more nodes, for example, node 140-1, node 140-2, ..., node 140-d. Similar to the nodes of the input layer 120 and the hidden layer, the nodes of the output layer 140 may apply an activation function to the outputs of the connected nodes of the previous layer and weight the outputs of the activation function by specific weights associated with the nodes.

[0032] Despite Figure 1 Although the full connection is described in the figure, the various layers 100 of the neural network can use any connection scheme. For example, one or more layers (e.g., input layer 120, hidden layer 130-1, ..., hidden layer 130-n, output layer 140, etc.) can be connected using a convolution scheme, a sparse connection scheme, or a similar connection scheme. Such an embodiment uses a ratio between the current layer and the previous layer. Figure 1 Fewer connections are shown.

[0033] In addition, despite the Figure 1 Although a feed-forward network is described in , neural network 100 may additionally or alternatively use back propagation (e.g., by using long short-term memory nodes or similar nodes). Thus, although neural network 100 is depicted as being similar to a convolutional neural network (CNN), neural network 100 may include a recurrent neural network (RNN) or any other neural network.

[0034] Generally speaking, there are two phases in the deep learning workflow for neural networks: training and inference. During the training phase, the neural network continuously learns parameter values ​​by iteratively updating the parameter values ​​to minimize the prediction error. When converged, the neural network with the learned parameter values ​​can be used to perform inference tasks on new cases.

[0035] Figure 2FIG. 2 is a schematic diagram of an exemplary neural network-based reasoning workflow 200 according to some embodiments of the present disclosure. Although the reasoning workflow 200 involves image recognition, it should be understood that this is only an example and not a limitation. Figure 2 As shown, the trained neural network (for example, Figure 1 A neural network 100 may receive an input 201, such as an image of a honey badger, and perform a computation 203 on the input 201. Specifically, forward propagation (FP) begins in the neural network, where data flows from the input layer through one or more hidden layers to the output layer. Figure 1 As shown, each layer in the neural network receives input from the previous layer (or layers), performs calculations on the input, and sends outputs to the next layer (or layers). After calculation, the neural network provides output 205, such as evaluation results. Figure 2 As shown, the output 205 may include a plurality of possible evaluation items with respective probabilities. The item with the highest probability may be determined as the final evaluation result.

[0036] Convolutional Neural Network (CNN) is a classification of DNN. CNN is widely used in many technical fields. For example, CNN can perform visual tasks, such as learning or recognizing image features or patterns.

[0037] Figure 3A A fragment 310 of a building block in an exemplary CNN is shown. For example, the exemplary fragment 310 may be an initial module. Figure 3A As shown, segment 310 may include multiple branches in parallel, such as convolution branches 311, 313, 315 and pooling branch 317. Convolution branch 311 includes a 1×1 convolution block (CONV). Convolution branch 313 includes a 3×3 convolution block (CONV) and a 1×1 convolution block (CONV) located before it. Convolution branch 315 includes a 5×5 convolution block (CONV) and a 1×1 convolution block (CONV) located before it. Pooling branch 317 includes a 3×3 pooling block (POOL) and a 1×1 convolution block (CONV) located after it. For example, the pooling block may be a 3×3 maximum pooling block. Together with each convolution block, there may be a batch normalization block (BN) and an activation block (ReLU). For example, the activation block may be ReLU, Leaky ReLU, Sigmoid, Tanh, etc.

[0038] like Figure 3AAs shown, the segment 310 may further include a connection block (CONCAT) 319. The connection block 319 may be connected to a plurality of branches, for example, branches 311, 313, 315, and 317. The branch may receive input from the previous layer (previous layers) and perform calculations. The connection block 319 may connect the results from the convolution branches 311, 313, 315 and the pooling branch 317 together and provide the results to other blocks or layers. The CNN may include a plurality of segments 310, an input layer, an output layer, and one or more other layers.

[0039] Figure 3B A fragment 330 of a building block in another exemplary CNN is shown. For example, the exemplary CNN may be a residual network. Figure 3B As shown, segment 330 may include multiple branches, such as branch 331 and convolution branch 333. Convolution branch 333 includes 1×1 convolution (CONV) block 333-1, 3×3 convolution block 333-2, and 3×3 convolution block 333-3. Convolution branch 333 receives input from the previous layer (previous layers) and performs calculations on the input. Branch 331 includes a jump connection across convolution branch 333. Segment 330 also includes an addition block 335, which receives input from branches 331 and 333 and performs addition. In addition, segment 330 may also include one or more batch normalization (BN) blocks and activation blocks (e.g., ReLU blocks). CNN may include multiple segments 330, an input layer, an output layer, and one or more other layers.

[0040] Figure 4 An exemplary neural processing unit (NPU) 400 is shown according to some embodiments of the present disclosure. Figure 4 As shown, NPU 400 may include at least one core 402 (e.g., 402a, 402b, 402c, and 402d), an interface 404, a command parser (CP) 406, a direct memory access (DMA) unit 408, etc. It is understood that NPU 400 may also include a bus 410, a global memory (not shown), and the like.

[0041] The interface 404 may provide communication between the NPU 400 and external devices. For example, the interface 404 may include a high-speed serial computer expansion bus standard (PCI-E) interface, which provides communication with a host unit ( Figure 4 The interface 404 may also include at least one of a universal serial bus (USB), a joint test action group (JTAG) interface, a TUN / TAP interface, and the like.

[0042] The command parser 406 may interact with the host unit under the supervision of the kernel mode driver (KMD) and pass the neural network task, related commands (or instructions) and data to each core 402. The command parser 406 may include a circuit that performs interaction with the host unit and passes the neural network task, related commands (or instructions) and data to each core 402. In some embodiments, the command parser 406 may receive a DMA command from the host unit and load instructions for the neural network (e.g., an instruction sequence for the neural network generated by a compiler in the host unit), weights or scale / bias constants of the neural network to the core 402 according to the DMA command. For example, the command parser 406 may load instructions for the neural network from an external memory to an instruction buffer of the core 402, load weights to a local memory 4022 of the core 402, or load scale / bias constants to a constant buffer of the core 402 according to the DMA command. In some embodiments, the command parser 406 may work with the host unit or the KMD to assign a neural network task (e.g., recognition of an image, including data of the image) to the core 402. For example, the host unit or KMD may send a neural network task to the queue of a core 402 to which the neural network task is assigned, and the command parser 406 assigns the neural network task to the core 402. In some embodiments, when the core 402 completes the neural network task (e.g., the core 402 may send a "computation completed" message to the command parser 406), the command parser 406 may notify the host unit or KMD. A new neural network task may be assigned to the core 402 by the host unit or KMD.

[0043] The DMA unit 408 can assist in transferring data between various components of the NPU 400. The DMA unit 408 can include circuits for executing data transfers or commands. For example, the DMA unit 408 can assist in transferring data between multiple cores (e.g., cores 402a-402d) or within each NPU core. The DMA unit 408 can also allow off-chip devices to access on-chip memory and off-chip memory via the interface 404 without causing interruptions. For example, the DMA unit 408 can load data or instructions into the local memory of the core. Therefore, the DMA unit 408 can also generate memory addresses and initiate memory read or write cycles. The DMA unit 408 can also include several hardware registers that can be written and read by one or more processors, including memory address registers, byte count registers, one or more control registers, and other types of registers. These registers can specify a combination of source, destination, transfer direction (read from or write to an input / output device), the size of the transfer unit, and / or the number of bytes to be transferred in one pulse. It will be appreciated that each core (eg, core 402a) may include a sub-DMA unit that may be used to transfer data within the core.

[0044] DMA unit 408 can also move block data between cores via bus 410. While a single core can handle typical inference tasks (e.g., ResNet50 v1), cores can also work together via the bus to undertake large and complex tasks (e.g., RestNet101, Mask R-CNN, etc.).

[0045] The bus 410 can provide high-speed cross-core communication. The bus 410 also couples each core of the NPU with other units, such as an off-chip memory or a peripheral device.

[0046] Core 402 (e.g., core 402a) may include one or more processing units that may be used to perform one or more operations (e.g., multiplication, addition, multiplication accumulation, element operations, etc.) based on commands (e.g., commands received from command parser (CP) 406). For example, core 402 may receive a neural network task, instructions, and data (e.g., weights or scale / bias constants of a neural network) from command parser 406 and execute the instructions using the data. In some embodiments, when core 402 completes a neural network task, it may notify command parser 406. For example, core 402 may send a "computation completed" message to command parser 406. Figure 4As shown, core 402a may include at least one operation unit 4020, a sequencer 4028, a convolution unit 4030, a pooling unit 4032, and a DMA unit 408a, which may be connected via a data structure and arbitration subsystem (also referred to as a hub unit). In some embodiments, the hub unit may include circuits for providing convolution data and pooling data associated with a neural network task to the convolution unit 4030 and the pooling unit 4032, respectively.

[0047] The operation unit 4020 may include circuits for performing operations on received data (e.g., matrices). In some embodiments, each operation unit 4020 may further include a local memory 4022, a matrix multiplication data path (DP) 4024, and an embedded element operation (EWOP) unit 4026. The local memory 4022 can provide storage space with a fast read / write rate. In order to reduce possible interactions with the global memory, the storage space of the local memory 4022 can be 180 megabytes (MB) and above. With the massive storage space, most data accesses can be performed within the core 402, thereby reducing the latency caused by data access. The DP 4024 may include circuits for performing matrix multiplication (e.g., dot production), and the EWOP unit 4026 may include circuits for performing element operations (e.g., vector and vector multiplication) on the received data. It will be understood that although Figure 4 Four operating units 4020 are shown, but core 402a may include more or fewer operating units 4020.

[0048] Sequencer 4028 may be coupled to the instruction buffer and include circuitry for retrieving instructions (or commands) and distributing the instructions to components such as core 402. For example, sequencer 4028 may include circuitry for distributing a convolution instruction to convolution unit 4032 to perform a convolution operation or a pool instruction to pooling unit 4033 to perform a pooling operation. In some embodiments, sequencer 4028 may include circuitry for modifying related instructions stored in the instruction buffer of each core 402 so that the various cores 402 can work in parallel as much as possible. Sequencer 4028 may also include circuitry for monitoring the execution of a neural network task and parallelizing subtasks of the neural network task to improve execution efficiency.

[0049] The convolution unit 4030 may be coupled to the sequencer 4028 and one or more operation units 4020, and include circuitry for instructing the one or more operation units 4020 to perform a convolution operation. In some embodiments, the convolution unit 4030 may send a command to the local memory 4022 to send activation data and weight data to the data path 4024 to perform a convolution operation.

[0050] The pooling unit 4032 may further include an interpolation unit, a pool data path, etc., and include a circuit for performing a pooling operation. For example, the interpolation unit may include a circuit for interpolating the pooled data. The pool data path may include a circuit for performing a pooling operation on the interpolated pooled data.

[0051] DMA unit 408a may be part of DMA unit 408 or a separate unit for each core. DMA unit 408a includes circuitry for transferring data or commands. A command may be assigned to DMA unit 408a to instruct DMA unit 408a to transfer data from local memory (e.g., Figure 4 The instructions / commands or data of the local memory 4022) are loaded into the corresponding units. Then, the loaded instructions / commands or data are distributed to the processing units to which the corresponding tasks have been assigned, and these one or more processing units can process these instructions / commands.

[0052] Figure 5A is a schematic diagram of an exemplary machine learning system 500 according to some embodiments of the present disclosure. Figure 5A As shown, the machine learning system 500 may include a host processor 502, a disk 504, a host memory 506, and a neural network processing unit (NPU) 400. In some embodiments, the host memory 506 may be an integral memory or an external memory associated with the host processor 502. The host memory 506 may be a local memory or a global memory. In some embodiments, the disk 504 may include an external memory for providing additional memory for the host processor 502.

[0053] The host processor 502 (e.g., an X86 or ARM central processing unit) can be coupled to a host memory 506 and a disk 504 for processing general instructions. The neural network processing unit 400 can be connected to the host processor 502 through a peripheral interface (e.g., interface 404). As described herein, the neural network processing unit (e.g., NPU 400) can be a computing device for accelerating neural network reasoning tasks. In some embodiments, the NPU 400 can be used as a coprocessor of the host processor 502.

[0054] In some embodiments, the compiler may be deployed on a host unit (e.g., Figure 5AThe compiler is a computer program or computer software that converts computer code written in a programming language into instructions for the NPU 400 to create an executable program. In machine learning applications, the compiler can perform various operations, such as preprocessing, lexical analysis, parsing, semantic analysis, conversion of input programs into intermediate representations, initialization of neural networks, code optimization and code generation, or a combination thereof. For example, in the machine learning system 500, the compiler can compile the neural network to generate static parameters, such as the connections between neurons and the weights of neurons.

[0055] As described above, these instructions or commands may be further loaded by the command parser 406 of the NPU 400, temporarily stored in the instruction buffer of the NPU 400, and accordingly allocated (e.g., allocated by the sequencer 4028) to the processing units of the NPU 400 (e.g., the convolution unit 4030, the pooling unit 4032, and the DMA unit 408a).

[0056] It will be appreciated that the first few instructions received by each core of the NPU may instruct the core to load / store data from the host memory 506 to one or more local memories within the core (e.g., Figure 4 Each core of the NPU may then initiate an instruction pipeline, which includes fetching instructions from an instruction buffer (e.g., by a sequencer), decoding instructions (e.g., by a DMA unit) and generating local memory addresses (e.g., corresponding to operands), reading source data, executing or load / store operations, and then writing back results.

[0057] A flexible and easily scalable environment can be built around the NPU 400 using a multi-layer software architecture. Figure 5B A schematic diagram of a multi-tier software architecture 520 is shown according to some embodiments of the present disclosure.

[0058] In order to deploy the neural network model, different neural network topologies built by different neural network frameworks 5211 (e.g., TensorFlow, MxNet, etc.) can be converted into a graphical intermediate expression (graphic IR). The front-end deployment and compiler 527 can start from the graphical IR, perform a series of development and refinement in terms of model quantization 523, segmentation 524, and optimization 525, and then generate an executable file that meets the accuracy requirements while having the best performance. In order to distribute tasks, the runtime (RunTime) layer 526 can act as the only access point for tasks to be distributed to the NPU 400. The RT layer 526 can collaborate with the user mode driver (UMD) 528 to establish task deployment and publish it to the NPU 400 via the kernel mode driver (KMD) 529. The RT layer 526 can also feed timely binding and completion information to the driver to provide the required device and context management for the NPU 400. Since the NPU 400 can provide full visibility on context resources and interact directly with the host at the task level, it can provide robust and consistent results.

[0059] Reference now Figure 5C , Figure 5C is a schematic diagram of an exemplary cloud system including an NPU according to some embodiments of the present disclosure.

[0060] With the assistance of NPU 400, cloud system 540 can provide expanded AI capabilities such as image recognition, facial recognition, translation, 3D modeling, etc.

[0061] It should be understood that the NPU 400 can be deployed in computing devices in other forms. For example, the NPU 400 can also be integrated into computing devices, such as smart phones, tablet computers, and wearable devices.

[0062] Fig. 6A is a schematic diagram of an exemplary inference workflow of a core of an NPU according to some embodiments of the present disclosure. For example, the core of the NPU may be Figure 4 Although the inference workflow 610 involves image recognition, it should be understood that this is only an example and not a limitation. Fig. 6A As shown, the core of the NPU can receive input, such as an image of a honey badger. For example, a DMA unit (not shown) of the core of the NPU (e.g., Figure 4 The DMA unit 408a of the core 402a in the embodiment of the present invention can communicate with external components, such as accessing on-chip or off-chip memory, to receive input data. The DMA unit can load the input data into the local memory (not shown) of the core (e.g., Figure 4The NPU core can execute a neural network to perform calculations with input data. For example, the calculation can be performed by the cooperation of the local memory 4022, sequencer 4028, operation unit 4020, convolution unit 4030, pooling unit 4032 and DMA unit 408a in the core 402a of the figure. By working together, the calculation can be performed uninterruptedly. The NPU core can generate outputs, such as evaluation results. Fig. 6A As shown, the output may include a plurality of possible evaluation items with respective probabilities. The item with the highest probability (e.g., the probability of honey badger is 80%) may be determined as the final evaluation result. For example, the DMA unit may send the output (e.g., the evaluation result) to the outside, such as another core, a host unit, an on-chip or off-chip memory, etc.

[0063] Figure 6B is a schematic diagram of an exemplary inference workflow 630 of a core of an NPU according to some embodiments of the present disclosure. For example, the core of the NPU may be Figure 4 Although the inference workflow 630 involves image recognition, it should be understood that this is only an example and not a limitation. Figure 6B As shown, the core of the NPU can receive a series of inputs, such as a cat image as a first input image 631-1, a car image as a second input image 631-2, a frog image as a third input image 631-3, and a dog image as a fourth input image 631-4. For example, a DMA unit (not shown) of the core of the NPU (e.g., Figure 4 The DMA unit 408 of the core 402a in the NPU can communicate with external components, such as accessing on-chip or off-chip memory, to receive input data. The DMA unit can load the input data into the local memory (not shown) of the core of the NPU (e.g., Figure 4 In the local memory 4022 of the core 402a in the Figure 6B As shown, the core of the NPU (e.g., a DMA unit in the core of the NPU) can receive a first input image 631-1 and execute a neural network to perform a first calculation 633-1 on the first input image 631-1. During the first calculation 633-1, the core of the NPU can receive a second input image 631-2. After the first calculation 633-1, the core of the NPU can perform a second calculation 633-2 on the second input image 631-2. During the second calculation 633-2, the NPU (e.g., a DMA unit of the NPU core) can output the result of the first calculation 633-1 (e.g., a first output 635-1), for example, an evaluation result of a cat, and can also receive a third input image 631-3.

[0064] Similarly, after the second calculation 633-2, the core of the NPU may perform a third calculation 633-3 on the third input image 631-3. During the third calculation 633-3, the NPU may output the result of the second calculation 633-2 (e.g., the second output 635-2), for example, the evaluation result of the car, and may also receive the fourth input image 631-4. After the third calculation 633-3, the core of the NPU may perform a fourth calculation 633-4 on the fourth input image 631-4. During the fourth calculation 633-4, the NPU may output the result of the third calculation 633-3 (e.g., the third output 635-3), for example, the evaluation result of the frog. After the fourth calculation 633-4, the NPU may output the result of the fourth calculation 633-4 (e.g., the fourth output 635-4), for example, the evaluation result of the dog. Therefore, the input of the next input data and the output of the previous calculation result may be performed during the current calculation, and the I / O delay may be effectively hidden by the calculation, and vice versa.

[0065] In some embodiments, calculations, such as calculations 633-1, 633-2, 633-3, or 633-4, may be performed by Figure 4 The local memory 4022, sequencer 4028, operation unit 4020, convolution unit 4030, pooling unit 4032 and DMA unit 408a in the core 402a of the NPU are used for execution. By working together, calculations can be performed uninterruptedly. Figure 6B As shown, the output, such as output 635-1, 635-2, 635-3 or 635-4, may include multiple possible evaluation items with respective probabilities. The item with the highest probability (e.g., the probability of a cat is 80%, the probability of a car is 85%, the probability of a frog is 81%, the probability of a dog is 82%, etc.) may be determined as the final evaluation result. For example, the DMA unit may send the output (e.g., the evaluation result) to the outside, such as another core, a host unit, an on-chip or off-chip memory, etc.

[0066] In some embodiments, two or more layers of a neural network are fused or aggregated, or two or more operations of a neural network task are fused or aggregated. The fused or aggregated layers or fused or aggregated operations can be executed by coarse-grain instructions or high-level instructions. Coarse-grained instructions can reduce the cost of processing instruction streams and increase the effective amount of computation per instruction. In some embodiments, coarse-grained instructions can include flags that control instruction streams. For example, the convolution instruction "CONV" may include a modify flag that allows online modification of various fields of the instruction for runtime binding and control. The pooling instruction "POOL" may include a wait flag for specifying data dependencies between various layers. If the wait flag is not asserted, it means that the layer associated with the instruction can be executed in parallel with the layer specified in the pooling instruction. The branch instruction "BR" may include a synchronization flag for coordinating tasks in different cores. Based on the various flags of the instruction, the operations of the neural network task can be executed together, serially or in parallel, which makes the instruction stream processing compact and efficient.

[0067] Figure 7 is a schematic diagram of an exemplary workflow of a neural network 701 according to some embodiments of the present disclosure. Figure 7 As shown, the neural network 701 may include a plurality of building blocks, for example, an input block 701-1, a 7×7 convolution (CONV) block 701-2, a 3×3 pooling (POOL) block 701-3, a 1×1 convolution block 701-4, a 3×3 convolution block 701-5, a 1×1 convolution block 701-6, a channel connection block 701-7, a 3×3 convolution block 701-8, an element summation (ELM summation) block 701-9, and the like. The 7×7 convolution block 701-2 is connected to the input block 701-1 and the 3×3 pooling block 701-3. The 3×3 pooling block 701-3 is connected to the 1×1 convolution block 701-4, the 3×3 convolution block 701-5, and the 1×1 convolution block 701-6 in parallel. The 1×1 convolution block 701-4 and the 3×3 convolution block 701-5 are connected to the channel connection block 701-7. The 1×1 convolution block 701-6 is connected to the 3×3 convolution block 701-8. The channel connection block 701-7 and the 3×3 convolution block 701-8 are connected to the element summation block 701-9. The element summation block 701-9 can be connected to another block or layer. The neural network 701 may also include multiple batch normalization (BN) blocks and activation blocks (e.g., ReLU blocks). Figure 7 , solid arrows indicate the flow of data through the neural network 701, while dashed arrows indicate dependencies between different blocks.

[0068] Neural network 701 is composed of NPU cores (e.g., Figure 4 ) is executed by any one of the cores 402a-d of the NPU. In workflow 703a, the core of the NPU executes input block 701-1 to receive input. Then, the core executes 7×7 convolution block 701-2 to perform 7×7 convolution on the input, and then executes BN block and ReLU block to perform batch normalization and ReLU respectively. The core of the NPU executes 3×3 pooling block 701-3 to perform 3×3 pooling on the result of the ReLU block. For the result of 3×3 pooling, the core of the NPU executes 1×1 convolution block 701-4 to perform 1×1 convolution and then perform batch normalization operation; executes 3×3 convolution block 701-5 to perform 3×3 convolution and then perform batch normalization operation; executes 1×1 convolution block 701-6 to perform 1×1 convolution and then perform batch normalization and ReLU operations. In the channel connection block 701-7, the core of the NPU performs a cascade of the outputs from the BN block after the 1×1 convolution block 701-4 and the BN block after the 3×3 convolution block 701-5. The core of the NPU performs a 3×3 convolution block 701-8 on the output from the ReLU block after the 1×1 convolution block 701-6 to perform a convolution, followed by a BN operation. The core of the NPU performs an element summation block 701-9 on the outputs from the channel connection block 701-7 and the BN block after the 3×3 convolution block 701-8 to perform a summation, followed by a ReLU operation. The core of the NPU also performs other operations in other blocks or layers and produces outputs. The workflow 703a can be block or layer based and executed by the NPU in a direct manner. In some embodiments, the operations in the first row of the workflow 703a, for example, convolution, can be performed by a convolution unit (for example, Figure 4 The operations in the second row of workflow 703a, such as BN operations, ReLU operations, element operations, and pooling, can be performed by a pooling unit (e.g. Figure 4 Pooling unit 4032), DP (e.g. Figure 4 DP 4024), element operation unit (e.g. Figure 4 The operations in the third row of the workflow 703a, such as connection, can be performed by a DMA unit (e.g., Figure 4 DMA unit 408a) executes.

[0069] In workflow 703b, the NPU core can fuse the BN operation and the ReLU operation with the convolution or element operation. For example, the result of the convolution can be passed to the element operation unit for further processing, such as BN or other element operations, without storing it in the local memory. Figure 7As shown, in workflow 703b, the core of the NPU can perform 7×7 convolution, 3×3 pooling, 1×1 convolution, 3×3 convolution, 1×1 convolution, connection, 3×3 convolution, element operation, etc. in series. Therefore, compared with workflow 703a, workflow 703b can reduce the time used to execute neural network 701.

[0070] In workflow 703c, the core of the NPU can aggregate convolution (e.g., convolution in 3×3 convolution block 701-8) with element operations (e.g., element operations in element summation block 701-9). For example, the result of the convolution can be passed to the element operation unit for element operation without storing it in local memory. As shown in Figure 7, in workflow 703c, the core of the NPU can perform 7×7 convolution, 3×3 pooling, 1×1 convolution, 3×3 convolution, 1×1 convolution, connection, 3×3 convolution, etc. in series. Therefore, compared with workflow 703b, workflow 703c can further reduce the time used to execute neural network 701.

[0071] In workflow 703d, if convolution and connection are not dependent on each other and there is no resource conflict between them, the core of the NPU can perform convolution (e.g., convolution at 1×1 convolution block 701-6) and connection (e.g., connection at channel connection block 701-7) in parallel. Figure 7 As shown, in workflow 703d, the core of the NPU can perform 7×7 convolution, 3×3 pooling, 1×1 convolution, 3×3 convolution in series, 1×1 convolution and connection in parallel, 3×3 convolution, etc. Therefore, compared with workflow 703c, workflow 703d can further reduce the time used to execute neural network 701.

[0072] In workflow 703e, the core of the NPU can perform pooling (e.g., pooling at 3×3 pooling block 701-3) at least partially in parallel with the convolution before it (e.g., the convolution of 7×7 convolution block 701-2) or the convolution after it (e.g., the convolution of 1×1 convolution block 701-4). For example, the core of the NPU (e.g., a sequencer) can monitor the result of the convolution before pooling. If a portion of the result has been completed, the pooling unit can perform a pooling operation on the portion of the result. The core of the NPU can also monitor the pooling result before convolution. If a portion of the result has been completed, the convolution unit can perform a convolution operation on the portion of the result. Figure 7 As shown, in workflow 703e, the core of the NPU can execute in series: 7×7 convolution partially in parallel with 3×3 pooling, the remaining part of 3×3 pooling partially in parallel with 1×1 convolution, the remaining part of 1×1 convolution, 3×3 convolution, 1×1 convolution in parallel with connection, 3×3 convolution, etc. Therefore, compared with workflow 703d, workflow 703e can further reduce the time used to execute neural network 701.

[0073] Figure 8 800 is a schematic diagram of an exemplary data movement in a core of an NPU according to some embodiments of the present disclosure. The core of the NPU may include a local memory and a hub system. The local memory may store data for multiple operations. The hub system may support multiple data streams simultaneously. For example, data movement 800 may be performed by Figure 4 The processing is completed by the hub system of the DP 4024, EWOP unit 4026, convolution unit 4030, pooling unit 4032, DMA unit 408a, local memory 4022 and NPU core 402a.

[0074] like Figure 8 As shown, there may be multiple data streams in the core of the NPU, for example, a convolution read data stream 801, a pooling and storage read data stream 802, an input engine write data stream 803, etc. The convolution read data stream 801 may include one or more components, such as a data path (e.g., Figure 4 DP 4024), convolutional units (e.g., Figure 4 Convolution unit 4030) and EWOP unit (e.g., Figure 4 Thus, the convolution read data stream 801 may include reading data from local memory 806a-806d (e.g., Figure 4 The pooling and storage read data flow 802 may include one or more components, such as a pooling unit (e.g., Figure 4 Pooling unit 4032), DMA unit or xDMA unit (e.g., Figure 4 Thus, pooling and storing read data streams 802 may include reading data from local memories 806a-806d (e.g., Figure 4 The input engine write data flow 803 may involve one or more components, such as a write control unit or back end (WCU / BE), etc. For example, the write control unit or back end may include a WCU or BE for a convolution engine (e.g., Figure 4 convolutional unit 4030), pooling unit (e.g., Figure 4 Pooling unit 4032), DMA unit (e.g., Figure 4 DMA unit 408a), etc. Pooling and storing read data stream 802 may include writing to local memory 806a-806d (e.g., Figure 4The local memory 4022 may contain multiple data, such as convolution write data (CONVw), pooled write data (POOLw), input data (IN) (for example, input data from a host unit), cross-core write data (xDMAw), etc.

[0075] Hub systems (e.g. Figure 4 The hub system of the core 402a of the processor 402a) can coordinate multiple data streams from or to local memory (e.g., local memory 806a to 806d) and form multiple read data groups and write data groups. As shown, after the hub system coordination, data movement 800 may include read data groups 804a to 804f and write data groups 805a to 805b. Each of the read data groups 804a, 804c, 804d and 804f may include one or more weights, activation data, etc. The read data group 804b may include data for element-by-element operations and pooling, etc. The write data group 805a may include one or more convolution write data, pooled write data, input data, etc. The read data group 804e may include data for element operations and pooling, data read by DMA, cross-core read data, etc. The write data group 805b may include one or more convolution write data, pooled write data, cross-core write data (xDMAw), etc.

[0076] In some embodiments, through the cooperation of the hub system and other components, the core of the NP can exploit data locality and channel coalescing and provide a well-balanced bandwidth, computation, or parallel multitasking solution.

[0077] Fig. 9 is a schematic diagram of the workflow between various processing units in the core of an NPU according to some embodiments of the present disclosure.

[0078] like Fig. 9 As shown, a sequencer (e.g., Figure 4 The sequencer 4028 of the NPU can retrieve instructions from the instruction buffer and distribute the instructions to the cores of the NPU (e.g., Figure 4 In some embodiments, the sequencer may also modify the instructions before issuing them. The modified instructions may be sent to the convolution units (e.g., Figure 4 Convolution unit 4030), a pooling unit for pooling operation (e.g., Figure 4 Pooling unit 4032) and a DMA unit for data transfer (e.g., Figure 4 DMA unit 408a).

[0079] For example, a convolution unit can be coupled to a sequencer, a matrix multiplication data path (e.g., Figure 4 data path 4024) and element operation units (e.g., Figure 4The convolution unit is coupled to the element operation unit 4026 of the local memory and is configured to instruct the matrix multiplication data path and the element operation unit to perform a convolution operation. In some embodiments, the convolution unit may also send a command to a local memory (e.g., local memory 4022) to send activation data and weight data to the data path to perform a convolution operation. For example, the convolution unit may send a read address of weight data to the local memory and retrieve the corresponding weight data from the local memory through a DMA unit and a data structure (datafabric) and an arbitration subsystem. The data path may then perform matrix multiplication on the activation data and the weight data. It will be appreciated that more than one data path may work together to produce the result of a matrix multiplication. As shown in FIG. Fig. 9 As shown, matrix multiplication can be performed by four data paths. The result of the matrix multiplication can be further processed by the element operation unit to generate a feature map as the convolution output. The feature map can be temporarily stored in the local memory via, for example, a DMA unit.

[0080] The pooling unit may also include an interpolation unit, a pooling data path, etc., and is used to perform a pooling operation. In some embodiments, the interpolation unit may perform interpolation (e.g., bilinear interpolation) on the feature map before pooling. Then, the interpolated feature map may be pooled according to the pooling size to generate a pooling output. For example, maximum pooling or average pooling may be performed on the feature map. The pooling output may also be temporarily stored in a local memory via, for example, a DMA unit.

[0081] In addition to transferring matrices, feature maps, etc. between these processing units and the cores of the NPU, the DMA unit can also reshape, pack, and merge data. In some embodiments, the DMA unit can transform images into matrices and vice versa. For example, data in the form of images can be used for convolution operations, and data in the form of matrices can be used for matrix operations (e.g., matrix-matrix multiplication).

[0082] Table 1 below further lists the key features of NPU 400.

[0083] Table 1

[0084]

[0085] Fig.10 Exemplary instructions of NPU 400 according to some embodiments of the present disclosure are shown.

[0086] As described above, instructions can be sent to the convolution unit, pooling unit, and DMA unit to cause these units to perform various operations of the neural network task. Fig.10As shown, instructions may be stored in an instruction buffer, including but not limited to "LMCPY", "CONV", "POOL", "MATMUL", "TRANS", "BR", "ROI", "INTERP", "SOP", and "VOP". Instructions in the instruction buffer may be located by a pointer to the address of the instruction. For example, a pointer to the address of the instruction may be determined based on a program counter. The program counter may be initialized and include the address of the next instruction. In Fig.10 In the example, the starting value of the program counter is initialized to the starting address of the instruction "LMCPY". When an instruction is executed, the program counter points to the next instruction. In some embodiments, the program counter jumps to the next instruction at a set distance (corresponding to labelDistance in the code).

[0087] The instruction "LMCPY" is a local memory copy instruction and can be used to perform a local memory copy operation. For example, the instruction "LMCPY" causes the DMA unit to copy block data from a read address and send the block data to a write address.

[0088] The instruction "CONV" is a convolution instruction, which can be used to instruct the convolution unit to perform a convolution operation. The instruction "CONV" may include a modification flag field, which allows various fields of the instruction to be modified online for runtime binding and control. The modification flag field may be one bit.

[0089] The instruction "POOL" is a pool instruction that can be used to instruct the pooling unit to perform a pooling operation. The instruction "POOL" may include a wait flag field, indicating that the pooling operation of the current layer must wait for the output of the specified layer before continuing. Therefore, the wait flag field may include a wait flag and a specified layer. In other words, the wait flag field may specify the data dependency between layers. If the wait flag is not set in the wait flag field, it means that the layer associated with the instruction can be executed in parallel with the layer specified in the wait flag field.

[0090] The instruction "MATMUL" is a matrix multiplication instruction and can be used to instruct the matrix multiplication data path to perform matrix multiplication.

[0091] The instruction "TRANS" is a transform instruction and can be used to instruct the DMA unit to transform an image into a matrix and vice versa.

[0092] Instruction "BR" is a branch instruction that can be used to modify the program counter to point to the specified address of the next instruction. In some embodiments, instruction "BR" may include a synchronization field to coordinate tasks in different cores. The synchronization field may be a one-bit field, and it may also be referred to as a barrier flag or a synchronization flag. In some embodiments, when a core completes its task, the core may set (assert) the synchronization field to notify the NPU that the task has been completed. The core is then suspended until other cores also complete their tasks and are assigned a new task. Therefore, a neural network task can be divided and assigned to different cores for parallel computing.

[0093] The command "ROI" is a region setting command that can be used to indicate a region of interest (ROI). In some embodiments, a region of interest for pooling can be determined to improve the accuracy of reasoning. The command "ROI" can specify the coordinates of at least one ROI and the number of the at least one ROI. The coordinates of the ROI can include four pairs of coordinates of the four corners of the ROI.

[0094] The instruction "INTERP" is an interpolation instruction and can be used in a pooling unit to perform interpolation on a feature map. For example, the interpolation can be a bilinear interpolation.

[0095] Instruction "SOP" is a scalar operation instruction that can be used to perform a scalar operation. For example, a scalar operation can be performed to determine a branch program counter based on a current program counter and a label distance. In some embodiments, instruction "SOP" can be executed by a branch / scalar unit, and the scalar operation result can be stored in a scalar register file, such as Fig. 9 shown.

[0096] The instruction "VOP" is a vector instruction that can be used to perform vector operations. For example, the instruction "VOP" can enable the element operation unit to perform vector operations, such as addition, vector and vector multiplication, etc. In some embodiments, the instruction "VOP" may also include an "end" field to indicate that the neural network task is completed or various operations of the neural network task are ended here.

[0097] Because the NPU 400’s instructions are designed to provide additional options and flags for optimization, high-quality results can be achieved without going through tedious and often less effective processes such as database searches and low-level assembly tuning.

[0098] Fig.11 is a flowchart of an exemplary method implemented by a processing unit according to some embodiments of the present disclosure. In some embodiments, method 1100 may be performed by Figure 4 , Figure 5A or Figure 5CIn some embodiments, the method 1100 may be implemented by a computer program product contained in a computer-readable medium, the computer program product comprising a computer (e.g., Figure 4 , Figure 5A or Figure 5C Computer-executable instructions such as program code executed by the NPU 400 in the processor.

[0099] In some embodiments, the processing unit may include a command parser (e.g., Figure 4 command parser 406) for distributing commands and computing tasks; and at least one core (e.g., Figure 4 The core 402a, 402b, 402c or 402d) is coupled to the command parser and is configured to process the assigned computing tasks. Each core may include a convolution unit (e.g., Figure 4 convolutional unit 4030), pooling unit (e.g., Figure 4 A pooling unit 4032), at least one operation unit (e.g., Figure 4 operation unit 4020) and a sequencer (e.g., Figure 4 The sequencer 4028 may be coupled to and communicated with the convolution unit, the pooling unit, and the at least one operation unit, and is configured to distribute instructions of the allocated computing tasks to the convolution unit, the pooling unit, and the at least one operation unit for execution.

[0100] At step 1102, the convolution unit retrieves data from a local memory (e.g., Figure 4 The data is read from the local memory 4022).

[0101] At step 1104, the convolution unit performs a convolution operation on the data to generate a feature map. For example, the convolution unit may perform a convolution operation via a convolution instruction to generate a feature map. In some embodiments, the convolution instruction may include a modification flag field associated with an online modification of various fields of the convolution instruction bound and controlled at runtime.

[0102] At step 1106, the pooling unit performs a pooling operation on the feature map. In some embodiments, the pooling unit may include an interpolation unit and a pooling data path. The interpolation unit may interpolate the feature map (e.g., by executing an interpolation instruction). The pooling data path may perform a pooling operation on the interpolated feature map (e.g., by executing a pooling instruction). The pooling unit may determine a region of interest on the feature map (e.g., by executing a region setting instruction).

[0103] In some embodiments, at least one operation unit may include a matrix multiplication data path (e.g., Figure 4data path 4024) and EWOP unit (e.g., Figure 4 The matrix multiplication data path may perform a matrix multiplication operation on the convolution data from the convolution unit (e.g., by executing a matrix multiplication instruction) to generate intermediate data, and the EWOP unit may perform an EWOP (e.g., by executing a vector instruction) to generate a feature map based on the intermediate data.

[0104] In some embodiments, the sequencer can monitor the execution of the neural network task and parallelize multiple subtasks of the neural network task. For example, the sequencer can use the synchronization field of the branch instruction to coordinate multiple subtasks of the neural network task in multiple different cores.

[0105] In some embodiments, each core may further include a DMA unit (e.g., Figure 4 DMA unit 408a). The DMA unit can transfer data within each core and between at least one core. For example, the DMA unit can transfer data within the core through a local memory copy instruction. In some embodiments, the DMA unit can input or output data in parallel with the calculation of the convolution unit, the pooling unit, or at least one operation unit (for example, by executing a copy instruction). In some embodiments, the DMA unit can transform data between image and matrix forms (for example, by executing a transform instruction).

[0106] In some embodiments, at least a portion of the pooling operation of the pooling unit may be performed in parallel with the convolution operation of the convolution unit. For example, through the wait flag field of the pooling instruction, at least a portion of the pooling operation of the pooling unit is performed in parallel with the convolution operation of the convolution unit.

[0107] In some embodiments, each core may further include a scalar unit and a scalar register file. The scalar unit may perform scalar operations (eg, by executing scalar operation instructions) and write the results of the scalar operations to the scalar register file.

[0108] Embodiments of the present disclosure may be applied to many products, environments, and scenarios. For example, some embodiments of the present disclosure may be applied to Ali-NPU (e.g., Hanguang NPU), Ali-Cloud, Ali PIM-AI (processor memory for AI), Ali-DPU (database acceleration unit), Ali-AI platform, GPU, tensor processing unit (TPU), etc.

[0109] The embodiments can be further described using the following statements:

[0110] 1. A method implemented by a processing unit, the processing unit comprising a command parser and at least one core, the command parser being used to allocate commands and computing tasks, the at least one core being coupled and communicating with the command parser and being used to process the allocated computing tasks, each core comprising a convolution unit, a pooling unit, at least one operation unit and a sequencer, the sequencer being coupled and communicating with the convolution unit, the pooling unit and the at least one operation unit and being used to allocate instructions of the allocated computing tasks to the convolution unit, the pooling unit and the at least one operation unit for execution, the method comprising:

[0111] Reading data from a local memory of the at least one operation unit by the convolution unit;

[0112] performing a convolution operation on the data by the convolution unit to generate a feature map; and

[0113] The pooling unit performs a pooling operation on the feature map.

[0114] 2. The method according to number 1, wherein the at least one operation unit comprises: a matrix multiplication data path (DP) and an element operation unit (EWOP), and the method further comprises:

[0115] performing a matrix multiplication operation on the convolution data from the convolution unit through the matrix multiplication data path to generate intermediate data; and

[0116] An EWOP is performed by the element operation unit to generate a feature map based on the intermediate data.

[0117] 3. The method according to number 1 or 2, wherein the pooling unit comprises: an interpolation unit and a pooling data path, and the method further comprises:

[0118] interpolating the feature map by an interpolation unit; and

[0119] The pooling operation is performed on the interpolated feature maps by the pooling data path.

[0120] 4. The method according to number 3, further comprising:

[0121] The region of interest on the feature map is determined by the pooling unit.

[0122] 5. The method according to any one of items 1 to 4, further comprising:

[0123] monitoring the execution of the neural network tasks by the sequencer; and

[0124] A plurality of subtasks of the neural network task are parallelized by the sequencer.

[0125] 6. The method according to number 1 to 5, wherein each core further comprises: a direct memory access unit, and the method further comprises:

[0126] Data is transferred within each core and between the at least one core by the direct memory access unit.

[0127] 7. The method according to number 6 also includes: the input and output data executed by the direct memory access unit are calculated in parallel with the convolution unit, the pooling unit or at least one operation unit.

[0128] 8. The method according to number 6 or 7, further comprising:

[0129] The direct memory access unit converts data between image and matrix forms.

[0130] 9. The method according to any one of items 1 to 8, further comprising:

[0131] The pooling unit performs at least a portion of the pooling operation in parallel with the convolution operation of the convolution unit.

[0132] 10. The method according to any one of numbers 1 to 9, wherein each core further comprises: a scalar unit and a scalar register file, and the method further comprises:

[0133] Scalar operations are performed by the scalar unit; and

[0134] The results of scalar operations are written by the scalar unit in the scalar register file.

[0135] 11. A non-transitory computer-readable storage medium for storing a set of instructions, the set of instructions being executable by at least one processing unit to cause a computer to perform a method, the processing unit comprising a command parser and at least one core, the command parser being used to assign commands and computing tasks, the at least one core being coupled to the command parser for communication and being used to process the assigned computing tasks, each core comprising a convolution unit, a pooling unit, at least one operation unit and a sequencer, the sequencer being coupled to the convolution unit, the pooling unit and the at least one operation unit for communication and being used to assign instructions of the assigned computing tasks to the convolution unit, the pooling unit and the at least one operation unit for execution, the method comprising:

[0136] Reading data from a local memory of the at least one operation unit by the convolution unit;

[0137] performing a convolution operation on the data by the convolution unit to generate a feature map; and

[0138] The pooling unit performs a pooling operation on the feature map.

[0139] 12. The non-transitory computer-readable storage medium of number 11, wherein the at least one operation unit comprises: a matrix multiplication data path (DP) and an element operation unit (EWOP), and the set of instructions is executed by the at least one processing unit to cause the computer to perform:

[0140] performing a matrix multiplication operation on the convolution data from the convolution unit through a matrix multiplication data path to generate intermediate data; and

[0141] The EWOP is performed by the element operation unit to generate a feature map based on the intermediate data.

[0142] 13. The non-transitory computer-readable storage medium according to number 11 or 12, wherein the pooling unit comprises an interpolation unit and a pooling data path, and the set of instructions is executed by the at least one processing unit to cause the computer to perform:

[0143] interpolating the feature map by an interpolation unit; and

[0144] The pooling operation is performed on the interpolated feature maps by the pooling data path.

[0145] 14. The non-transitory computer-readable storage medium of number 13, wherein the set of instructions is executed by the at least one processing unit to cause the computer to perform:

[0146] The region of interest on the feature map is determined by the pooling unit.

[0147] 15. The non-transitory computer-readable storage medium according to any one of numbers 11 to 14, wherein the set of instructions is executed by the at least one processing unit to cause the computer to perform:

[0148] monitoring the execution of the neural network tasks by the sequencer; and

[0149] A plurality of subtasks of the neural network task are parallelized by a sequencer.

[0150] 16. The non-transitory computer-readable storage medium of any one of numbers 11 to 15, wherein each core further comprises a direct memory access unit, and the set of instructions is executed by the at least one processing unit to cause the computer to perform:

[0151] Data is transferred within each core and between the at least one core by the direct memory access unit.

[0152] 17. The non-transitory computer-readable storage medium of number 16, wherein the set of instructions is executed by the at least one processing unit to cause the computer to perform:

[0153] The input and output data performed by the direct memory access unit are parallel to the calculation of the convolution unit, the pooling unit or at least one operation unit.

[0154] 18. The non-transitory computer-readable storage medium of number 17, wherein the set of instructions is executed by the at least one processing unit to cause the computer to perform:

[0155] The direct memory access unit converts data between image and matrix forms.

[0156] 19. The non-transitory computer-readable storage medium according to any one of numbers 11 to 18, wherein the set of instructions is executed by the at least one processing unit to cause the computer to perform:

[0157] The pooling unit performs at least a portion of the pooling operation in parallel with the convolution operation of the convolution unit.

[0158] 20. The non-transitory computer-readable storage medium of any one of numbers 11 to 19, wherein each core further comprises a scalar unit and a scalar register file, and the set of instructions is executed by the at least one processing unit to cause the computer to perform:

[0159] Scalar operations are performed by the scalar unit; and

[0160] The results of scalar operations are written by the scalar unit in the scalar register file.

[0161] 21. A processing unit, comprising:

[0162] A command parser that distributes commands and computational tasks; and

[0163] At least one core coupled to the command parser for processing the assigned computing tasks, each core comprising:

[0164] a convolution unit having circuits for performing convolution operations via convolution instructions to generate feature maps;

[0165] A pooling unit having a circuit for performing a pooling operation on the feature map through a pooling instruction;

[0166] at least one operating unit having circuitry for processing data; and

[0167] A sequencer is coupled to communicate with the convolution unit, the pooling unit and the at least one operation unit, and has a circuit for assigning instructions of the assigned computing tasks to the convolution unit, the pooling unit and the at least one operation unit for execution.

[0168] 22. The processing unit according to number 21, wherein the at least one operating unit comprises:

[0169] Local storage, used to store data;

[0170] a matrix multiplication data path having circuitry for performing matrix multiplication operations via matrix multiplication instructions; and

[0171] An element-wise operations unit with circuitry to perform EWOPs via vector instructions.

[0172] 23. A processing unit according to number 22, wherein the matrix multiplication data path has a circuit for performing a matrix multiplication operation on the convolution data from the convolution unit through a matrix multiplication instruction to generate intermediate data; and the EWOP unit has a circuit for generating a feature map based on the intermediate data through a vector instruction.

[0173] 24. The processing unit according to number 22 or 23, wherein each core further comprises:

[0174] A hub unit having circuitry for transferring read data and write data associated with the neural network task between the convolution unit, the pooling unit, the at least one operation unit, and the local memory.

[0175] 25. A processing unit according to any one of numbers 22 to 24, wherein the vector instruction further includes: an end field for indicating that the neural network task has been completed.

[0176] 26. The processing unit according to any one of items 21 to 25, wherein the pooling unit further comprises:

[0177] an interpolation unit having circuits for interpolating the feature map via interpolation instructions; and

[0178] A pooling data path having circuits for performing pooling operations on the interpolated feature maps via pooling instructions.

[0179] 27. A processing unit according to any one of numbers 21 to 26, wherein the sequencer also has a circuit for monitoring the execution of the neural network task and using the synchronization field of the branch instruction to coordinate multiple subtasks of the neural network task distributed in multiple different cores.

[0180] 28. The processing unit according to any one of numbers 21 to 27, wherein each core further comprises:

[0181] An instruction buffer is coupled in communication with the sequencer.

[0182] 29. The processing unit according to any one of numbers 21 to 28, wherein each core further comprises:

[0183] A direct memory access unit with circuitry to transfer data within the core via local memory copy instructions.

[0184] 30. The processing unit according to number 29, wherein the direct memory access unit has a circuit for inputting or outputting data in parallel with the calculation of the convolution unit, the pooling unit or the at least one operation unit through a copy instruction.

[0185] 31. The processing unit according to any one of numbers 29 or 30, wherein the direct memory access unit has a circuit for converting between image and matrix forms of data.

[0186] 32. A processing unit according to any one of numbers 21 to 31, wherein the pooling unit has a circuit for parallelizing at least a portion of the pooling operation with the convolution operation of the convolution unit through a wait flag field of the pooling instruction.

[0187] 33. A processing unit according to any one of numbers 21 to 32, wherein the pooling unit has a circuit for determining the region of interest of the feature map through a region setting instruction.

[0188] 34. The processing unit according to any one of numbers 21 to 33, wherein each core further comprises a scalar unit and a scalar register file, wherein the scalar unit has a circuit for performing scalar operations through scalar operation instructions and writing scalar operation results to the variable register file.

[0189] 35. A processing unit according to any one of numbers 21 to 34, wherein the convolution instruction further comprises: a modification flag field associated with online modification of fields in the convolution instruction bound and controlled at runtime.

[0190] Various exemplary embodiments are described above in the general context of steps or processes of the method, which, on the one hand, can be implemented by a computer program product contained in a computer-readable medium, which includes computer executable instructions such as program code executed by a computer in a network environment. Computer-readable media may include removable and non-removable storage devices, including but not limited to read-only memory (ROM), random access memory (RAM), optical disks, digital versatile disks (DVD), etc. In general, program modules may include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Computer executable instructions, related data structures, and program modules represent examples of program codes for executing the steps of the methods disclosed herein. A specific sequence of such executable instructions or related data structures represents an example of corresponding actions for implementing the functions described in such steps or processes.

[0191] The disclosure provides the above description for the purpose of illustration. It is not exhaustive and is not limited to the precise form or embodiment disclosed. By considering the description and practice of the disclosed embodiments, modifications and adaptations to the embodiments will be apparent. For example, the described implementation includes hardware, but systems and methods consistent with the present disclosure can be implemented with hardware and software. In addition, although certain components have been described as being coupled to each other, these components can be integrated with each other or deployed in any suitable manner.

[0192] In addition, although illustrative embodiments have been described herein, the scope includes any and all embodiments having equivalent elements, modifications, omissions, combinations (e.g., combining aspects of different embodiments), adaptations, or adaptations based on the present disclosure. The elements in the claims are to be interpreted broadly based on the language used in the claims and are not limited to the examples described in this specification or during the prosecution, which are to be interpreted as non-exclusive. In addition, the steps of the disclosed methods may be modified in any way, including reordering steps and / or inserting or deleting steps.

[0193] The features and advantages of the present disclosure are apparent based on the detailed description, and therefore, the appended claims are intended to cover all systems and methods that fall within the spirit and scope of the present disclosure. As used herein, the indefinite articles "a" and "an" mean "one or more". Similarly, the use of plural terms does not necessarily mean a plurality unless it is clear in a given context. In addition, since many modifications and changes are prone to occur through study of the present disclosure, it is not desired to limit the present disclosure to the exact structure and operation shown and described, and therefore, all suitable modifications and equivalents fall within the scope of protection of the present disclosure.

[0194] As used herein, unless otherwise specifically stated, the term "or" includes all possible combinations, unless otherwise feasible. For example, if it is stated that a component may include A or B, then unless otherwise specifically stated or not feasible, the component may include A, B, or A and B. As a second example, if it is stated that a component may include A, B, or C, then unless otherwise specifically stated or not feasible, the component may include A, B, C, A and B, A and C, B and C, or A and B and C.

[0195] Other embodiments will be apparent from consideration of the specification and practice of the embodiments disclosed herein.It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the disclosed embodiments being indicated by the claims.

Claims

1. A method implemented by a processing unit, the processing unit comprising a command parser and at least one core, the command parser is used to allocate commands and computing tasks, the at least one core is coupled to the command parser for communication and is used to process the allocated computing tasks, each core comprises a convolution unit, a pooling unit, at least one operation unit and a sequencer, the sequencer is coupled to the convolution unit, the pooling unit and the at least one operation unit for communication and has a circuit for allocating instructions of the allocated computing tasks to the convolution unit, the pooling unit and the at least one operation unit for execution, the sequencer also includes a circuit configured to modify instructions in the core, the method comprising: Reading data from a local memory of the at least one operation unit by the convolution unit; The convolution unit performs a convolution operation on the data to generate a feature map; and The pooling unit performs a pooling operation on the feature map.

2. The method according to claim 1, wherein: The at least one operation unit comprises: a matrix multiplication data path and an element operation unit, and the method further comprises: performing a matrix multiplication operation on the convolution data from the convolution unit through the matrix multiplication data path to generate intermediate data; and An EWOP is performed by the element operation unit to generate the feature map based on the intermediate data.

3. The method according to claim 1, wherein: The pooling unit includes: an interpolation unit and a pooling data path, and the method further includes: interpolating the feature map by the interpolation unit; and A pooling operation is performed on the interpolated feature map through the pooling data path.

4. The method according to claim 3, further comprising: The pooling unit determines a region of interest on the feature map.

5. The method according to claim 1, further comprising: Monitoring the execution of the neural network task by the sequencer; and Multiple subtasks of the neural network task are parallelized by the sequencer.

6. The method according to claim 1, wherein: Each core further includes: a direct memory access unit, and the method further includes: transferring data within each core and between the at least one core by the direct memory access unit; and Data is input or output by the direct memory access unit in parallel with calculation of the convolution unit, the pooling unit or the at least one operation unit.

7. The method according to claim 6, further comprising: Data is converted between image form and matrix form by the direct memory access unit.

8. The method according to claim 1, further comprising: The pooling unit performs at least a portion of the pooling operation in parallel with the convolution operation of the convolution unit.

9. The method according to claim 1, wherein: Each core also includes a scalar unit and a scalar register file, and the method further includes: performing scalar operations by the scalar unit; and The result of the scalar operation is written by the scalar unit into the scalar register file.

10. A non-transitory computer-readable storage medium, for storing a set of instructions, the set of instructions being executed by at least one processing unit to enable a computer to perform a method, the processing unit comprising a command parser and at least one core, the command parser being used to assign commands and computing tasks, the at least one core being coupled to the command parser for communication and being used to process the assigned computing tasks, each core comprising a convolution unit, a pooling unit, at least one operation unit and a sequencer, the sequencer being coupled to the convolution unit, the pooling unit and the at least one operation unit for communication and having a circuit for assigning instructions of the assigned computing tasks to the convolution unit, the pooling unit and the at least one operation unit for execution, the sequencer also comprising a circuit configured to modify instructions in the core, the method comprising: Reading data from a local memory of the at least one operation unit by the convolution unit; The convolution unit performs a convolution operation on the data to generate a feature map; and The pooling unit performs a pooling operation on the feature map.

11. The non-transitory computer-readable storage medium of claim 10, wherein: The at least one operation unit includes: a matrix multiplication data path and an element operation unit, and the set of instructions is executed by the at least one processing unit to cause the computer to perform: performing a matrix multiplication operation on the convolution data from the convolution unit through the matrix multiplication data path to generate intermediate data; and An EWOP is performed by the element operation unit to generate the feature map based on the intermediate data.

12. The non-transitory computer-readable storage medium of claim 10, wherein: The set of instructions is executed by the at least one processing unit to cause the computer to perform: monitoring, by the sequencer, execution of the neural network task; and Multiple subtasks of the neural network task are parallelized by the sequencer.

13. The non-transitory computer-readable storage medium of claim 10, wherein: Each core also includes a direct memory access unit, and the set of instructions is executed by the at least one processing unit to cause the computer to perform: transferring data within each core and between the at least one core by the direct memory access unit; and Data is input or output by the direct memory access unit in parallel with calculation of the convolution unit, the pooling unit or the at least one operation unit.

14. The non-transitory computer-readable storage medium of claim 10, wherein: The set of instructions is executed by the at least one processing unit to cause the computer to perform: The pooling unit performs at least a portion of the pooling operation in parallel with the convolution operation of the convolution unit.

15. A processing unit, comprising: Command parser, used to distribute commands and computing tasks; and At least one core coupled to the command parser for processing the assigned computing tasks, each core comprising: a convolution unit having circuits for performing convolution operations via convolution instructions to generate feature maps; A pooling unit having a circuit for performing a pooling operation on the feature map through a pooling instruction; at least one operating unit having circuitry for processing data; and A sequencer is coupled to communicate with the convolution unit, the pooling unit and the at least one operation unit, and has a circuit for assigning instructions of the assigned computing tasks to the convolution unit, the pooling unit and the at least one operation unit for execution, and the sequencer also includes a circuit configured to modify instructions in the core.

16. The processing unit according to claim 15, wherein: The at least one operating unit comprises: Local storage, used to store data; a matrix multiplication data path having circuitry for performing matrix multiplication operations via matrix multiplication instructions; and An element-wise operations unit with circuitry to perform EWOPs via vector instructions.

17. The processing unit according to claim 15, wherein: The sequencer also has circuitry for monitoring execution of the neural network task and using a synchronization field of a branch instruction to coordinate multiple subtasks of the neural network task distributed across multiple different cores.

18. The processing unit according to claim 15, wherein: Each core also includes: A direct memory access unit having a circuit for transferring data within the core via a local memory copy instruction, and a circuit for inputting or outputting data in parallel with the calculation of the convolution unit, the pooling unit or the at least one operation unit via the copy instruction.

19. The processing unit according to claim 15, wherein: The pooling unit has a circuit for parallelizing at least a portion of the pooling operation with the convolution operation of the convolution unit through a wait flag field of the pooling instruction.

20. The processing unit according to claim 15, wherein: The convolution instruction further includes a modification flag field associated with online modification of various fields in the convolution instruction bound and controlled at runtime.

Images