Neural processing units and methods capable of reusing data

Abstract

A neural processing unit includes: a mode selector configured to select either a first mode or a second mode; and a processing element (PE) array operating in one of the first and second modes and comprising a plurality of processing elements arranged in PE rows and PE columns, the PE array being configured to receive input of the first input data and input of the second input data, respectively. In the second mode, the first input data is input and transmitted along the PE column direction of the PE array while being delayed for a specific number of clock cycles, while the second input data is broadcast to the plurality of processing elements of the PE array to which the first input data is input delayed by the specific number of clock cycles.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to Korean Patent Application No. 2021-0048753, filed with the Korean Intellectual Property Office on April 14, 2021, and Korean Patent Application No. 10-2022-0018340, filed with the Korean Intellectual Property Office on February 11, 2022, the disclosure of which is incorporated herein by reference. Technical Field

[0003] This disclosure relates to a neural processing unit (NPU) capable of reusing data and a method of operating the NPU. More specifically, this disclosure relates to an NPU that reuses weights during depthwise convolution operations and a method of operating the NPU. Background Technology

[0004] Humans possess intelligence capable of recognition, classification, reasoning, prediction, and control / decision-making. Artificial intelligence (AI) refers to the artificial imitation of human intelligence.

[0005] The human brain is composed of numerous nerve cells called neurons, each connected to hundreds or thousands of other neurons via connectors called synapses. To mimic human intelligence, the modeling of how biological neurons operate and the connections between them is called an artificial neural network (ANN) model. In other words, an artificial neural network is a system of nodes that mimic neurons in a layered structure.

[0006] These artificial neural network models are classified into "single-layer neural networks" and "multi-layer neural networks" based on the number of layers.

[0007] A typical multilayer neural network consists of an input layer, hidden layers, and an output layer. (1) The input layer receives external data; the number of neurons in the input layer is the same as the number of input variables. (2) The hidden layer is located between the input and output layers, receiving signals from the input layer, extracting features, and transmitting them to the output layer. (3) The output layer receives signals from the hidden layers and outputs them to the outside. The input signals between neurons are multiplied by the connection strength of each connection, with values ​​between zero and one, and then summed. If the sum is greater than a neuron threshold, the neuron is activated and implemented as an output value through an activation function.

[0008] Meanwhile, in order to achieve higher levels of artificial intelligence, artificial neural networks with an increased number of hidden layers are called deep neural networks (DNNs).

[0009] There are many types of DNNs, but as is well known, Convolutional Neural Networks (CNNs) are good at extracting features from input data and recognizing feature patterns.

[0010] CNN refers to a network structure that uses convolution of matrix-type input signals and matrix-type weight kernels to achieve operations between neurons in each layer.

[0011] Convolutional neural networks (CNNs) are neural networks whose function is similar to image processing in the human brain's visual cortex. CNNs are well-known for their suitability for object classification and detection.

[0012] refer to Figure 3 Convolutional neural networks are configured with alternating convolutional and pooling channels. In convolutional neural networks, most of the computation time is consumed by convolution operations.

[0013] Convolutional neural networks infer objects by extracting image features for each channel via matrix-type kernels and providing a dynamic balance, such as dynamism or distortion, through pooling. For each channel, a feature map is obtained by convolving the input data with the kernel, and an activation map for the corresponding channel is generated by applying an activation function such as the Corrected Linear Unit (ReLU). Pooling can then be applied.

[0014] The actual neural network that classifies patterns is located at the end of the feature extraction neural network and is called a fully connected layer. In the computational processing of convolutional neural networks, most of the calculations are performed through convolution or matrix multiplication.

[0015] At this point, the necessary weight kernels are read from memory quite frequently. A large part of the operation of a convolutional neural network requires time to read the weight kernels corresponding to each channel from memory.

[0016] Memory can be divided into main memory, internal memory, and on-chip memory. Each memory consists of multiple memory cells, and each memory cell has a unique memory address. When a neural processing unit reads weights or parameters stored in main memory, there may be a delay of several clock cycles until the memory cell corresponding to that memory address is accessed. This delay can include column address strobing (CAS) latency and row address strobing (RAS) latency.

[0017] Therefore, there is a problem that reading the necessary parameters from main memory and performing convolution consumes a lot of time and power. Summary of the Invention

[0018] The inventors of this disclosure have recognized the following matters.

[0019] First, the inventors of this disclosure have recognized that during the inference process of an artificial neural network model, the neural processing unit (NPU) frequently reads feature maps or weight kernels of specific layers of the ANN model from main memory.

[0020] The inventors of this disclosure have recognized that reading the feature map or kernel of the ANN model from main memory to the NPU is slow and consumes a lot of energy.

[0021] The inventors of this disclosure have recognized that increasing access to on-chip memory or NPU internal memory, rather than to main memory, can improve processing speed and reduce power consumption.

[0022] The inventors of this disclosure have recognized that in a processing element array with a specific structure, the PE utilization (%) of the processing element array drops rapidly during a specific convolution operation. For example, when there are one hundred processing elements in the processing element array, if only fifty processing elements are operating, the utilization of the processing element array is 50%.

[0023] The inventors of this disclosure have recognized that data reuse may be impossible during depth-wise convolution operations in a specific structure of a processing element array, thus causing a rapid decrease in the utilization of the processing element array.

[0024] In particular, the inventors of this disclosure have recognized that, in the case of depthwise convolution, the utilization of the processing element array is reduced compared to standard or pointwise convolution, and the resources, power and processing time required for depthwise convolution become inefficient, to the point that they become substantially similar to standard or pointwise convolution operations, even though the computational cost of depthwise convolution is relatively small compared to that of standard or pointwise convolution.

[0025] In particular, the inventors of this disclosure have recognized that even though the computational cost of depthwise convolution is relatively small, the performance of the NPU may become a bottleneck due to the low utilization of the processing element array.

[0026] Accordingly, this disclosure provides a neural processing unit that can reuse weights during depthwise convolution operations in an NPU, thereby reducing the number of main memory read operations and lowering power consumption. This disclosure also provides a method for operating the neural processing unit.

[0027] To address the aforementioned problems, a neural processing unit according to an example of this disclosure is provided.

[0028] According to one aspect of this disclosure, a neural processing unit (NPU) is provided, comprising: a mode selector configured to select a first mode or a second mode; and a processing element (PE) array operating in one of the first and second modes and including a plurality of processing elements arranged in PE rows and PE columns, the PE array being configured to receive input of first input data and input of second input data, respectively. In the second mode, the first input data can be input along the PE column direction of the PE array and can be transmitted along the PE column direction while being delayed for a specific number of clock cycles, while the second input data can be broadcast to the plurality of processing elements of the PE array to which the first input data is input delayed by the specific number of clock cycles.

[0029] The PE array can also be configured to perform pointwise convolution operations in the first mode.

[0030] The PE array can also be configured to perform depthwise convolution operations in the second mode.

[0031] The specific number of clock cycles can be determined based on the size of the weight kernel or the stride value of the convolution in the artificial neural network model.

[0032] In the first mode, the plurality of processing elements in each PE column of the PE array can be pipelined to transmit the first input data.

[0033] In the first mode, the second input data can be unicast to each of the plurality of processing elements in each PE row of the PE array.

[0034] The PE array may also include a delay buffer configured to output the first input data by delaying the first input data by the specified number of clock cycles.

[0035] The PE array can also be configured to determine the specific number of clock cycles based on the size of the weight kernel of the artificial neural network model.

[0036] In the second mode, the second input data can be broadcast to the PE column of the PE array via a branched signal line, and the first input data delayed by the specific number of clock cycles can be applied to the signal line of the PE column via the branch.

[0037] In the second mode, the PE rows of the PE array can consist of a first group of PE rows and a second group of PE rows, wherein the first group of PE rows is configured to enable the weight kernel size based on the artificial neural network model, and the second group of PE rows excludes the first group of PE rows and is configured to be disabled.

[0038] The PE array may further include: a first multiplexer disposed within at least some of the PE rows; a second multiplexer disposed at the input portions of the at least some of the PE rows; and a delay buffer disposed within the at least some of the PE rows.

[0039] According to another aspect of this disclosure, a neural processing unit (NPU) is provided, comprising: a mode selector configured to select a first mode or a second mode; and a processing element (PE) array including a plurality of processing elements arranged in PE rows and PE columns, the PE array being configured to perform a first convolution operation in the first mode and a second convolution operation in the second mode. The PE array may also be configured to reuse weight data for the second convolution operation within the PE array.

[0040] The first convolution operation may include a standard or pointwise convolution operation.

[0041] The second convolution operation may include a depthwise convolution operation.

[0042] The PE array can be configured to include a delay buffer, which is configured to reuse the weight data from the depthwise convolution operation.

[0043] In the first mode, the PE array can also be configured to receive: input of weight data, which is used for the first convolution operation and is input to pipelined processing elements of each PE column of the PE array, and input of feature map data, which is used for the first convolution operation and is unicast to each PE in the PE row of the PE array.

[0044] The NPU may further include a delay buffer disposed in at least some of the PE rows of the PE array, the delay buffer being configured to receive, in the second mode: input of the weight data for the second convolution operation, and input of the weight data delayed by the delay buffer and output from the delay buffer.

[0045] The PE array may further include a delay buffer configured to delay the weight data by a predetermined number of clock cycles, and the delay buffer may also be configured to delay based on the size of the weight kernel of the artificial neural network model.

[0046] According to another aspect of this disclosure, a neural processing unit (NPU) is provided, comprising: a weight storage unit configured to load weight data for a convolution operation; a feature map storage unit configured to load feature map data for the convolution operation; and a processing element (PE) array including a plurality of processing elements and a plurality of delay units arranged to correspond to at least some of the processing elements of the PE array, the plurality of delay units being configured to selectively delay the weight data by means of switching units corresponding to the plurality of delay units.

[0047] According to another aspect of this disclosure, a processing element array is provided, comprising: a first processing element configured to receive weight data; a delay unit configured to receive the weight data, delay the weight data by a specific number of clock cycles, and transmit the weight data to a second processing element; and a broadcast signal line configured to simultaneously provide feature map data to both the first and second processing elements. The delay unit may be configured to process depthwise convolutions by reusing the weight data.

[0048] According to this disclosure, by reusing weights in depthwise convolution operations in the NPU, the number of main memory read operations can be reduced and power consumption can be lowered.

[0049] Furthermore, according to this disclosure, power consumption can be minimized by disabling processing elements that are not used during depthwise convolution operations.

[0050] Furthermore, according to this disclosure, by delaying and reusing weights during depthwise convolution operations, a neural processing unit can be provided that saves energy used in an NPU and improves the efficiency and throughput of the processing element array.

[0051] The effects of this disclosure are not limited to those illustrated above; this specification includes a variety of other effects. Attached Figure Description

[0052] Figure 1 This is a schematic conceptual diagram illustrating an apparatus including a neural processing unit, based on examples of this disclosure.

[0053] Figure 2 This is a schematic conceptual diagram illustrating the compiler relevant to this disclosure.

[0054] Figure 3 This is a schematic conceptual diagram of a convolutional neural network according to this disclosure.

[0055] Figure 4 This is a schematic conceptual diagram illustrating a neural processing unit based on examples of this disclosure.

[0056] Figure 5 This is a schematic conceptual diagram illustrating one of the processing elements in the array of processing elements related to this disclosure.

[0057] Figure 6 This is an example illustration of the configuration of a processing element in an array of processing elements according to the present disclosure.

[0058] Figure 7 This is a schematic configuration diagram illustrating the structure of the processing element array according to examples of this disclosure.

[0059] Figure 8 This is a schematic diagram illustrating the structure of a processing element array operating in a first mode, based on examples of this disclosure.

[0060] Figure 9 This is a schematic diagram illustrating the structure of a processing element array operating in a second mode, based on examples of this disclosure.

[0061] Figure 10 This is a schematic configuration diagram illustrating the structure of the processing element array according to examples of this disclosure.

[0062] Figure 11 This is a schematic configuration diagram illustrating the structure of the processing element array according to examples of this disclosure.

[0063] Figure 12 This is a schematic diagram illustrating the structure of a processing element array according to examples of this disclosure.

[0064] Figure 13 This is an example diagram used to explain weighted data and feature map data based on the examples in this disclosure.

[0065] Figure 14 This is an example diagram used to explain the depthwise convolution operation on weight data and feature map data, based on the examples in this disclosure.

[0066] Figure 15 This is a schematic diagram illustrating the structure of a processing element array according to examples of this disclosure.

[0067] Figure 16 This is an example diagram illustrating the weighted data stored in the delay buffer over time, based on examples from this disclosure.

[0068] Figure 17This is an example diagram used to explain the depthwise convolution operation on weight data and feature map data, based on the examples in this disclosure.

[0069] Figure 18 This is an example diagram illustrating the weighted data stored in the delay buffer over time, based on examples from this disclosure.

[0070] Figure 19 This is an example diagram illustrating the weighted data stored in the delay buffer over time, based on examples from this disclosure.

[0071] Figure 20 This is a conceptual diagram used to explain the exemplary ANN model. Detailed Implementation

[0072] The specific structure or step-by-step description of the examples of the concept disclosed in this specification or application is merely illustrative for the purpose of explaining the examples of the concept according to this disclosure. Examples of the concept according to this disclosure may be embodied in various forms and should not be construed as limited to the examples described in this specification or application.

[0073] Because examples of the concept according to this disclosure can have various modifications and can take various forms, specific examples will be shown in the accompanying drawings and described in detail in this specification or application. However, this is not intended to limit the examples of the concept according to this disclosure in terms of a particular form of disclosure, and should be understood to include all modifications, equivalents, and alternatives within the spirit and scope of this disclosure.

[0074] Terms such as first and / or second may be used to describe various elements, but these elements should not be limited by these terms. The terms above are used only to distinguish one element from another; for example, without departing from the scope of the concept according to this disclosure, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.

[0075] When one element is said to be "connected" or "in contact" with another element, it should be understood that the other element may be directly connected to or in contact with the other element, but other elements may be positioned between them. On the other hand, when it is said that an element is "directly connected" or "directly in contact" with another element, it should be understood that there are no other elements between them. Other expressions describing the relationship between elements, such as "between" and "immediately adjacent" or "nearby" and "closely adjacent," should be interpreted similarly.

[0076] In this disclosure, expressions such as “A or B”, “at least one of A and / or B”, or “one or more of A and / or B” can include all possible combinations thereof. For example, “A or B”, “at least one of A and B”, or “at least one of A or B” can mean (1) including at least one A, (2) including at least one B, or (3) including at least one A and at least one B.

[0077] As used herein, expressions such as “first,” “second,” and “first or second” may modify various elements regardless of order and / or importance. Furthermore, it is used only to distinguish one element from others and does not limit those elements. For example, a first user device and a second user device may represent different user devices regardless of order and / or importance. For example, a first element may be named a second element without departing from the scope of the claims described in this disclosure, and similarly, a second element may be renamed a first element.

[0078] The terminology used in this disclosure is for the purpose of describing specific examples only and is not intended to limit the scope of other examples. Singular expressions may include plural expressions unless the context clearly specifies otherwise. The terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by one of ordinary skill in the art as described in this document.

[0079] In this disclosure, terms as defined in general dictionaries may be interpreted as having the same or similar meaning as in the context of the relevant art. Furthermore, unless explicitly defined in this document, they should not be interpreted in an ideal or overly formal sense. In some cases, even terms defined in this disclosure should not be interpreted as excluding examples of this disclosure.

[0080] The terminology used herein is for the purpose of describing specific examples only and is not intended to limit this disclosure. Singular expressions may include plural expressions unless the context clearly specifies otherwise. It should be understood that, as used herein, terms such as “comprising” or “having” are intended to indicate the presence of the stated features, quantities, steps, actions, components, parts, or combinations thereof, but do not preclude the possibility of the addition or presence of at least one other feature or quantity, step, action, element, part, or combination thereof.

[0081] Unless otherwise defined, all terms used herein, including technical or scientific terms, shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Terms such as those defined in common dictionaries shall be construed as having meanings consistent with their meanings in the relevant technical context and shall not be construed as having ideal or overly formal meanings unless expressly defined in this specification.

[0082] Each of the features in the various examples of this disclosure can be combined, either partially or entirely, or with each other. Furthermore, those skilled in the art will fully appreciate that multiple interlocks and drives are possible, and each example can be implemented independently of the others or together in a related manner.

[0083] When describing the implementation scheme, descriptions of technical content that is well-known in the technical field to which this disclosure pertains and is not directly related to this disclosure may be omitted. This is to more clearly convey the main points of this disclosure without obscuring them by omitting unnecessary descriptions.

[0084] In the following text, for the purpose of understanding the disclosure presented in this specification, the terminology used in this specification will be briefly summarized.

[0085] NPU: an abbreviation for Neural Processing Unit, which can refer to a processor specifically designed for computing artificial neural network models, separate from the Central Processing Unit (CPU).

[0086] ANN: An abbreviation for Artificial Neural Network. It can refer to a network that mimics human intelligence by connecting nodes in a layered structure, similar to neurons in the human brain, through synaptic connections.

[0087] ANN information includes network structure information, number of layers, connection relationships between layers, weights of each layer, computational processing methods, activation functions, and other information.

[0088] DNN: an abbreviation for Deep Neural Network, which can refer to an increase in the number of hidden layers in an artificial neural network to achieve a higher level of artificial intelligence.

[0089] CNN: an abbreviation for Convolutional Neural Network, is a type of neural network that functions similarly to image processing in the human brain's visual cortex. Convolutional neural networks are well-known for their suitability for image processing, and are renowned for their superior ability to extract and recognize features from input data.

[0090] Kernel: The weights of an N×M matrix used for convolution. Each layer of an ANN model has multiple kernels; the number of kernels can be referred to as the number of channels or filters.

[0091] The present disclosure will be described in detail below by referring to examples of the accompanying drawings.

[0092] Figure 1 An apparatus including a neural processing unit is illustrated based on examples of this disclosure.

[0093] refer to Figure 1Device B, including NPU 1000, includes on-chip region A. Each component of device B can be connected via an interface, system bus, and / or wiring. That is, each component of device B can communicate with bus 5000. Device B may include neural processing unit (NPU) 1000, central processing unit (CPU) 2000, on-chip memory 3000, and main memory 4000.

[0094] The main memory 4000 is located outside on-chip region A and may include ROM, SRAM, DRAM, resistive RAM, magnetoresistive RAM, phase-change RAM, ferroelectric RAM, flash memory, or high-bandwidth memory (HBM). The main memory 4000 may be configured with at least one memory cell. The main memory 4000 may be configured as a homogeneous memory cell or a heterogeneous memory cell.

[0095] The NPU 1000 is a processor specifically designed to perform operations for ANNs. The NPU 1000 may include internal memory 200.

[0096] Internal memory 200 may include volatile memory and / or non-volatile memory. For example, internal memory 200 may include ROM, SRAM, DRAM, resistive RAM, magnetoresistive RAM, phase-change RAM, ferroelectric RAM, flash memory, or HBM. Internal memory 200 may include at least one memory cell. Internal memory 200 may be configured as homogeneous memory cells or heterogeneous memory cells.

[0097] On-chip memory 3000 can be located in on-chip region A. On-chip memory 3000 can be memory mounted on a semiconductor die and can be memory used to cache or store data processed in on-chip region A. On-chip memory 3000 can include ROM, SRAM, DRAM, resistive RAM, magnetoresistive RAM, phase-change RAM, ferroelectric RAM, flash memory, and HBM. On-chip memory 3000 can include at least one memory cell. On-chip memory 3000 can be configured as homogeneous memory cells or heterogeneous memory cells.

[0098] The CPU2000 can be located in on-chip region A and can include a general-purpose processing unit. The CPU2000 is operatively connected to the NPU 1000, on-chip memory 3000, and main memory 4000.

[0099] Device B, which includes NPU 1000, may include at least one of the internal memory 200, on-chip memory 3000, and main memory 4000 of NPU 1000 described above. However, this disclosure is not limited thereto.

[0100] In the following text, the reference to "at least one memory" is intended to include at least one of internal memory 200, on-chip memory 3000, and main memory 4000. Furthermore, the description of on-chip memory 3000 is intended to include internal memory 200 of NPU 1000 or memory located outside NPU 1000 but in on-chip region A.

[0101] In the following text, reference will be made to Figure 20 Describes an ANN model, which is a conceptual diagram used to explain an exemplary ANN model configured to include a multi-layered structure. References Figure 20 The MobileNet V1.0 model can have 28 layers.

[0102] An ANN (Artificial Neural Network) is a network composed of artificial neurons. When it receives an input signal, the artificial neurons assign weights to the input signal and selectively apply activation functions. Such an ANN can be used to output inference results from input data.

[0103] Figure 1 The NPU 1000 can be a semiconductor device implemented as an electrical / electronic circuit. Electrical / electronic circuits can refer to a number of electronic components (e.g., transistors or capacitors).

[0104] The NPU 1000 may include an array of processing elements, internal memory 200, a controller, and an interface. Each of the processing element array, internal memory 200, controller, and interface may be a semiconductor circuit connecting many transistors. Therefore, some transistors may be difficult or impossible to identify and distinguish with the naked eye, and may be identified only by their function. For example, any circuit can operate as an array of processing elements or as a controller.

[0105] The NPU 1000 may include an array of processing elements, internal memory 200 configured to store at least a portion of an ANN model that can be inferred from the array of processing elements, and a scheduler configured to control the array of processing elements and internal memory 200 based on data locality information or information about the structure of the ANN model. Here, the ANN model may include information about data locality or information about the structure of the ANN model. However, this disclosure is not limited thereto. An ANN model may refer to an AI recognition model trained to perform a specific inference function.

[0106] For example, an ANN model can be a model trained to perform inference operations such as object detection, object segmentation, image / video reconstruction, image / video enhancement, object tracking, event recognition, event prediction, anomaly detection, density estimation, event search, measurement, etc.

[0107] For example, ANN models can be models such as BiseNet, ShelfNet, AlexNet, DenseNet, EfficientNet, EfficientDet, GoogleNet, MnasNet, MobileNet, ResNet, ShuffleNet, SqueezeNet, VGG, YOLO, RNN, CNN, DBN, RBM, LSTM, etc. However, this disclosure is not limited to these, and entirely new ANN models operating in NPUs are constantly being released.

[0108] PE arrays can perform operations for ANNs. For example, when inputting data, a PE array can train an ANN. Furthermore, when inputting data, a PE array can perform operations to derive inference results from the trained ANN model.

[0109] For example, the NPU 1000 can access at least a portion of the data of the ANN model stored in the main memory 4000 to the internal memory 200 via an interface.

[0110] The controller can be configured to control the operation of the PE array for inference processing and to control the read and write sequences of the internal memory 200. Furthermore, the controller can be configured to adjust the size of at least a portion of the batch channels corresponding to the input data.

[0111] Based on the structure of the artificial neural network model, the computation of each layer can be executed sequentially. That is, when the structure of the ANN model is determined, the order of operations for each layer can be determined. Depending on the size of the NPU 1000's internal memory 200 or on-chip memory 3000, the operations of each layer may not be processed all at once. In this case, the NPU 1000 can divide an operation processing step into multiple operation processing steps by tiling the corresponding layers to an appropriate size. The structure of the ANN model and the order of operations or data flow according to the hardware constraints of the NPU 1000 can be defined as the data locality of the ANN model inferred from the NPU 1000.

[0112] In other words, when the compiler compiles the ANN model so that it is executed on an NPU 1000, the ANN data locality at the NPU-memory level can be reconstructed. For example, the compiler can be executed by a CPU 2000. Alternatively, the compiler can run on a separate machine.

[0113] In other words, based on the compiler, the algorithm applied to the ANN model, and the operating characteristics of the NPU 1000, the size of the weight values, the number of feature maps or channels, and the size and sequence of the data required to process the ANN model loaded into the internal memory 200 can be determined.

[0114] For example, even with the same ANN model, the computation method for the ANN model to be processed can be configured based on the method and characteristics of the NPU 1000 in computing the corresponding ANN model. These characteristics include, for example, the feature map tiling method, the stationary method of processing elements, the number of processing elements in the NPU 1000, the size and weights of the feature maps in the NPU 1000, the internal memory capacity, the memory hierarchy of the NPU 1000, and the algorithmic characteristics of the compiler that determine the order of operations used by the NPU 1000 to process the ANN model. This is because even when processing the same ANN model using the above factors, the NPU 1000 can differently determine the data sequence required at each moment in each clock cycle.

[0115] In the following text, reference will be made to Figure 2 The compiler described above is described in detail, illustrating compiler 6000 which is relevant to this disclosure.

[0116] refer to Figure 2 The compiler 6000 has a front-end and a back-end, with an intermediate representation (IR) between them for program optimization. For example, the compiler 6000 can be configured to receive ANN models generated by deep learning frameworks such as ONNX, TensorFlow, PyTorch, mxnet, and Keras.

[0117] The front end can perform hardware-independent transformations and optimizations on the input ANN model, while the middle layer represents the source code. The back end can generate machine code in binary form from the source code (i.e., code that can be used in the NPU 1000).

[0118] Furthermore, the compiler 6000 can analyze the convolution methods of the ANN model to generate pattern information that includes information about all operations to be performed by the NPU 1000, thereby providing the generated pattern information to the NPU 1000. Here, the pattern information may include information about the first and / or second convolution operations for each layer, each channel, or each tile of the ANN model. For example, the first convolution operation may include a standard convolution operation or a pointwise convolution operation, and the second convolution operation may include a depthwise convolution operation, but is not limited to these.

[0119] Based on the mode information provided above, the NPU 1000 can determine the operating mode and perform arithmetic operations according to the determined operating mode.

[0120] In the following text, reference will be made to Figure 3 Describe in detail the Convolutional Neural Network (CNN), which is a type of Deep Neural Network (DNN) among various ANNs.

[0121] Figure 3 This disclosure describes convolutional neural networks.

[0122] A CNN may consist of one or more convolutional layers, a pooling layer, and a fully connected layer. CNNs have a structure suitable for learning and inference on two-dimensional data and can be trained using the backpropagation algorithm.

[0123] In the examples disclosed herein, the CNN contains kernels (i.e., weight kernels) that extract features from the input image for each channel. Kernels can consist of two-dimensional matrices and perform convolution operations while traversing the input data. The size of the kernel can be arbitrarily determined, as can the stride of the kernel traversing the input data. The convolution result of all input data for each kernel can be called a feature map or activation map. In the following text, a kernel may include one or more sets of weight values. The number of kernels per layer can be called the number of channels.

[0124] Therefore, since convolution is an operation that combines input data and a kernel, an activation function can be applied to add non-linearity. When an activation function is applied to a feature map that is the result of a convolution operation, it can be called an activation map.

[0125] Specifically, refer to Figure 3 A CNN may include at least one convolutional layer, at least one pooling layer, and at least one fully connected layer.

[0126] For example, convolution can be defined by two main parameters: the size of the input data (typically a 1×1, 3×3, or 5×5 matrix) and the depth of the output feature map (the number of kernels). These key parameters can be computed through convolution. These convolutions can start at a depth of 32, continue to a depth of 64, and end at a depth of 128 or 256. The convolution operation can be described as sliding a kernel of size 3×3 or 5×5 across the input image matrix, multiplying each element of the kernel by each element of the overlapping input image matrix, and then summing them all.

[0127] Activation functions can be applied to the output feature map to ultimately output an activation map. Pooling layers can perform pooling operations to reduce the size of the feature map by downsampling the output data (i.e., the activation map). For example, pooling operations can include, but are not limited to, max pooling and / or average pooling.

[0128] Max pooling uses a kernel and outputs the maximum value in the region where the feature map and kernel overlap by sliding the feature map and kernel. Average pooling outputs the average value in the region where the feature map and kernel overlap by sliding the feature map and kernel. Thus, because the size of the feature map is reduced through pooling operations, the number of parameters in the feature map is also reduced.

[0129] Fully connected layers can classify the data output from pooling layers into multiple categories (i.e., inferred values) and output the classified categories and their scores. The data output from pooling layers forms a 3D feature map, which can be converted into a 1D vector and used as input to fully connected layers.

[0130] In the following text, reference will be made to Figure 4 Provide a detailed description of the NPU.

[0131] Figure 4 The neural processing unit is illustrated by examples from this disclosure.

[0132] refer to Figure 4 The neural processing unit (NPU) 1000 includes a processing element array (PE array) 100, an internal memory 200, and a controller 300.

[0133] The PE array 100 can be configured to include multiple processing elements (PE1, PE2, ...) 110, which are configured to compute node data and weight data of the ANN network. Each processing element may include multiplication and accumulation (MAC) operators and / or arithmetic logic unit (ALU) operators. However, the examples according to this disclosure are not limited thereto.

[0134] Furthermore, the described processing element 110 is merely an example for ease of explanation, and the number of processing elements 110 is not limited. The size or number of PE arrays can be determined by the number of processing elements 110. The size of the PE array can be implemented as an N×M matrix, where N and M are integers greater than zero. Therefore, the PE array 100 can include N×M processing elements.

[0135] The size of the PE array 100 can be designed taking into account the characteristics of the ANN model that operates in the NPU 1000. In other words, the number of processing elements can be determined by considering the data size of the ANN model to be operated, the required computation, and the required power consumption. The data size of the ANN model can be determined based on the number of layers in the ANN model and the weight data size of each layer.

[0136] Therefore, the size of the PE array 100 according to the example of this disclosure is not limited. As the number of processing elements 110 of the PE array 100 increases, the parallel computing capability for processing ANN models increases, but the manufacturing cost and physical chip size of the NPU 1000 also increase.

[0137] For example, the ANN model operating in the NPU 1000 could be an ANN trained to detect thirty specific keywords, i.e., an AI keyword recognition model. In this case, considering the computational characteristics, the size of the PE array 100 can be designed to be 4×3. In other words, the PE array 100 can be configured to include twelve processing elements. However, it is not limited to this, and the number of the plurality of processing elements 110 can be selected, for example, in the range of 8 to 16,384. That is to say, the examples of this disclosure are not limited in terms of the number of processing elements.

[0138] PE array 100 can be configured to perform functions such as addition, multiplication, and accumulation required for ANN operations. In other words, PE array 100 can be configured to perform multiplication and accumulation (MAC) operations.

[0139] In the following text, Figure 4 One of the processing elements of the processing element array 100 will be described in detail with reference to FIG5.

[0140] Figure 5 A processing element (e.g., PE1) in the array of processing elements related to this disclosure is described.

[0141] refer to Figure 5 The first processing element PE1 (110) may include a multiplier 641, an adder 642, and an accumulator 643. The first processing element PE1 may optionally include a bit quantization unit 644. However, the examples according to this disclosure are not limited thereto, and various modifications can be made to the PE array 100 in consideration of the computational characteristics of ANNs.

[0142] Multiplier 641 multiplies the received (N)-bit data and (M)-bit data. The operation value of multiplier 641 can be output as (N+M)-bit data, where N and M are integers greater than zero. The first input unit receiving (N)-bit data can be configured to receive values ​​with characteristics such as variability, and the second input unit receiving (M)-bit data can be configured to receive values ​​with characteristics such as constancy.

[0143] For example, the first input unit can receive feature map data. That is, since feature map data can be data obtained by extracting features such as input images and speech, it can be data input in real time from an external source (e.g., a sensor). The feature map data input to the processing element can be referred to as input feature map data. The feature map data output from the processing element after the MAC operation is completed can be referred to as output feature map data. The NPU 1000 can further selectively apply additional operations such as batch normalization, pooling, and activation functions to the output feature map data.

[0144] For example, the second input unit can receive weights, i.e., kernel data. That is, once the training of the weight data of the ANN model is complete, the weight data of the ANN model may not change unless separate training is performed.

[0145] In other words, multiplier 641 can be configured to receive a variable and a constant. More specifically, the variable value input to the first input unit can be the feature map data of the ANN model. The constant value input to the second input unit can be the weight data of the ANN model.

[0146] In this way, when the controller 300 controls the internal memory 200 by classifying the characteristics of variable values ​​and constant values, the controller 300 can increase the memory reuse rate of the internal memory 200.

[0147] However, the input data of multiplier 641 is not limited to constant values ​​and variable values. That is, according to the examples of this disclosure, since the input data of the processing element can be manipulated by understanding the characteristics of constant and variable values, the operating efficiency of NPU 1000 can be improved. However, the operation of NPU 1000 is not limited to the characteristics of constant and variable values ​​of the input data.

[0148] Based on this, the controller 300 can be configured to improve memory reuse by taking into account the characteristics of constant values.

[0149] Refer again Figure 20 The controller 300 can confirm that the kernel size, input feature map size, and output feature map size of each layer of the ANN model are different from each other.

[0150] For example, when the size of the internal memory 200 is determined and the size of the input feature map and output feature map of a particular layer or slice of a particular layer is less than the capacity of the internal memory 200, then the controller 300 can control the NPU 1000 to reuse the feature map data.

[0151] For example, when the size of the internal memory 200 is determined, and when the weight of a particular layer or slice of a particular layer is significantly small, the controller 300 can control the NPU 1000 to reuse feature map data. (Return to Reference) Figure 20 As can be seen, the weights of the first to eighth layers are very small. Therefore, the controller 300 can control the internal memory 200 to retain the weights in the internal memory 200 for a specific period of time so that the weights can be reused.

[0152] In other words, the controller 300 can identify each reusable variable data based on data locality information or structural information including data reuse information of the ANN model, and selectively control the internal memory 200 to reuse the data stored in the memory.

[0153] In other words, the controller 300 can identify each reusable constant data based on data locality information or structural information including data reuse information of the ANN model, and selectively control the internal memory 200 to reuse the data stored in the memory. For the above operations, the compiler 500 or the controller 300 can classify the size of the weight data below the threshold size of the ANN model.

[0154] In other words, the controller 300 can identify reusable variable values ​​and reusable constant values ​​based on data locality information or structural information including data reuse information of the ANN model, and thus can selectively control the internal memory 200 to reuse data stored in the memory.

[0155] At the same time, when a zero value is input to one of the first input unit and the second input unit of the multiplier 641, the first processing element PE1 can recognize that the operation result is zero even if no operation is performed, and therefore, the operation of the multiplier 641 can be restricted so that the operation is not performed.

[0156] For example, when zero is input to one of the first and second input units of the multiplier 641, the multiplier 641 can be configured to operate in a zero-jump mode.

[0157] The bit width of the data input to the first and second input units of the multiplier 641 can be determined based on the quantization and weights of each feature map of the ANN model. For example, when the feature map of the first layer is quantized to five bits and the weights of the first layer are quantized to seven bits, the first input unit can be used to receive 5-bit wide data, and the second input unit can be configured to receive 7-bit wide data.

[0158] NPU 1000 can control the first processing element PE1 so that when quantized data stored in internal memory 200 is input to the first processing element PE1, the quantization bit width is converted in real time. That is, the quantization bit width of each layer can be different. Therefore, whenever the bit width of the input data is converted, the first processing element PE1 can receive bit width information from NPU 1000 and convert the bit width based on the provided bit width information to generate the input data.

[0159] Adder 642 adds the calculated value from multiplier 641 and accumulator 643. When loop L is 0, since there is no data to accumulate, the operation value of adder 642 will be the same as that of multiplier 641. When loop L is 1, the value obtained by adding the operation value of multiplier 641 and accumulator 643 can be the operation value of adder 642.

[0160] Accumulator 643 temporarily stores the data output from the output unit of adder 642, thereby accumulating the operation value of adder 642 and the operation value of multiplier 641 for L cycles. Specifically, the calculated value of adder 642, output from the output unit of adder 642, is input to the input unit of accumulator 643. The operation value input to the accumulator is temporarily stored in accumulator 643 and output from the output unit of accumulator 643. The output operation value is cyclically input to the input unit of adder 642. At this time, the newly output operation value from the output unit of multiplier 641 is input to the input unit of adder 642. That is, the operation value of accumulator 643 and the new operation value of multiplier 641 are input to the input unit of adder 642, these values ​​are added by adder 642 and output through the output unit of adder 642. The data output from the output unit of adder 642, i.e. the new operation value of adder 642, is input to the input unit of accumulator 643, and the subsequent operation is performed in basically the same way as the above operation, as many times as the number of loops.

[0161] In this way, accumulator 643 temporarily stores the data output from the output unit of adder 642, so as to accumulate the operation values ​​of multiplier 641 and adder 642 by the number of loops. Therefore, the data input to the input unit of accumulator 643 and the data output from the output unit can have the same bit width as the data output from the output unit of adder 642, that is, (N+M+log2(L)) bits, where L is a positive integer.

[0162] When the accumulation is complete, the accumulator 643 may receive an initialization reset signal to initialize the data stored in the accumulator 643 to zero. However, the examples according to this disclosure are not limited thereto.

[0163] The quantization unit 644 can reduce the number of bits of data output from the accumulator 643. The bit quantization unit 644 can be controlled by the controller 300. The number of bits of quantized data can be output as X bits, where X is a positive integer. According to the above configuration, the PE array 100 is configured to perform MAC operations, and the PE array 100 has the effect of quantizing and outputting the result of the MAC operation. In particular, as the number of cycles increases by L, this quantization has the effect of further reducing power consumption. In addition, if power consumption is reduced, it also has the effect of reducing heat generation of the edge device. In particular, reducing heat generation has the effect of reducing the possibility of failure due to high temperature of the NPU 1000.

[0164] The X-bit output data of the bit quantization unit 644 can be node data of the next layer or input data of the convolution. If the ANN model has been quantized, the bit quantization unit can be configured to receive quantization information from the ANN model. However, it is not limited to this; the controller 300 can be configured to extract quantization information by analyzing the ANN model. Therefore, the output data X bits can be converted into quantized bit bits to correspond to the quantized data size and output. The output data X bits of the bit quantization unit 644 can be stored as quantized bit bits in the internal memory 200.

[0165] According to the example of this disclosure, each processing element 110 of the NPU 1000 can reduce the number of bits of (N+M+log2(L)) bits of data output from the accumulator 643 to X bits via the bit quantization unit 644. The NPU controller 300 can control the bit quantization unit 644 to reduce the number of bits of the output data from the least significant bit (LSB) to the most significant bit (MSB) by a predetermined number of bits.

[0166] Reducing the number of bits in the output data can decrease the power consumption, computational load, and memory usage of the NPU1000. However, when the number of bits is reduced below a specific bit width, the inference accuracy of the ANN model may decrease rapidly. Accordingly, the reduction in the number of bits in the output data (i.e., the degree of quantization) can be determined by comparing the reduction in power consumption, computational load, and memory usage with the reduction in the inference accuracy of the ANN model. Alternatively, the degree of quantization can be determined by setting a target inference accuracy for the ANN model and testing it while gradually reducing the number of bits. The degree of quantization can be determined for each operation value at each layer.

[0167] According to PE1 above, by adjusting the number of bits of N-bit and M-bit data of multiplier 641 and reducing the number of bits of operation value X through bit quantization unit 644, PE has the effect of improving MAC operation speed while reducing power consumption, and has the effect of performing ANN convolution operation more efficiently.

[0168] Based on this, the internal memory 200 of the NPU 1000 can be a memory system configured with consideration of the MAC operation characteristics and power consumption characteristics of the PE array 100.

[0169] For example, considering the MAC operation characteristics and power consumption characteristics of the PE array 100, the NPU 1000 can be configured to reduce the bit width of the operation values ​​of the PE array 100.

[0170] For example, the NPU 1000 can be configured to reduce the bit width of the operating values ​​of the PE array 100 in order to reuse the feature map or weights of the internal memory 200.

[0171] The NPU internal memory 200 of the NPU 1000 can be configured to minimize the power consumption of the NPU 1000.

[0172] The internal memory 200 of the NPU 1000 can be a memory system configured to control the memory with low power, taking into account the parameter size and operation sequence of the ANN model to be operated.

[0173] The NPU 1000's internal memory 200 can be a low-power memory system configured to reuse specific memory addresses where weight data is stored, taking into account the data size and operation order of the ANN model.

[0174] The NPU 1000 can be configured to further include an operation unit configured to process various activation functions used to impart non-linearity. For example, activation functions may include the sigmoid function, hyperbolic tangent (tanh) function, ReLU function, Leaky-ReLU function, Maxout function, or ELU function, which derive a non-linear output value with respect to the input value; however, it is not limited to these. Such activation functions can be selectively applied after a MAC operation. The operation value of applying activation functions to a feature map can be referred to as the activation map.

[0175] Return to reference Figure 4 The internal memory 200 can be configured as volatile memory. Volatile memory stores data only when powered on and destroys the stored data when power is off. Volatile memory can include static random access memory (SRAM), dynamic random access memory (DRAM), etc. Internal memory 200 is preferably SRAM, but is not limited to it.

[0176] At least a portion of the internal memory 200 can be configured as non-volatile memory. Non-volatile memory is memory that can store data even without power. Non-volatile memory may include read-only memory (ROM), etc. Training weights may also be stored in non-volatile memory. That is, weight storage unit 210 may include volatile memory or non-volatile memory.

[0177] The internal memory 200 may include a weight storage unit 210 and a feature map storage unit 220. The weight storage unit 210 may store at least a portion of the weights of the ANN model, and the feature map storage unit 220 may store at least a portion of the node data or feature map of the ANN model.

[0178] The ANN data that can be included in the ANN model may include node data or feature maps for each layer, as well as weight data for each connection network connecting the nodes in each layer. At least some of the ANN data or parameters may be stored in memory set within the controller 300 or in internal memory 200.

[0179] In the parameters of an ANN, feature maps can be configured as batch channels. Here, multiple batch channels can be, for example, input images captured by multiple image sensors or cameras within substantially the same time period (e.g., within 10ms or 100ms).

[0180] Meanwhile, the controller 300 can be configured to control the PE array 100 and the internal memory 200, taking into account the magnitude of the weight values ​​of the ANN model, the size of the feature map, and the order in which the weight values ​​and feature maps are calculated.

[0181] The controller 300 may include a mode selector 310 and a scheduler 320.

[0182] The mode selector 310 can select whether the PE array 100 operates in the first mode or the second mode based on the magnitude of the weight values, the size of the feature maps, and the order in which the weight values ​​and feature maps will be calculated in the PE array 100.

[0183] Here, the first mode is the operation mode used to perform the first convolution operation, which can be a standard convolution operation or a pointwise convolution operation, but is not limited to these. The second mode is the operation mode used to perform the second convolution operation, which can be a depthwise convolution operation, but is not limited to these.

[0184] The mode selector 310 can send a selection signal indicating the operating mode to be selected in the first mode or the second mode to the PE array 100, so that the PE array 100 operates in the first mode or the second mode.

[0185] In various examples, mode selector 310 can select whether to operate in a first mode or a second mode based on mode information provided from compiler 500. For example, based on mode information provided from compiler 500, mode selector 310 can select either a first mode or a second mode and send a selection signal indicating the selected first or second mode to PE array 100.

[0186] Next, the scheduler 320 can control the PE array 100 and the internal memory 200 to operate according to the selected mode.

[0187] For example, when the mode selector 310 selects the first mode, the scheduler 320 loads the weight data corresponding to the first input data into the weight storage unit 210 of the internal memory 200, and the feature map data corresponding to the second input data can be loaded into the feature map storage unit 220 of the internal memory 200. The scheduler 320 can control the PE array 100 to compute the weight data and feature map data 100 through a first convolution operation in each of the plurality of PEs constituting the PE array.

[0188] When the mode selector 310 selects the second mode, the scheduler 320 can load the weight data into the weight storage unit 210 and the feature map data into the feature map storage unit 220 as described above. The controller 300 can control the PE array 100 to compute the weight data and feature map data through a second convolution operation in each of the plurality of PEs constituting the PE array 100.

[0189] Although the internal memory 200 is shown as including weight storage unit 210 and feature map storage unit 220 respectively, this is only an example, and the internal memory 200 can be logically partitioned or variably partitioned by memory address control, or it may not be partitioned at all.

[0190] More specifically, the size of the weight storage unit 210 and the size of the feature map storage unit 220 can be set differently for each layer and each slice. Return to Reference Figure 20 As can be seen, the data size of the feature map (IFMAP or OFMAP) and the data size of the weights are different for each layer.

[0191] In the example above, it has been described that the parameters of the ANN (e.g., weights and feature maps) are stored in the internal memory 200 of the NPU 1000, but this disclosure is not limited thereto and may be stored in the on-chip memory 3000 or the main memory 4000.

[0192] On the other hand, considering fairness, efficiency, stability, and response time, CPU scheduling operations generally aim for optimal efficiency. That is, considering priorities, computation time, and other factors, scheduling operations to execute the most processes simultaneously. Therefore, conventional CPUs use algorithms to schedule tasks, taking into account factors such as the priority of each process and its processing time.

[0193] In contrast, the controller 300 can select an operating mode and control the PE array 100 to perform convolution operations according to the determined operating mode, based on the calculation method of the parameters of the ANN model, and in particular, based on the characteristics of the convolution operation method to be performed in the PE array 100.

[0194] In addition, the controller 300 can control the PE array 100 to perform a first convolution operation, such as a pointwise convolution operation, in a first mode, and a second convolution operation, such as a depthwise convolution operation, in a second mode.

[0195] Generally, pointwise convolution is performed using kernel data in the form of a 1×1×M matrix, while depthwise convolution is performed using kernel data in the form of an N×M×1 matrix. Here, N and M can be integers greater than zero, and N and M can be the same number.

[0196] When performing depthwise convolution, the NPU 1000 may use only a subset of the PE rows of the PE array 100, which contains a matrix of PEs, thus potentially leaving some PEs unused for the operation. Furthermore, even when using only a subset of PE rows, depthwise convolution is not necessarily faster than pointwise convolution, so it can become inefficient in the NPU 1000. In other words, the utilization of the PE array 100 may be reduced.

[0197] To overcome this inefficiency, this disclosure proposes a neural processing unit configured to minimize data movement between main memory 4000 and on-chip region A by allowing PE array 100 to reuse weight data during depthwise convolution operations.

[0198] To overcome this inefficiency, this disclosure proposes a neural processing unit configured to power off unoperated PEs during depthwise convolution operations by reusing weight data in the PE array 100.

[0199] To overcome this inefficiency, this disclosure proposes a neural processing unit with high computational performance, while reducing the time and power required for depthwise computation by reusing weight data and / or feature map data during depthwise convolution operations of the PE array 100.

[0200] The following will describe in detail the PE array used for the neural processing unit, which operates the PE array according to a first mode or a second mode to reduce hardware resource usage and power consumption and improve computing performance.

[0201] Figure 6 An exemplary configuration of a processing element in a processing element array is illustrated by examples from this disclosure.

[0202] To describe in detail the operation of the processing element in this example, we can use information about... Figure 4 The components described are (i.e., weight storage unit 210, feature map storage unit 220, and mode selector 310).

[0203] refer to Figure 6 One of the multiple processing elements 110 can be PE_00, and the PE array 100 can include a register 120 corresponding to PE_00. Each register 120 can be referred to as one of register files RF1, RF2, ... Figure 4 As shown. Register 120 can correspond to storage. Figure 5The accumulator 643 is a temporary storage for the accumulated value.

[0204] PE_00 can be connected to the weight storage unit 210 via the signal line W_in_0 configured to transmit weight data, and can be connected to the feature map storage unit 220 via the signal line F_in_0 configured to transmit feature map data.

[0205] The PE_00 of the multiple processing elements 110 can perform operations (i.e., MAC operations) on weight data transmitted from the weight storage unit 210 and feature map data transmitted from the feature map storage unit 220, and can store the operation values ​​in register 120. Here, the operation values ​​can be feature map data representing the result of the MAC operation on the weight data and feature map data. For example, PE_00 may need nine clock cycles to perform a convolution operation using a 3×3 matrix weight kernel. The accumulated value over nine clock cycles can be stored in register 120. When the operation in PE_00 is completed, a reset signal Reset_00 can be received to initialize the operation values, thereby initializing the operation values ​​of PE_00.

[0206] PE_00 can be configured to reduce the power consumption of the NPU1000 by applying an enable signal En0 depending on whether PE_00 is enabled. Furthermore, the utilization rate of the PE array 100 of the NPU 1000 can be determined based on whether each processing element is operated.

[0207] Whether each processing element is operated can be controlled by the controller 300. The controller 300 can be configured to generate an enable signal corresponding to each processing element.

[0208] Register 120 can refer to the above reference. Figure 4 One of the register files described in file 120.

[0209] When an output command signal is received for outputting an operation value to the feature map storage unit 220, register 120 outputs the operation value through the output signal line F_out_00 connected to the feature map storage unit 220, and the output operation value can be stored in the feature map storage unit 220. Such a register 120 may be provided optionally.

[0210] When register 120 is not provided, the operation value of PE_00 can be directly transferred to and stored in the feature map storage unit 220.

[0211] PE_00 can be connected to signal line F_out_00, to which output data is transmitted when the MAC operation is complete. Signal line F_out_00 can be connected to internal memory 200 or can be configured to be connected to a separate vector processing unit or activation function operation unit.

[0212] More specifically, the processing element according to the example of this disclosure can be configured to transfer received weight data to another processing element. Therefore, since the transferred weight data can be reused within the processing element array, the frequency of reloading weight data from internal memory 200, on-chip memory 3000, and / or main memory 4000 can be reduced.

[0213] In addition to register 120, PE array 100 may include a first multiplexer MUX1 in multiplexers (MUX1, MUX2, ...) 130 corresponding to PE_00 and a delay buffer (i.e., Z...). -k 140.

[0214] Depending on the operating mode, the first multiplexer MUX1 can transmit weighted data output from delay buffer 140 or weighted data output from PE_00 to an adjacent processing element.

[0215] Specifically, when the mode selector 310 receives a selection signal SELECT_0 for operation in the first mode, the first multiplexer MUX1 can operate in the first mode.

[0216] When the first multiplexer MUX1 receives the selection signal SELECT_1 for operation in the second mode from the mode selector 310, it can operate in the second mode.

[0217] In the first mode, the first multiplexer MUX1 can transmit weighted data output from the first processing element PE1 to adjacent processing elements. Here, the weighted data can be transmitted to at least one processing element adjacent to the first processing element PE1. However, the term "adjacent processing element" is used only for the convenience of describing this disclosure, and "adjacent processing element" can refer to "corresponding processing element".

[0218] In the second mode, the first multiplexer MUX1 can transmit weight data output from the delay buffer 140 to an adjacent processing element. The weight data output from the delay buffer 140 can be weight data delayed by a preset number of clock cycles.

[0219] In this way, the delayed weight data can be transmitted to at least one processing element connected to PE_00. In various examples, the delayed weight data can be transmitted delayed and sequentially to at least one processing element corresponding to a column of processing elements connected to the first processing element PE1.

[0220] In other words, for each clock cycle, a specific processing element can transfer input weights to another adjacent processing element or an adjacent delay buffer. A multiplexer can be provided for this operation.

[0221] In other words, the first multiplexer MUX1 can be configured to receive weights output from a specific processing element and weights output from the delay buffer 140.

[0222] In other words, the first multiplexer MUX1 can be configured to receive weighted data output from the delay buffer 140 and the processing element PE_00.

[0223] The delay buffer 140 temporarily stores the weight data W_in_0 transmitted from the weight storage unit 210 for a preset clock cycle, and then outputs it. The weight data W_in_0 output from the delay buffer 140 is input to the first multiplexer MUX1. The weight data W_in_0 output from the delay buffer 140 can be weight data delayed by a preset number of clock cycles as described above. The delay buffer 140 may not operate in the first mode, but only in the second mode.

[0224] In other words, the first multiplexer MUX1 can select the first input in the first mode and the second input in the second mode.

[0225] When performing a convolution operation, the feature map data and kernel data (i.e. weight data) calculated using the feature map data can be in matrix form.

[0226] Based on the delay unit (i.e., delay buffer) of the processing element array according to the example of this disclosure, a plurality of processing elements corresponding to at least one column of processing elements in the processing element array can be configured to perform depthwise convolution operations using the delay buffer.

[0227] In other words, when a convolution operation is performed on matrix-type kernel data in a specific processing element by sliding the delay buffer 140 on the matrix-type feature map data with a preset stride, a portion of the kernel data can be reused for convolution operations in other adjacent processing elements.

[0228] Instead of repeatedly loading kernel data from weight storage unit 210 to PE array 100, the performance of depthwise convolution operations can be improved by reusing a portion of the reused kernel data using delay buffer 140.

[0229] Meanwhile, in the PE array 100, the processing elements operating in the second mode are enabled by the enable signal En0, and the remaining non-operating processing elements can be disabled, thereby reducing the power consumption of the NPU 1000.

[0230] In the following text, reference will be made to Figure 7 Describes the PE array in which these processing elements are configured in a matrix.

[0231] Figure 7 The structure of a processing element array according to an example of this disclosure is illustrated. In the presented example, redundant descriptions of the operation of the processing elements may be omitted.

[0232] refer to Figure 7 PE array 100 may include multiple PEs, and the multiple PEs include multiple PE rows and multiple PE columns.

[0233] Each PE in the PE array 100 can be configured to receive weights via the W_in signal line connected to the weight storage unit 210, and can be connected to the F_in signal line connected to the feature map storage unit 220.

[0234] The PE array 100 can operate in the first mode or the second mode according to the selection signals SELECT_0 and SELECT_1 of the mode selector 310.

[0235] When the mode selector 310 receives a selection signal SELECT_0 for operation in the first mode, the received selection signal is transmitted to multiplexers MUX1 and MUX2, causing multiplexers MUX1 and MUX2 to operate in the first mode. Conversely, when the mode selector 310 receives a selection signal SELECT_1 for operation in the second mode, the received selection signal is transmitted to multiplexers MUX1 and MUX2, causing multiplexers MUX1 and MUX2 to operate in the second mode.

[0236] Here, the first mode can refer to the operation mode used for standard convolution operations or pointwise convolution operations, and the second mode can refer to the operation mode used for depthwise convolution operations.

[0237] In this disclosure, a multiplexer may also be referred to as a selector or a switch.

[0238] Each of the first multiplexers MUX1 can be connected to the output line of weighted data for at least k steps of PE in the column direction, vertical direction, or a first direction. The number of first multiplexers MUX1 can be equal to the number of processing elements corresponding to (k-step) PE rows. For example, a PE row can include M processing elements. However, this disclosure is not limited thereto.

[0239] The second multiplexer MUX2 can be connected to the input row of feature map data of at least (k-step) PE rows in the row direction (i.e., the horizontal direction or the second direction). The number of second multiplexers MUX2 can be at least (k-step). However, this disclosure is not limited thereto.

[0240] Here, "k" can be the size of the weighted kernel. For example, if the kernel size is 3×3, then k equals 3.

[0241] Here, stride refers to the stride value of the convolution. For example, stride can be an integer value greater than or equal to one. For example, if k is "3" and stride is "1", then each of the first multiplexers MUX1 can be connected to the weight data output lines of at least two PE rows, and each of the second multiplexers MUX2 can be connected to the input lines of the feature map data of at least two PE rows. Here, the input lines of the feature map data can be a signal bus consisting of M channels. Here, "M" can refer to the number of processing elements arranged in a PE row. However, this disclosure is not limited thereto.

[0242] In other words, the first multiplexer MUX1 can be connected to the output line of the weighted data of the processing element and the corresponding delay buffer Z. -k The output line of weighted data about at least (k-step) PE rows.

[0243] In other words, the second multiplexer MUX2 can be connected to the input line of the feature map storage unit 220, which outputs feature map data with respect to at least (k-step) PE rows. The number of multiplexers MUX1 and MUX2 can be determined with reference to, but is not limited to, the kernel size of the artificial neural network model to be processed.

[0244] More specifically, the first PE line can refer to a first plurality of processing elements (PE_00, PE_01, ...), while the second PE line can refer to a second plurality of processing elements (PE_10, PE_11, ...).

[0245] More specifically, the first PE column can refer to a third or more processing elements (PE_00, PE_10, PE_20, PE_30, ...), and the second PE column can refer to a fourth or more processing elements (PE_01, PE_11, PE_21, PE_31, ...).

[0246] In the following description, a PE array 100 according to an example of this disclosure will be used as an example of a first mode.

[0247] In the first mode, the weight data output from the weight storage unit 210 is input to each of the multiple PE columns via each W_in signal line. For example, the first weight data is input to the PEs (PE_00, PE_10, PE_20, PE_30, ...) of the first PE column corresponding to the W_in_0 signal line. In this case, the first weight data input to the first processing element PE_00 is output from the first processing element PE_00 and input to the first multiplexer MUX1 corresponding to the first processing element PE_00. In the first mode, the first multiplexer MUX1 transmits the weight data output from the first processing element PE_00 to the second processing element PE_10 of the next adjacent row in the next clock cycle.

[0248] Then, the weight data input to the second processing element PE_10 is output from the second processing element PE_10 in the next clock cycle and input to another first multiplexer MUX1 connected to the output signal line of the second processing element PE_10. In the first mode, the first multiplexer MUX1 also transmits the weight data output from the second processing element PE_10 to the third processing element PE_20 in the next adjacent row in the same column.

[0249] Then, the weight data input to the third processing element PE_20 is output from the third processing element PE_20 and input to the fourth processing element PE_30 in the next adjacent column. This operation can continue until the last processing element of the matrix.

[0250] For example, weighted data can be transmitted sequentially along the PEs in the first PE column connected by the W_in_0 signal line.

[0251] In other words, the PE columns in the PE array 100 according to the example of this disclosure can be configured to have a pipeline structure configured to transfer weight data to adjacent PEs.

[0252] The same operation as described above can also be performed on the second PE column corresponding to the W_in_1 signal line, and the same operation can be performed on multiple PE columns of the PE array 100.

[0253] In the first mode, feature map data output from feature map storage unit 220 can be input to each of multiple PE rows via each F_in signal line. For example, feature map data can be unicast or broadcast to PE rows such as F_in_00, F_in_10, F_in_20, and F_in_30 signal lines. This operation can continue until the last processing element of the matrix.

[0254] The F_in signal line can be a bus comprising M channels. The F_in signal line can also be a bus line comprising individual signal lines corresponding to a PE row. For example, if the PEs (PE_00, PE_01, ...) of the first PE row are configured to have 64 processing elements, then the F_in_00 signal line can be a bus line configured with a set of 64 individual lines. Furthermore, the F_in signal line can be configured to unicast individual feature map data to each processing element in the PE row or broadcast the same feature map data.

[0255] In this way, when feature map data and weight data are input to each PE, a MAC operation is performed on the feature map data and weight data input from each PE in each clock cycle, and the operation result data (i.e. feature map data) calculated by the operation can be output from each PE and stored in the feature map storage unit 220.

[0256] In more detail, Figure 7 Not shown in the image, but referenced. Figure 6 Each PE can be configured to include an F_out signal line that outputs the MAC operation value from a register in which the MAC operation value is stored. However, this disclosure is not limited thereto, and the F_out signal line can be configured to be connected to internal memory 200 or other additional operating units.

[0257] In the following description, the PE array 100 according to the example of this disclosure will be described using the second mode as an example.

[0258] When the mode selector 310 outputs a selection signal SELECT_1 for operation in the second mode, the selection signal is transmitted to multiplexers MUX1 and MUX2, causing the PE array 100 to operate in the second mode. Here, the second mode can refer to the operation mode used for depthwise convolution operations.

[0259] In the second mode, the weight data output from the weight storage unit 210 is input to each of the multiple PE columns via the W_in signal line. In this case, a delay buffer Z provided in some PE rows (i.e., k-step PE rows) of the multiple PE columns can be utilized. -k To reuse the weighted data.

[0260] Each W_in signal line connected to the weighted storage unit 210 can have a branch, and through this branch, a connection can be made between each W_in signal line and the delay buffer Z. -k The PE column of the delay buffer Z. -k The output is weighted data delayed by k clock cycles. That is, multiple delay buffers Z... -k Cascading can be increased by the number of clock cycles to be delayed.

[0261] For example, in the second mode, when weight data is input to the first processing element PE_00 via the W_in_0 signal line connected to the weight storage unit 210, the corresponding weight data is also transmitted through this branch to the delay buffer Z corresponding to the first processing unit PE_00. -k Transmitted to delay buffer Z -k The weighted data is delayed by k clock cycles and transmitted to the first multiplexer MUX1 corresponding to the first processing element PE_00. The delayed weighted data is then transmitted to the second processing element PE_10 in the next row via the first multiplexer MUX1 corresponding to the first processing element PE_00. Furthermore, the weighted data is transmitted via the delay buffer Z... -k The output delay weight data is input into the next delay buffer Z of the second processing element PE_10 corresponding to the next row. -k .

[0262] Transmitted to the delay buffer Z corresponding to the second processing unit PE_10 -k The delayed weight data is delayed by k clock cycles and transmitted to the first multiplexer MUX1 corresponding to the second processing unit PE_10. This delayed weight data is then input to the third processing element PE_20 in the next adjacent row via the first multiplexer MUX1 corresponding to the second processing element PE_10. This operation can continue up to the (k-step) processing element of each PE column of the PE array 100.

[0263] Thus, a delay buffer Z is provided for the (k-step) PE row. -k The structure is a cascaded structure, so it is possible to expand the design of the processing element array.

[0264] In the second mode, the feature map data stored in the feature map storage unit 220 is broadcast to multiple PE rows via an F_in signal line with k branches. For example, the F_in signal line with k branches can be connected to the PEs of each of the first, second, and third PE rows.

[0265] In the second mode, the F_in signal line can have k branches, and these branches can be connected to the (k-step) input line of the PE row corresponding to the F_in signal line.

[0266] In the second mode, the feature map data input through k branches is transmitted to the PEs in the first PE row and to the second multiplexer MUX2 connected to the PEs in each of the second and third PE rows. Therefore, the feature map data is broadcast to the PEs in each of the first, second, and third PE rows.

[0267] Therefore, when feature map data (i.e., input feature map data) and weight data are input to each PE, a MAC operation is performed on the input feature map data and weight data in each PE. The operation result data calculated by the operation (i.e., output feature map data) can be output from each PE and stored in the internal memory 200 or the feature map storage unit 220.

[0268] In the second mode, multiple (k) PE lines can be enabled by the En signal, and the remaining inactive PE lines can be deactivated to save power on the NPU 1000.

[0269] In the second mode, the feature map signal changes compared to the first mode as the feature map data is broadcast to specific PE rows. Therefore, this will be discussed later. Figure 8 Describe the second mode in detail.

[0270] In the following text, reference will be made to Figure 8 Describe in detail the operation of the PE array in the first mode.

[0271] Figure 8 The structure of a processing element array operating in a first mode is illustrated by examples of this disclosure.

[0272] For ease of description, Figure 7 The components shown are those that do not operate substantially in the first mode. Figure 8 The middle part will be omitted.

[0273] In the example presented, the PE array is described as comprising N×M processing elements (PE). Here, N and M can be integers, and N and M can be the same number.

[0274] refer to Figure 8 In the first mode, the PE array 100 can be configured in the output fixed shrink array method, but is not limited thereto.

[0275] In the first mode, the weight data output from the weight storage unit 210 is transmitted to each PE constituting the PE array 100 via the W_in signal lines (W_in_0, W_in_1, ... W_in_M) connected to the output of the weight storage unit 210. For example, the W_in_0 signal lines connected along the first PE column of the plurality of PE columns can be pipelined so that the weight data can be cascaded along the PE columns (PE_00, PE_10, PE_20, PE_30, ... PE_N0), and the W_in_1 signal lines connected along the second PE column of the plurality of PE columns can be pipelined so that the weight data can be cascaded along the PE columns (PE_01, PE_11, PE_21, PE_31, ... PE_N1).

[0276] In the first mode, feature map data output from feature map storage unit 220 is transmitted to each PE constituting PE array 100 via F_in signal lines (F_in_00, F_in_01, ..., F_in_NM) connected to the output of feature map storage unit 220. For example, feature map data can be provided to the first PE row (PE_00, PE_01, ..., PE_0M) via signal lines (F_in_00, F_in_01, ..., F_in_0M) connected to the first PE row among the multiple PE rows, and can be provided to the second PE row (PE_10, PE_11, ..., PE_1M) via signal lines (F_in_10, F_in_11, ..., F_in_1M) connected to the second PE row among the multiple PE rows.

[0277] Furthermore, feature map data can be provided to the third PE row (PE_20, PE_21, ..., PE_2M) via signal lines (F_in_20, F_in_21, ..., F_in_2M) connected to the third PE row among multiple PE rows, and can be provided to the fourth PE row (PE_30, PE_31, ..., PE_3M) via signal lines (F_in_30, F_in_31, ..., F_in_3M) connected to the fourth PE row among multiple PE rows.

[0278] When weight data and feature map data are input in this manner, each PE performs a MAC operation on the weight data and feature map data and transmits the operation result to the feature map storage unit 220.

[0279] Therefore, the PE array 100 can be connected to the weight storage unit 210 via the W_in signal lines (W_in_0, W_in_1, ..., W_in_M), and can be connected to the feature map storage unit 220 via the F_in signal lines (F_in_00, F_in_01, ..., F_in_NM).

[0280] In addition, the signal lines (F_in_00, F_in_01, ..., F_in_0M) can be a signal bus including M-1 signal lines.

[0281] Each signal line can be coupled to the corresponding PE of the PE array 100. Therefore, the feature map data transmitted through the corresponding F_in signal line can be executed via unicast communication, which is point-to-point communication.

[0282] Simultaneously, a signal line can be used as the input signal line for W_in_0. The corresponding signal line can be connected to each PE in the column direction of the PE array 100. Therefore, for each clock cycle, the weight data transmitted to a PE (e.g., PE_00) via the W_in_0 signal line can be shifted, i.e., transferred to another PE in the next row (e.g., PE_10). In this way, the weight data is reused in the PE array 100 to minimize resource consumption and memory usage for operation.

[0283] In the following text, reference will be made to Figure 9 Describe in detail the operation of the processing element array in the second mode.

[0284] Figure 9 The structure of a processing element array operating in a second mode is illustrated by example of this disclosure.

[0285] For ease of description, Figure 7 Of the components shown, those that are not substantially operating in the second mode will Figure 9 The middle part is omitted.

[0286] In the example presented, the PE array 100 is described as consisting of N×M PEs. Here, N and M can be integers, and N and M can be the same number.

[0287] In the second mode, for example, if the kernel size is 3×3, the feature map data output from the feature map storage unit 220 is broadcast to k PE lines via signal lines (F_in_00, F_in_01, ..., F_in_0M) connected to k branches corresponding to the F_in signal line.

[0288] For example, in the second mode, the F_in_00 signal is broadcast to the first PE column (PE_00, PE_10, PE_20) connected to the k branches.

[0289] For example, in the second mode, the F_in_01 signal is broadcast to the second PE column (PE_01, PE_11, PE_21) connected to the k branches.

[0290] In the second mode, the weight data output from the weight storage unit 210 is transmitted to each PE of the PE array 100 via the W_in signal lines (W_in_0, W_in_1, ..., W_in_M) connected to the output of the weight storage unit 210. The corresponding weight data can then be transmitted to the delay buffer Z corresponding to each PE. -k It can also be transmitted to the delayed buffer Z. -k The next PE in each connected row.

[0291] For example, weighted data is transmitted to PE_00 via the W_in_0 signal line connected to the first PE column among multiple PE columns. The corresponding weighted data undergoes a MAC operation in PE_00 and is then transmitted to the delay buffer Z via the branch corresponding to PE_00. -k The weight data can be delayed by k clock cycles. This delayed weight data is then transmitted along the column direction to the next adjacent PE, PE_10.

[0292] Subsequently, the delayed weight data can undergo a MAC operation in PE_10 and be transmitted to the delay buffer Z through the branch corresponding to PE_10. -k And then, it is delayed by k clock cycles again. That is, the weight data is transmitted from PE_00 to PE_20 after a delay of 2k clock cycles.

[0293] The delayed weighted data undergoes a MAC operation in PE_20. This operation can be performed on the PE column corresponding to each signal line.

[0294] Thus, through the delay buffer Z -k The weight data transmitted with delay can be broadcast delayed in the direction of each PE column. Therefore, the weight data transmitted from the weight storage unit 210 can be reused in each PE through a delayed broadcasting technique that delays the weight data.

[0295] In the second mode, the feature map data output from the feature map storage unit 220 is broadcast to a predetermined PE row of the PE array 100 via F_in signal lines (F_in_00, F_in_01, ..., F_in_0M) connected to the output of the feature map storage unit 220.

[0296] In the second mode, feature map data input through k branches is transmitted to the first PE row (PE_00, PE_01, ..., PE_0M), the second PE row (PE_10, PE_11, ..., PE_1M), and the third PE row (PE_20, PE_21, ..., PE_2M). Therefore, the feature map data is broadcast to the first PE row (PE_00, PE_01, ...), the second PE row (PE_10, PE_11, ...), and the third PE row (PE_20, PE_21, ...). Thus, when feature map data (i.e., input feature map data) and weight data are input to each PE, a MAC operation is performed on the feature map data and weight data input to each PE for each clock cycle. The operation result data (i.e., output feature map data) calculated through the operation can be output from each PE and stored in the internal memory 200 or the feature map storage unit 220.

[0297] In other words, each PE receives weight data and feature map data, and performs MAC operations on the weight data and feature map data.

[0298] In other words, the PE array can be configured to include PE_00 for receiving weight data and a delay buffer Z configured to receive and delay the received weight data for a specific number of clock cycles before outputting it to PE_10. -k And a broadcast signal line F_in_00 configured to simultaneously provide feature map data to PE_00 and PE_10. Therefore, the delay unit can be configured to process depthwise convolutions while reusing weight data.

[0299] In other words, depthwise convolution that can reuse data can be achieved by providing a delay buffer for the transmission weights, two processing elements corresponding to the input and output of the delay buffer, and a signal line for simultaneously inputting the feature map to the two processing elements.

[0300] Furthermore, when the weight kernel size of the ANN model is 3×3, depthwise convolution that can reuse data can be achieved by providing two delay buffers for transmitting weights, three processing elements corresponding to the inputs and outputs of the two delay buffers, and signal lines for simultaneously inputting feature maps into the three processing elements.

[0301] If the weight kernel of the ANN model is 3×3, then the first PE column (PE_00, PE_10, PE_20) can process the first depthwise convolution while delaying the first weight kernel. In addition, the second PE column (PE_01, PE_11, PE_21) can process the second depthwise convolution while delaying the second weight kernel.

[0302] In other words, two delay buffers, three processing elements, and a broadcast signal line with three branches can be considered as a unit capable of processing depthwise convolutions of a 3×3 kernel. Therefore, if the number of these units is increased, the number of depthwise convolutions that can be processed simultaneously can also be increased proportionally.

[0303] Thus, if the PE array 100 implements delayed broadcasting through a delay buffer when transmitting weight data to each PE, the energy consumed in the NPU can be reduced, and the efficiency of the PE array and the throughput of the NPU can be improved in the second mode operation. Furthermore, the PE array 100 can minimize power consumption by disabling components that do not operate in the second mode.

[0304] In other words, comparison Figure 8 and Figure 9 The first mode is configured such that each of the F_in signal lines operates individually for each PE, and the second mode is configured such that at least some PEs in each PE column are connected via branches of the F_in signal line, thereby configuring some PEs in each PE column to be broadcast.

[0305] In the following text, reference will be made to Figure 10 Describes a PE array for operation in low-power mode by grouping PEs to enable or disable each PE operating in a first or second mode.

[0306] Figure 10 This is a schematic configuration diagram illustrating the structure of the processing element array according to examples of this disclosure.

[0307] refer to Figure 10 The PE array 100 can identify PEs operating according to a first mode and a second mode. That is, multiple PEs operating according to the first mode can be grouped, and multiple PEs operating according to the second mode can be grouped.

[0308] refer to Figure 7 The PEs corresponding to the first group 150 and the second group 160 can operate in the first mode, while the PEs corresponding to the first group 150 can operate only in the second mode.

[0309] When a selection signal for operation in the first mode is received from the mode selector 310, the PE array 100 transmits an enable signal to each PE in the first group 150 and the second group 160 to enable the PEs in the first group 150 and the second group 160. Therefore, the PEs in the first group 150 and the second group 160 can be enabled to perform operation in the first mode.

[0310] When a selection signal for operation in the second mode is received from the mode selector 310, the PE array 100 transmits an enable signal to each PE in the first group 150 to enable the PEs in the first group 150 to operate in the second mode. Therefore, the PEs in the first group 150 can be enabled to perform operation in the second mode, and the second group 160 can be disabled. This allows PEs not used in the second mode to be disabled, thereby enabling low-power mode operation of the PE array 100.

[0311] In various examples, the PE array 100 can be configured to include multiple first groups 150. Therefore, the utilization of the PE array 100 can be improved during depthwise convolution operations on the PE array 100.

[0312] In various examples, the PE array 100 can individually drive the corresponding PE by applying an enable signal according to each layer of the ANN model. The PE array 100 analyzes the layer structure of the ANN model and individually enables (turns on) or disables (turns off) the PE corresponding to each layer of the ANN model, thereby minimizing the power consumption of the NPU.

[0313] In the following text, reference will be made to Figure 11 Describes an array of processing elements with delay buffers of various structures.

[0314] Figure 11 The structure of the processing element array is illustrated by examples from this disclosure. In the presented examples, for ease of description, descriptions of those redundant elements as described above will be omitted.

[0315] refer to Figure 11 The k-step PEs in the PE array 100 are respectively connected to two delay buffers (e.g., the first delay buffer Z). -k Second delay buffer Z -2k Specifically, the first delay buffer Z is connected to each PE in the second PE row (PE_10, PE_11, ..., PE_1M) of the PE array 100. -k It can output weighted data delayed by k clock cycles. Furthermore, a second delay buffer Z is connected to each PE in the third PE row (PE_20, PE_21, ..., PE_2M) of the PE array 100. -2k It can output weight data with a delay of 2k clock cycles.

[0316] When the mode selector 310 receives the selection signal SELECT_1 for operation in the second mode, the weight data transmitted from the weight storage unit 210 can be transmitted to the PEs in the k PE rows via the W_in signal line connected to the weight storage unit 210. For example, the weight data is transmitted to PE_00 via the W_in_0 signal line, and the corresponding weight data is transmitted to the first delay buffer Z via the branch corresponding to PE_00. -k Transmitted to the first delay buffer Z -k The weight data is delayed by k clock cycles and transmitted to the first multiplexer MUX1 corresponding to the output line of PE_00. The weight data delayed by k clock cycles is then transmitted to the adjacent PE_10 via the first multiplexer MUX1 corresponding to the output line of PE_00. Furthermore, the corresponding weight data is also transmitted to the second delay buffer Z via the branch corresponding to PE_00. -2k And transmitted to the second delay buffer Z -2k The weight data is delayed by 2k clock cycles and transmitted to the first multiplexer MUX1 corresponding to the output line of PE_10. The weight data delayed by 2k clock cycles is then transmitted to the adjacent PE_20 via the first multiplexer MUX1 corresponding to the output line of PE_10.

[0317] Thus, each PE corresponding to the (k-step) PE row provides two delay buffers Z. -k and Z -2k The structure allows custom designs of PE arrays to be used to compute ANN models with small kernel sizes.

[0318] More specifically, the PE array 100 can be configured to differently set the number of delay clock cycles of the delay buffer, which is suitable for a specific processing element to implement the second mode.

[0319] In detail, Figure 7 The PE array 100 shown and Figure 11 The PE array 100 shown is configured to operate in essentially the same second mode.

[0320] In the following text, reference will be made to Figure 12 Describe the structure of the PE array of the second multiplexer MUX2 for individually enabling / disabling the input lines of feature map data connected to multiple PE rows.

[0321] Figure 12 The structure of the processing element array is illustrated by examples from this disclosure.

[0322] refer to Figure 12At least one second multiplexer MUX2 connected to the feature map storage unit 220 can be connected to at least one PE row. Each second multiplexer MUX2 can be individually enabled or disabled by the MUX2 selection signal SELECT_M2 transmitted from the mode selector 310. That is, the MUX2 selection signal SELECT_M2 can be configured to control the on / off signal of the second multiplexer MUX2 respectively.

[0323] The MUX2 selection signal SELECT_M2 transmitted from the mode selection unit 310 is connected to each of the second multiplexers MUX2. The number of signal lines (i.e., buses) of the mode selector 310 corresponding to the MUX2 selection signal SELECT_M2 can correspond to the number of second multiplexers MUX2.

[0324] Each secondary multiplexer MUX2 can be individually enabled or disabled via the MUX2 select signal SELECT_M2. For example, the number of signal lines transmitting the MUX2 select signal SELECT_M2 can be the same as the number of secondary multiplexers MUX2.

[0325] When the mode selection signal SELECT controls the first multiplexer MUX1, the first multiplexer MUX1 can control the corresponding processing element to operate in the first mode (i.e., standard convolution) or the second mode (i.e., depthwise convolution).

[0326] However, in Figure 12 In the example presented, each PE row is configured to be selectively controlled by the MUX2 select signal SELECT_M2.

[0327] Therefore, multiple PE lines can be configured to simultaneously receive feature map data in a broadcast manner based on the MUX2 selection signal SELECT_M2.

[0328] Specifically, a particular PE line can be configured to receive feature map data in a unicast manner based on the MUX2 selection signal SELECT_M2.

[0329] In addition, specific PE lines can be accessed via Figure 6 The enable signal En0 shown is used to enable or disable the device. For example, when the second PE row (PE_10, PE_11, ...) is disabled by the enable signal En0, the delay buffer Z... -k The delayed weights can be transferred to the third PE row. Therefore, convolution operations can be implemented or modified in various ways within the PE array 100.

[0330] In addition, in some implementations, a separate delay buffer Z can be set for each PE row. -kThe delayed clock.

[0331] Thus, since the second multiplexer MUX2 connected to the output of the feature map storage unit 220 is enabled or disabled by the MUX2 selection signal SELECT_M2, the PE array 100 can easily perform operations even if the kernel size of the ANN model changes and the kernel size of each layer of the ANN model is different.

[0332] In the following text, reference will be made to Figures 13 to 16 The operation of the processing element array that performs depthwise convolution in the second mode is described in detail.

[0333] Figure 13 This is used to explain the weight data and feature map data according to examples of this disclosure. Figure 14 is used to explain the depthwise convolution operation on the weight data and feature map data according to examples of this disclosure. Figure 15 The structure of the processing element array is illustrated by examples from this disclosure. Figure 16 Used to explain, based on examples of this disclosure, the weight data stored in the delay buffer over time.

[0334] First, refer to Figure 13 Suppose that depthwise convolution operations are performed on kernel data 1300 in the form of a 3×3×m matrix and input feature map data 1310 in the form of a 5×5×M matrix. Assume that the stride between the kernel data 1300 and the input feature map data 1310 used for the convolution operation is "1".

[0335] In this case, the convolution operation is performed by sliding a kernel data 1300 of size 3×3×m over an input feature map data 1310 of size 5×5×M by one stride, such that each value of the kernel data 1300 is multiplied by each value of the overlapping input feature map data 1310, and all multiplied values ​​are summed.

[0336] Specifically, will refer to Figures 14 to 16 Describe each step.

[0337] In step (1), as Figure 14 The first kernel (i.e., a0, b0, c0, d0, e0, f0, g0, h0, i0) of the kernel data 1300 of size 3×3×m shown slides with a stride of 1 and is computed using the overlapping first feature map portions (i.e., A0, B0, C0, F0, G0, H0, K0, L0, M0) in the elements of the input feature map data 1310.

[0338] At this point, step (1) is in Figure 15In PE_00, nine clock cycles are processed sequentially. That is, for the convolution operation of the first kernel and the first feature map part, PE_00 requires nine clock cycles to operate.

[0339] At this time, in step (1), the signal W_in_0 of PE_00 (which is as follows) Figure 16 The signals of the first core shown) and signals F_in_00 (A0, B0, C0, F0, G0, H0, K0, L0, M0) (which are the signals of the first feature map portion) are input to the first core in nine clock cycles. Figure 15 PE_00. That is, each element of the first kernel (a0, b0, c0, d0, e0, f0, g0, h0, i0) and each element of the first feature map part (A0, B0, C0, F0, G0, H0, K0, L0, M0) are sequentially input into PE_00.

[0340] like Figure 16 As shown, the MAC operation in step (1) is performed on the elements (A0, B0, C0, F0, G0, H0, K0, L0, M0) of the feature map data 1310 input to the PE and the elements (a0, b0, c0, d0, e0, f0, g0, h0, i0) of the input kernel data 1300. This operation can correspond to Figure 14 Step (1) is described in the text.

[0341] Compared to step (1), step (2) is delayed by three clock cycles. In step (2), the following will be performed: Figure 14 The first kernel (i.e., a0, b0, c0, d0, e0, f0, g0, h0, i0) of the kernel data 1300 with a size of 3×3×m shown slides with a stride of 1 and is computed using the overlapping second feature map portions (i.e., F0, G0, H0, K0, L0, M0, P0, Q0, R0) from the elements of the input feature map data 1310.

[0342] At this point, step (2) is in Figure 15 In PE_10, nine clock cycles are processed sequentially. That is, for the convolution operation of the first kernel and the second feature map part, which is delayed by three clock cycles compared to step (1), PE_10 requires nine clock cycles to operate.

[0343] At this point, in step (2), the signal W_in_0(Z) of PE_10, which is delayed by three clock cycles, is... -3 (It is like) Figure 16The signals of the first core shown) and signals F_in_00 (F0, G0, H0, K0, L0, M0, P0, Q0, R0) (which are signals used for the second feature map portion) are input to the second core in nine clock cycles. Figure 15 The PE. That is, each element of the first kernel (a0, b0, c0, d0, e0, f0, g0, h0, i0) and each element of the second feature map part (F0, G0, H0, K0, L0, M0, P0, Q0, R0) are sequentially input into PE_10 after a delay of three clock cycles.

[0344] As in Figure 14 In step (2), since the operation is performed by sliding the kernel data 1300 over the feature map data 1310 by a step size of 1, the elements (a0, b0, c0, d0, e0, f0, g0, h0, i0) of the kernel data 1300 can be reused after three clock cycles.

[0345] In order to reuse kernel data 1300 for operations, kernel data 1300 output from weighted storage unit 210 is transferred to delay buffer Z corresponding to PE_00. -k Transmitted to delay buffer Z -k The kernel data 1300 is delayed by three clock cycles, and the delayed kernel data is transmitted to the next line PE_10 through the first multiplexer MUX1 corresponding to PE_00.

[0346] like Figure 16 As shown, the MAC operation of step (2) is performed on the elements (F0, G0, H0, K0, L0, M0, P0, Q0, R0) of the feature map data 1310 input to PE_10 and the kernel data delayed by three clock cycles. This operation can correspond to Figure 14 Step (2) is described in the text.

[0347] Compared to step (2), step (3) is delayed by three clock cycles. In step (3), the following will be performed: Figure 14 The first kernel (i.e., a0, b0, c0, d0, e0, f0, g0, h0, i0) of the kernel data 1300 with a size of 3×3×m shown slides step 1 and is computed using the overlapping third feature map portion (i.e., K0, L0, M0, P0, Q0, R0, U0, V0, W0) in the elements of the input feature map data 1310.

[0348] At this point, step (3) is in Figure 15In PE_20, nine clock cycles are processed sequentially. That is, for the convolution operation of the first kernel and the third feature map part, which is delayed by three clock cycles compared to step (2), PE_20 requires nine clock cycles to operate.

[0349] At this point, in step (3), the signal W_in_0(Z) of PE_20, which is delayed by six clock cycles, is... -6 (It is like) Figure 16 The signals of the first core shown) and F_in_00 (K0, L0, M0, P0, Q0, R0, U0, V0, W0) (which are the signals used for the third feature map part) are input in nine clock cycles. Figure 15 PE_20. That is, each element of the first kernel (a0, b0, c0, d0, e0, f0, g0, h0, i0) and each element of the third feature map (K0, L0, M0, P0, Q0, R0, U0, V0, W0) delayed by six clock cycles are sequentially input into PE_20.

[0350] As in Figure 14 In step (3), since the operation is performed by sliding the kernel data 1300 on the feature map data 1310 by a step size of 1, all elements (a0, b0, c0, d0, e0, f0, g0, h0, i0) of the kernel data 1300 can be reused.

[0351] The kernel data 1300 output from the weighted storage unit 210 is transmitted to the delay buffer Z in response to PE_10. -k This is to perform the operation by reusing kernel data 1300. The data is then transferred to the delay buffer Z. -k The kernel data 1300 is further delayed by three clock cycles, and the delayed kernel data is transmitted to PE_20 through the first multiplexer MUX1 corresponding to PE_10.

[0352] like Figure 16 As shown, the MAC operation of step (3) is performed on the elements (K0, L0, M0, P0, Q0, R0, U0, V0, W0) of the feature map data 1310 input to PE_20 and the kernel data delayed by six clock cycles. This operation can correspond to Figure 14 Step (3) is described in the text.

[0353] In summary, after nine clock cycles in step (1), the convolution of PE_00 is completed. Therefore, the accumulated value of PE_00 can be stored in the internal memory 200 or the feature map storage unit 220. After nine clock cycles in step (2), the convolution of PE_10 is completed. Therefore, the accumulated value of PE_10 can be stored in the internal memory 200 or the feature map storage unit 220. After nine clock cycles in step (3), the convolution of PE_20 is completed. Therefore, the accumulated value of PE_20 can be stored in the internal memory 200 or the feature map storage unit 220.

[0354] Return to reference Figure 6 The accumulated value of each PE can be stored in register 120. Additionally, each PE can communicate with the feature map storage unit 220 via the F_out signal line. After the MAC operation is completed, the accumulated value stored inside each PE can be initialized by receiving a reset signal in a specific clock cycle, thus putting the initialized PE in a ready state to execute a new MAC operation.

[0355] When each step is completed and the result is stored in the internal memory 200 or the feature map storage unit 220, the value stored in each processing element can be determined by... Figure 6 The reset signal shown is initialized. Therefore, the PE_00 preparation process step (4) of step (1) is completed. Therefore, the PE_10 preparation process step (5) of step (2) is completed. Therefore, the PE_20 preparation process step (6) of step (3) is completed.

[0356] Subsequently, as mentioned above Figure 14 The steps described in (4), (5), (6), etc. can be performed by reusing the weighted data.

[0357] For steps (4), (5), and (6), the signal line F_in_00 can sequentially provide new elements (B0, C0, D0, G0, H0, I0, L0, M0, N0, Q0, R0, S0, V0, W0, X0) of the feature map data 1310 to the first PE column (PE_00, PE_10, PE_20). However, this disclosure is not limited thereto, and steps (4), (5), and (6) can also be processed in another PE column. Furthermore, steps (1), (2), and (3) and steps (4), (5), and (6) can be processed sequentially in one PE column or in parallel in different PE columns.

[0358] In other words, when steps (1) to (3) are completed, steps (4) to (6) can be repeated in the same way. Therefore, the first PE column (PE_00, PE_10, PE_20) can sequentially receive various kernels and various feature maps to process multiple depthwise convolution operations.

[0359] In this scenario, at least a portion of the kernel data 1300 can be transmitted through the delay buffer Z in the PE array 100. -k Reuse.

[0360] In other words, if no delay buffer is provided in the processing element array and kernel data is unnecessarily loaded from the weight storage unit 210 into the processing element array for each MAC operation, the reuse of kernel data becomes impossible.

[0361] However, the PE array 100 of the example according to this disclosure uses a delay buffer to reuse the weight data of the depthwise convolution to reduce the resources and memory usage used in computation. Therefore, efficient depthwise convolution operations are possible.

[0362] As described above, the PE array 100 can use the delay buffer Z provided in the PE array 100. -k To perform effective depthwise convolution operations.

[0363] like Figure 16 As shown, the elements of kernel data 1300 overlap with PE_00 and PE_10 for six clock cycles, and the elements of kernel data 1300 overlap with PE_10 and PE_20 for six clock cycles. Therefore, such overlap can be used to improve computation speed.

[0364] According to an example of this disclosure, feature map data 1310 output from feature map storage unit 220 can be broadcast to multiple PE columns via a branched F_in signal line.

[0365] For example, the F_in_00 signal line broadcasts feature map data 1310 to the first PE column (PE_00, PE_10, and PE_20). The F_in_01 signal line broadcasts feature map data to the second PE column (PE_01, PE_11, and PE_21). The F_in_0M signal line broadcasts feature map data to the MPE column (PE_0M, PE_1M, and PE_2M). Therefore, the NPU 1000 according to the example of this disclosure can be configured to perform depthwise convolution operations capable of reusing kernel data from each PE column separately.

[0366] In the following text, reference will be made to Figure 13 , 15Sections 17 and 18 describe in detail the operation of the processing element array that performs depthwise convolution when the stride value is "2".

[0367] Figure 17 This is used to explain the depthwise convolution operation on weighted data and feature map data according to the examples in this disclosure. Figure 18 This is used to explain the weight data stored in the delay buffer over time, based on the examples in this disclosure. For the sake of clarity, redundant descriptions may be omitted.

[0368] First, refer to Figure 13 Suppose we perform a depthwise convolution operation on kernel data 1300 in the form of a 3×3×m matrix and input feature map data 1310 in the form of a 5×5×M matrix. Assume the stride between the kernel data 1300 and the input feature map data 1310 used for the convolution operation is "2". That is, the stride can change from "1" to "2".

[0369] In this case, the convolution operation is performed by sliding a kernel data 1300 of size 3×3×m over an input feature map data 1310 of size 5×5×M by 2 steps, such that each value of the kernel data 1300 is multiplied by each value of the overlapping input feature map data 1310, and all multiplied values ​​are added together.

[0370] Compare Figure 14 and 17 When the stride changes from 1 to 2, steps (2) and (5) can be omitted. Therefore, refer to Figure 17 and 18 Describe each step in detail.

[0371] Figure 17 and 18 Step (1) and Figure 14 and 16 Step (1) is essentially the same. Therefore, redundant descriptions can be omitted. Figure 14 As shown, a first kernel (i.e., a0, b0, c0, d0, e0, f0, g0, h0, i0) with kernel data 1300 of size 3×3×m slides 2 steps and is computed using overlapping first feature map portions (i.e., A0, B0, C0, F0, G0, H0, K0, L0, M0) from the elements of input feature map data 1310.

[0372] At this point, step (1) is in Figure 18 In PE_00, the operations are processed sequentially over nine clock cycles. That is, for the convolution operation of the first kernel and the first feature map part, PE_00 requires nine clock cycles to perform the operation.

[0373] Figure 14and 16 Example step (2) in Figure 17 and 18 In the examples, it is basically not executed. However, according to Figure 17 and 18 The NPU 1000 implementation scheme can be used with Figure 14 and 16 The steps (2) are basically the same.

[0374] However, in Figure 17 and 18 In the example, with a step size of 2, since step (2) is unnecessary, the controller 300 of the NPU 1000 can disable the F_out signal line that outputs the MAC operation value of the exemplary PE_10 to perform step (2). That is, various step sizes can be easily adjusted without taking the MAC operation value of a specific processing element. According to the above configuration, the step size value can be easily applied by selectively controlling only the output of the F_out signal line of the PE.

[0375] Figure 17 and 18 Step (3) and Figure 14 and 16 Step (3) is essentially the same. Therefore, redundant descriptions can be omitted. Compared to step (1), step (3) is delayed by six clock cycles. In step (3), it will be as follows: Figure 17 The first kernel (i.e., a0, b0, c0, d0, e0, f0, g0, h0, i0) of the kernel data 1300 with a size of 3×3×m is slid by 2 steps and is computed using the overlapping third feature map portion (i.e., K0, L0, M0, P0, Q0, R0, U0, V0, W0) in the elements of the input feature map data 1310.

[0376] At this point, step (3) is in Figure 15 In PE_20, the operations are processed sequentially over nine clock cycles. That is, for the convolution operations of the first kernel and the third feature map portion, which are delayed by six clock cycles compared to step (1), PE_20 requires nine clock cycles to perform the operation.

[0377] In other words, the convolution of PE_00 is completed over nine clock cycles of step (1). Therefore, the accumulated value of PE_00 can be stored in the internal memory 200 or the feature map storage unit 220.

[0378] As mentioned above, the convolution of PE_10 is completed within nine clock cycles in step (2). However, the accumulated value of PE_10 may not be output.

[0379] The convolution of PE_20 is completed in nine clock cycles of step (3). Therefore, the accumulated value of PE_20 can be stored in the internal memory 200 or the feature map storage unit 220.

[0380] If each step is completed, the result can be selectively stored in internal memory 200 or feature map storage unit 220. Therefore, PE_00, which has completed step (1), is ready to process step (4). Therefore, PE_20, which has completed step (3), is ready to process step (6).

[0381] After that, Figure 14 Steps (4), (6), etc. can also be performed by reusing the weight data as described above.

[0382] For steps (4) and (6), signal line F_in_00 can sequentially provide new elements (B0, C0, D0, G0, H0, I0, L0, M0, N0, Q0, R0, S0, V0, W0, X0) of feature map data 1310 to the first PE column (PE_00, PE_10, PE_20).

[0383] In other words, when steps (1) and (3) are completed, steps (4) and (6) can be repeated in the same way.

[0384] In this scenario, at least a portion of the kernel data 1300 can be transmitted through the delay buffer Z in the PE array 100. -k It was reused.

[0385] like Figure 18 As shown, the elements of kernel data 1300 overlap with PE_00 and PE_20 by three clock cycles. Therefore, such overlap can be used to improve computation speed.

[0386] In the following text, when the PE in a particular column is computed with a specific kernel size and a specific stride during a depthwise convolution operation, reference will be made. Figure 13 , 15 Sections 17 and 19 describe unnecessary operations of the PE array.

[0387] Figure 19 Used to explain, based on examples of this disclosure, the weight data stored in the delay buffer over time.

[0388] First, as referenced Figure 13 and 17 The description assumes a specific kernel size of 3×3×m and a specific stride of 2.

[0389] For depthwise convolution operations, the weight data output from weight storage unit 210 (i.e., kernel data 1300) is input into the PE (PE_00, PE_01, ...) corresponding to the first column of each of the multiple PE rows, and the corresponding weight data is input into the delay buffer Z. -k The delay is set to a preset number of clock cycles. However, when performing a depthwise convolution operation with a kernel of size 3×3×m and a stride of 2, causing the kernel data 1300 to slide on the feature map data 1310 with a stride of 2, the delay buffer Z... -k Weighted data delayed by k clock cycles can be routed to the delay buffer Z corresponding to the second PE row (PE_10, PE_11, ...). -k Delay for 2k clock cycles, then input to the third PE line (PE_20, PE_21, ...), i.e. the next line.

[0390] In this case, such as Figure 15 and 19 As shown, since the second PE row (PE_10, PE_11, ...) does not perform any arithmetic operations, the second PE row (PE_10, PE_11, ...) can be disabled by transmitting an En signal for disabling (e.g., En1 = Low).

[0391] In this way, when the second PE line is disabled, the MAC operation can be performed only on each of the first and third PE lines. Thus, this disclosure can reduce the power consumption of the NPU by disabling unnecessary PEs that do not perform MAC operations.

[0392] The examples shown in the specification and accompanying drawings are for illustrative purposes only and to provide concrete examples to aid in understanding the subject matter of this disclosure, and are not intended to limit the scope of this disclosure. It will be apparent to those skilled in the art to which this disclosure pertains that other modifications may be made based on the technical spirit of this disclosure, in addition to the examples disclosed herein.

Claims

1. A neural processing unit (NPU), comprising: A mode selector, which is configured to select either a first mode or a second mode; and A processing element PE array, operating in one of the first and second modes, and comprising a plurality of processing elements arranged in rows and columns, the processing element array being configured to receive inputs of first input data and second input data, respectively. In the second mode, the first input data is input and transmitted along the column direction of the processing element array, while being delayed by a specific number of clock cycles. The second input data is then broadcast to the plurality of processing elements of the processing element array to which the first input data was input after the specific number of clock cycles. The processing element array is further configured to perform depthwise convolution operations in the second mode, and The specific number of clock cycles is determined based on the size of the weight kernel or the stride value of the convolution in the artificial neural network model.

2. The neural processing unit according to claim 1, wherein, The processing element array is also configured to perform pointwise convolution operations in the first mode.

3. The neural processing unit according to claim 1, wherein, In the first mode, the plurality of processing elements in each column of the processing element array are pipelined to transmit the first input data.

4. The neural processing unit according to claim 1, wherein, In the first mode, the second input data is unicast to each of the plurality of processing elements in each row of the processing element array.

5. The neural processing unit of claim 1, wherein the array of processing elements further comprises a delay buffer configured to output the first input data by delaying the first input data by the specified number of clock cycles.

6. The neural processing unit according to claim 1, wherein, The processing element array is also configured to determine the specific number of clock cycles based on the size of the weight kernel of the artificial neural network model.

7. The neural processing unit according to claim 1, wherein, In the second mode, the second input data is broadcast to the processing element column of the processing element array via a branched signal line, and the first input data delayed by the specific number of clock cycles is applied to the signal line of the processing element column via the branch.

8. The neural processing unit according to claim 1, wherein, In the second mode, the processing element rows of the processing element array consist of a first group of processing element rows and a second group of processing element rows, wherein the first group of processing element rows is configured to be enabled based on the weight kernel size of the artificial neural network model, and the second group of processing element rows excludes the first group of processing element rows and is configured to be disabled.

9. The neural processing unit of claim 1, wherein the array of processing elements further comprises: A first multiplexer is disposed in at least some of the rows of processing elements; A second multiplexer is disposed at the input portions of at least some of the processing element rows; as well as Delay buffers are provided within at least some of the rows of processing elements.

10. A neural processing unit (NPU) comprising: A mode selector, which is configured to select either a first mode or a second mode; and A processing element (PE) array comprising a plurality of processing elements arranged in rows and columns, the processing element array being configured to perform a first convolution operation in a first mode and a second convolution operation in a second mode. The processing element array is further configured to reuse weight data for the second convolution operation within the processing element array. The second convolution operation includes a depthwise convolution operation, and The processing element array is configured to delay the weight data by a specific number of clock cycles, the specific number of clock cycles being determined based on the size of the weight kernel or the stride value of the convolution in the artificial neural network model.

11. The neural processing unit of claim 10, wherein the first convolution operation comprises a standard or pointwise convolution operation.

12. The neural processing unit according to claim 10, wherein, The processing element array is configured to include a delay buffer, which is configured to reuse the weight data of the depthwise convolution operation.

13. The neural processing unit according to claim 10, wherein, In the first mode, the processing element array is also configured to receive: The weight data, which is input for the first convolution operation, is fed into the pipelined processing elements of each column of the processing element array, and The feature map data is input for the first convolution operation and is unicast to each processing element in the processing element row of the processing element array.

14. The neural processing unit of claim 10, further comprising a delay buffer disposed in at least some of the processing element rows of the processing element array, the delay buffer being configured to receive in the second mode: The input of the weight data for the second convolution operation, and The input of the weight data is delayed by the delay buffer and output from the delay buffer.

15. The neural processing unit according to claim 10, in, The processing element array also includes a delay buffer configured to delay the weight data by a predetermined number of clock cycles.

16. A neural processing unit (NPU) comprising: Weight storage unit, which is configured to load weight data for convolution operations; A feature map storage unit configured to load feature map data for the convolution operation; and A processing element (PE) array includes a plurality of processing elements and a plurality of delay units arranged to correspond to at least some of the processing elements in the processing element array, the plurality of delay units being configured to selectively delay the weight data via switching units corresponding to the plurality of delay units. The processing element array is further configured to perform depthwise convolution operations, and The processing element array is configured to delay the weight data by a specific number of clock cycles, the specific number of clock cycles being determined based on the size of the weight kernel or the stride value of the convolution in the artificial neural network model.

17. An array of processing elements, comprising: The first processing element is configured to receive weighted data; A delay unit is configured to receive the weight data, delay the weight data by a specific number of clock cycles, and transmit the weight data to a second processing element. as well as A broadcast signal line is configured to simultaneously provide feature map data to both the first processing element and the second processing element. The delay unit is configured to process depthwise convolutions by reusing the weight data, and The specific number of clock cycles is determined based on the size of the weight kernel or the stride value of the convolution in the artificial neural network model.

Images