Dynamic allocation of bus bandwidth for sending tensors to neural processing units

Abstract

According to one example of the present disclosure, a system may be provided. The system may include at least one processing core configured to process calculations of at least one neural network model including at least one tensor; at least one memory circuit configured to store at least one tensor; a bus circuit electrically coupled to the at least one processing core and the at least one memory circuit, configured to transmit at least one tensor based on a memory access operation instruction; and a controller configured to control priorities of memory access operations for each tensor of the at least one processing core.

Description

Cross-reference to related applications

[0001] This application claims priority to Korean Patent Application No. 10-2024-0114593, filed on August 26, 2024, with the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference. Technical Field

[0002] This disclosure relates to systems and methods for controlling processing cores. Background Technology

[0003] Humans possess intelligence such as recognition, classification, inference, prediction, and control / decision-making. Artificial intelligence (AI) is the artificial imitation of human intelligence.

[0004] The human brain is composed of a large number of nerve cells called neurons. Each neuron is connected to hundreds to thousands of other neurons through connections called synapses. To mimic human intelligence, the operation of biological neurons and the connections between neurons are modeled in neural network (NN) models. In other words, a neural network is a system of nodes connected together in a layered structure that mimics neurons. Summary of the Invention

[0005] The implementation involves dynamically prioritizing each tensor to at least one processing core via a bus circuit. The at least one processing core performs computational operations on at least one neural network model associated with the tensor. At least one memory circuit stores the tensors. The bus circuit is operatively coupled to both the at least one processing core and the at least one memory circuit. In response to receiving a request for a memory access operation, the bus circuit sends tensors from the at least one memory circuit to the at least one processing core. A controller is operatively coupled to the bus circuit. The controller determines the priority of the memory access operation for each tensor and controls the bus circuit to send each tensor to the at least one processing core according to the determined priority.

[0006] In one or more embodiments, the controller determines the priority of memory access operations for each tensor by comparing the memory cycle length and the tensor computation cycle length using bus circuitry.

[0007] In one or more implementations, the controller controls one or more quality of service parameters associated with the bus circuitry based on the determined priority of memory access operations for each tensor.

[0008] In one or more embodiments, the controller determines the priority of memory access operations for each tensor by at least comparing the computation cycle of a first tensor in the tensor at the processing core with the memory cycle of a subsequent second tensor in the tensor.

[0009] In one or more embodiments, in response to determining that a data starvation period is predicted to occur or has occurred in a first processing core, the controller increases the bus bandwidth allocated to the first processing core by reducing the bus bandwidth allocated to the second processing core that is allocated to the at least one processing core.

[0010] In one or more embodiments, in response to determining that a data starvation period is predicted to occur or has already occurred in a first processing core, the controller increases the priority of sending tensors of the first processing core to at least one processing core, thereby increasing the bus bandwidth allocated to the first processing core.

[0011] In one or more embodiments, in response to determining that at least one processing core is in a computationally limited state, the controller controls the bus circuitry to relinquish at least a portion of the bandwidth allocated to the bus circuitry of the at least one processing core.

[0012] In one or more embodiments, in response to receiving a data hunger signal, the controller determines the priority of sending tensors to at least one processing core via a bus circuit.

[0013] In one or more embodiments, the system further includes a counter for determining a data starvation state. Whether at least one processing core is in a data starvation state is determined by comparing the counter value with a preset counter threshold.

[0014] In one or more embodiments, each of at least one processing core includes a plurality of processing elements (PEs). The plurality of PEs includes at least one of a multiplication and accumulation (MAC) operator, an addition tree, or an arithmetic logic unit (ALU) operator.

[0015] The implementation also involves allocating bandwidth for transmitting tensors. It is determined whether data starvation has occurred or is predicted to occur in at least one processing core configured to perform computational operations on at least one neural network model associated with the tensor. In response to determining whether data starvation has occurred or is predicted to occur in at least one processing core, a priority is determined for memory access operations of each tensor via a bus circuit. Bandwidth for transmitting each tensor to the at least one processing core is allocated according to the determined priority. Each tensor is transmitted to the at least one processing core via the bus circuit according to the bandwidth allocated to the bus circuit for transmitting each tensor to the at least one processing core.

[0016] In one or more embodiments, determining whether data starvation has occurred or is predicted to occur in at least one processing core includes comparing a computation cycle of a first tensor to be processed by at least one processing core with a memory cycle for sending a second tensor to at least one processing core via a bus circuit, the second tensor being subsequently processed by at least one processing core.

[0017] In one or more embodiments, bandwidth allocation includes controlling one or more quality of service parameters associated with the bus circuitry based on a determined priority of memory access operations for each tensor.

[0018] In one or more embodiments, determining whether data starvation has occurred or may occur includes determining the memory cycle of a first tensor in the tensor and the computation cycle of a second tensor following the first tensor in the tensor.

[0019] In one or more embodiments, bandwidth allocation includes reallocating at least a portion of the bus bandwidth allocated to a first processing core that is otherwise allocated to at least one processing core to a second processing core.

[0020] In one or more embodiments, determining priority includes prioritizing the transmission of at least one tensor based on a data hunger signal indicating that a processing element in at least one processing core is idle.

[0021] In one or more embodiments, determining whether data starvation has occurred or may occur includes comparing the counter value of a counter with a preset counter threshold.

[0022] In one or more embodiments, allocating bandwidth for the bus circuitry includes increasing the bus bandwidth to transmit tensors that are memory-bound to at least one processing core.

[0023] In one or more embodiments, the priority of memory operations for each tensor is one of a first priority, a second priority which is higher than the first priority, or a third priority which is higher than the second priority. Attached Figure Description

[0024] Figure 1A and Figure 1B This diagram illustrates the bottleneck that occurs when performing read and write operations via a neural processing unit in a conventional control system.

[0025] Figure 2 This is a schematic diagram illustrating a system for controlling a processing core according to an example of this disclosure.

[0026] Figure 3 This is a schematic diagram illustrating a processing element according to an example of this disclosure.

[0027] Figure 4 This is a schematic diagram illustrating an example neural network.

[0028] Figure 5 This is a table showing the energy consumption per unit operation of a neural processing unit according to an example of this disclosure.

[0029] Figure 6A and Figure 6B It is a timing diagram illustrating exemplary operations for reducing latency in a tensor according to a first example of this disclosure, wherein bus congestion occurs in a system used to control the processing core.

[0030] Figure 7 It is a flowchart illustrating a method for controlling the processing core according to a first example of this disclosure.

[0031] Figure 8 This is a diagram illustrating a method for determining the priority of a transmitted tensor according to a first example of this disclosure.

[0032] Figure 9 This is a diagram illustrating how, according to a first example of this disclosure, data processing speed can be improved by giving priorities to reduce data starvation periods in processing cores.

[0033] Figure 10 It is a flowchart illustrating an example of a method for determining the core priority of control processing according to the second example of this disclosure.

[0034] Figure 11 It is a timing diagram that illustrates an example of a data hunger signal generated during the runtime of the processing core, according to the second example of this disclosure.

[0035] Figure 12 It is a timing diagram illustrating a method for determining priorities to reduce delays identified based on counter count values, according to a third example of this disclosure.

[0036] Figure 13 It is a flowchart illustrating a method for determining the priority of a transmitted tensor according to the fourth example of this disclosure.

[0037] Figure 14 This is a diagram illustrating how, according to the fourth example of this disclosure, data processing speed can be improved by giving priorities to reduce data starvation periods. Detailed Implementation

[0038] The specific structures or step-by-step descriptions of examples based on the concepts disclosed in this specification or application are merely illustrative for the purpose of explaining examples based on the concepts disclosed.

[0039] Examples based on the concepts of this disclosure may be embodied in various forms. Examples based on the concepts of this disclosure should not be construed as limited to the examples described in this specification or application.

[0040] Various modifications can be applied to the examples based on the concepts of this disclosure. This disclosure can take many forms. Therefore, specific examples are shown in the accompanying drawings and described in detail in this disclosure. However, this is not intended to limit the examples based on the concepts of this disclosure to a particular form of disclosure. Therefore, it should be understood that all modifications, equivalents, or substitutions included within the spirit and scope of this disclosure are included in this disclosure.

[0041] Terms such as first and / or second may be used to describe various components. However, this disclosure should not be limited to the terms mentioned above. These terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the claims according to the concepts of this disclosure, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.

[0042] When an element is said to be "connected" or "in contact" with another element, it is understood that the element may be directly connected or in contact with that other element, but there may also be other elements between them. On the other hand, when it is said that an element is "directly connected" or "directly in contact" with another element, it should be understood that there are no other elements between them. Other expressions describing the relationship between elements, such as "between" and "immediately adjacent" or "adjacent" and "directly adjacent," should be interpreted similarly.

[0043] In this disclosure, expressions such as “A or B”, “at least one of A and / or B” or “one or more of A and / or B” can include all possible combinations thereof. For example, “A or B”, “at least one of A and B” or “at least one of A or B” can mean: (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.

[0044] As used herein, expressions such as “first,” “second,” or “first or second” may modify various elements regardless of their order and / or importance. These expressions are used only to distinguish one element from others and do not limit the elements. For example, a first user device and a second user device may refer to different user equipment regardless of their order or importance. For example, a first element may be named a second element without departing from the scope of the claims described in this disclosure, and similarly, a second element may be renamed a first element.

[0045] The terminology used in this disclosure is for the purpose of describing specific examples only and is not intended to limit the scope of other examples. Singular expressions may include plural expressions unless the context clearly specifies otherwise. The terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by one of ordinary skill in the art as described herein.

[0046] In this disclosure, terms defined in general dictionaries may be interpreted as having the same or similar meaning as in the relevant technical context. Unless explicitly defined herein, they should not be interpreted in an ideal or overly formal sense. In some cases, even terms defined in this disclosure should not be interpreted as excluding examples of this disclosure.

[0047] The terminology used herein is for the purpose of describing specific examples only and is not intended to limit this disclosure. Unless the context clearly specifies otherwise, singular expressions include plural expressions. In this specification, terms such as “comprising” or “having” are intended to indicate the presence of the described features, quantities, steps, operations, components, parts, or combinations thereof. Therefore, it should be understood that the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof is not excluded.

[0048] Unless otherwise defined, all terms used herein, including technical or scientific terms, shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Terms such as those defined in common dictionaries shall be interpreted as having a meaning consistent with their meaning in the relevant technical context. Unless expressly defined in this disclosure, they shall not be interpreted in an ideal or overly formal sense.

[0049] Each feature of the various examples of this disclosure can be combined, either partially or entirely, or with each other. Those skilled in the art will fully understand that the various examples of this disclosure are technically capable of various interlocking and driving mechanisms. Each example of this disclosure can be implemented independently of each other, or can be implemented together in an associated relationship.

[0050] In describing examples, descriptions of technical content that is well-known in the technical field to which this disclosure pertains and is not directly related to this disclosure may be omitted. This is to convey the main points of this disclosure more clearly without obscuring them by omitting unnecessary descriptions.

[0051] <Definition of Terminology>

[0052] For ease of understanding this disclosure, the following is a brief overview of the terminology used herein.

[0053] NPU: an abbreviation for Neural Processing Unit, which can refer to a processor that is independent of the Central Processing Unit (CPU) and is specifically designed for computing neural network models.

[0054] SoC: An abbreviation for System-on-a-Chip, which refers to a semiconductor chip that integrates at least one processor and various circuit elements of an electronic system onto a single integrated circuit (IC). An SoC can integrate digital circuits, analog circuits, mixed-signal circuits, and radio frequency processing circuits on a single semiconductor chip. An SoC may include at least one processor. For example, the at least one processor included in an SoC may be at least one of a Central Processing Unit (CPU), a Digital Signal Processor (DSP), an Image Signal Processor (ISP), a Graphics Processing Unit (GPU), and a Neural Processing Unit (NPU). An SoC may include at least one memory. For example, the memory included in an SoC may be at least one of Random Access Memory (RAM), Read-Only Memory (ROM), and cache memory. An SoC may include a high-speed data bus, such as AXI, AHB, APB, etc., for efficient communication between multiple IP blocks included in the SoC. An SoC may include at least one interface, such as PCIe, USB, I2C, SPI, UART, GPIO, for connecting to external devices and sensors. An SoC may include an on-chip power management unit that regulates the voltage and power distribution on the semiconductor chip. A System-on-a-Chip (SoC) may include communication interfaces for data transmission that integrate wired and wireless communication protocols such as Ethernet, Wi-Fi, Bluetooth, and cellular connectivity. SoCs can be manufactured using a variety of packaging technologies.

[0055] NN: an abbreviation for Neural Network, a network of nodes connected in a layered structure, mimicking the way neurons in the human brain connect through synapses to imitate human intelligence.

[0056] Neural network model information includes: network structure, number of layers, connection relationships within each layer, parameters for each layer, computational processing methods, activation functions, data type of each layer's parameters (e.g., floating-point or integer), and bit width of each parameter. Each layer's parameters can be represented by a tensor of a certain size. During the compilation process, at least one layer can be divided into tensors as blocks based on the computational circuit architecture and internal memory size. Based on the parameter sizes of each tensor (e.g., input parameter sizes and weight parameter sizes) and the required computational algorithms (e.g., matrix multiplication, activation functions, and softmax functions), the clock cycles for the computational circuitry processing the tensors and the clock cycles for data transfer to memory can be calculated.

[0057] DNN: an abbreviation for Deep Neural Network, which can refer to increasing the number of hidden layers in a neural network to achieve higher levels of artificial intelligence.

[0058] CNN: an abbreviation for Convolutional Neural Network, a type of neural network that functions similarly to the human brain's visual cortex when processing images. Convolutional neural networks are known to be very well-suited for image processing and are renowned for their ability to extract features from input data and identify patterns within those features.

[0059] Transformer: A transformer neural network is a DNN based on attention techniques. It utilizes many matrix multiplication operations. The transformer can receive input values ​​and parameters, such as a query (Q), a keyword (K), and a value (V), to obtain an output value, i.e., attention (Q, K, V). Based on the output value (i.e., attention (Q, K, V)), the transformer can handle various inference operations. Transformers have been actively used in language generation models.

[0060] Kernel: Refers to the weights of an N x M convolution matrix. Each layer of a neural network model has multiple kernels; the number of kernels can also be referred to as the number of channels, the number of filters, etc.

[0061] Tensors: Tensors are multidimensional matrix parameters processed by neural network models. Tensors can refer to various parameters of a neural network model, such as weights, feature maps, kernels, and attention parameters. A tensor can refer to the input parameters input to a neural processing unit and the output parameters computed by the neural processing unit. A tensor can be the parameters of a patch computed by a neural processing unit at a time. A neural network model can contain multiple layers, and each layer can be configured to include at least one tensor. For example, the input parameters of the first layer of a neural network model can be called the first tensor, the weight parameters of the first layer can be called the second tensor, and the output parameters of the first layer can be called the third tensor. For example, the input parameters of the first layer of a first neural network model can be called the first tensor, while the input parameters of the first layer of a second neural network model can be called the second tensor.

[0062] Neural network (NN) models are classified into "single-layer neural networks" and "multi-layer neural networks" based on the number of layers. A typical multi-layer neural network includes an input layer, hidden layers, and an output layer. (1) The input layer is the layer that receives external data, and the number of neurons in the input layer is the same as the number of input variables. (2) The hidden layer is located between the input layer and the output layer. It receives signals from the input layer, extracts features, and passes them to the output layer. (3) The output layer receives signals from the hidden layer and outputs them. The input signals between neurons are multiplied by their respective weights (the values ​​of the weights are between 0 and 1), and then summed. If the sum is greater than the neuron's threshold, the neuron is activated and outputs the signal through an activation function.

[0063] On the other hand, neural networks with more hidden layers to achieve higher levels of artificial intelligence are called deep neural networks (DNNs). There are many types of DNNs, but convolutional neural networks (CNNs) are known for their ability to extract features from input data and identify patterns in those features.

[0064] Convolutional neural networks (CNNs) are neural networks that function similarly to the visual cortex of the human brain, which processes images. CNNs are known to be suitable for image classification, object detection, and other applications.

[0065] Convolutional Neural Networks (CNNs) consist of iterative processes of convolution and pooling channels. Convolution operations account for the majority of computation time in a CNN. A CNN extracts features from the image in each channel using kernels in matrix form and uses pooling to provide steady-state conditions such as translation or distortion for object recognition. In each channel, a feature map is obtained by convolving the input data and the kernel, and an activation function such as ReLU (Rectified Linear Unit) is applied to generate the activation map for that channel. Pooling can then be applied. The neural network that actually classifies the patterns is located at the end of the feature extraction neural network, called a fully connected layer. In the computational processing of a CNN, most operations are performed via convolution or matrix multiplication. The necessary kernels are frequently fetched from memory. A significant portion of the computation in a CNN is the time spent fetching the kernel corresponding to each channel from memory. However, the examples in this disclosure are not limited to CNNs and can be applied to transformer neural networks, etc.

[0066] Memory (also referred to herein as "memory circuitry") can be categorized into memory sections (e.g., main memory or off-chip memory), internal memory, on-chip memory, etc. Each memory may include multiple memory cells, each with a unique memory address. In particular, whenever a neural processing unit calls weight parameters stored in memory or calls other parameters, a delay of several clock cycles may occur before accessing the memory cell corresponding to the memory address.

[0067] The neural processing unit can perform memory access operations to write data to or read data from memory, as well as computational operations to compute neural network models based on the data.

[0068] Recently, systems have been developed in which multiple neural processing units or multiple processing cores contained in a neural processing unit are configured to simultaneously send data to or receive data from memory.

[0069] In such systems, multiple neural processing units or multiple processing cores may simultaneously attempt to access memory via the bus. In this situation, the bus handling data communication may prioritize the first memory access command to arrive. Therefore, memory access contention on the bus may occur. Furthermore, the next memory access operation may be delayed until the first memory access operation completes, potentially leading to data starvation.

[0070] This contention can further increase the processing time and power required to read the necessary parameters from memory and execute AI operations in the neural processing unit. Furthermore, if a memory access operation is not completed, a data starvation period occurs, during which the neural processing unit cannot perform computational operations because the required data is not available.

[0071] On the other hand, the processing time of memory access operations used to perform computations by the neural processing unit is related to the data size. That is, the larger the data size, the greater the amount of data transferred, which can increase data transfer time. The time it takes for the neural processing unit to process AI operations using data provided by memory access operations is related to the complexity of the computational algorithm. That is, as AI computational algorithms become more complex, the computational load increases, which can increase data computation time. Therefore, the processing time of memory access operations and the processing time of AI algorithm operations may differ. For example, the time required to complete a memory access operation within a specific interval may be shorter or longer than the time required to complete a computational operation.

[0072] When the completion time of a memory access operation is longer than the completion time of a computation operation, it can be called a memory-limited state. A memory-limited state may occur when computation is limited by memory access speed rather than the computational capacity of the neural processing unit. In this state, data starvation may occur, during which the neural processing unit is inactive as it waits to fetch data from memory or write data to memory.

[0073] Conversely, when the completion time of a computational operation is longer than that of a memory access operation, it can be termed a computationally constrained state. A computationally constrained state may occur when the processing power of the neural processing unit is a limiting factor, for example, when AI computation time is longer than memory access time. In this state, the neural processing unit may experience data starvation due to the inefficiency of scheduling AI computations or allocating memory bandwidth. Therefore, a data starvation period occurs, during which the computational operations of the neural processing unit cannot be executed in either a memory-constrained or computationally constrained state.

[0074] According to one aspect of this disclosure, the Quality of Service (QoS) priority of read and write operations for accessing memory of each neural processing unit or processing core is modified to improve the efficiency of read or write operations of the Direct Memory Access Controller (DMA). Furthermore, the QoS priority of read and write operations for accessing memory of each neural processing unit or processing core can be set based on analysis of memory-limited and computation-limited states to improve the efficiency of DMA read and write operations. When a data starvation period is predicted, the bus bandwidth allocated to the neural processing unit or processing core can be reallocated on a tensor basis, since the time required to complete a memory access operation of a tensor is shorter or longer than the time required to complete a computation operation of another tensor. This allows the computational circuitry to operate without data starvation periods, improving data processing performance and reducing power consumption.

[0075] An example of this disclosure will be described below with reference to the accompanying drawings. Figure 1A and Figure 1B This diagram illustrates the bottleneck that occurs when performing read and write operations on each of multiple neural processing units in a conventional control system.

[0076] Traditional control systems control the operation of neural processing units to access data from memory (i.e., memory access operation); and / or control the operation of neural processing units to compute data provided to them (i.e., computation operation).

[0077] first, Figure 1A The diagram illustrates a scenario where the memory is connected to a bus, and the bus is implemented as a single-bus architecture. In this case, the memory cannot process read and write operations simultaneously; it can only perform either a read or write operation on one of the neural processing units at a time.

[0078] on the other hand, Figure 1B The diagram illustrates a scenario where the memory is connected to a bus, and the bus is implemented as two independent buses. In this case, although the memory can handle read and write operations separately via the bus, a bottleneck arises because read and write operations from multiple neural processing units must be processed sequentially.

[0079] Furthermore, if there is a difference in the request time between read and write operations, the corresponding neural processing unit will inevitably experience a data starvation period. That is to say, according to... Figure 1A and Figure 1B When multiple neural processing units (RNUs) or processing cores simultaneously attempt to access memory, traditional control systems process these attempts on a first-come, first-served basis. Consequently, multiple RNUs compete for memory access. This competition increases the time required for each RNU to complete a memory access operation. Furthermore, because memory access operations are not completed in a timely manner, data starvation occurs within the RNU, preventing it from initiating computational operations.

[0080] Even if this isn't a competition between multiple neural processing units or multiple processing cores, at any given point in time, the time taken to complete a memory access operation and the time taken to complete a computation operation are not always equal. Therefore, in certain intervals, the memory access operation may take longer than the computation operation (i.e., a memory-constrained state), which can also lead to data starvation periods during which the computation circuitry is not running.

[0081] In such Figure 1BIn the scenario depicted, the bus, lacking a specific scheduling criterion, generates an ordered queue on a first-come, first-served basis, prioritizing read and write operations for each tensor requested by the corresponding neural processing unit that requests them first. Consequently, data starvation frequently occurs across multiple neural processing units. In other words, based on the ordered queue of the bus used to operate neural network models according to conventional control systems, bus bottlenecks can frequently occur due to inefficient data starvation and the resulting degradation of effective bus bandwidth.

[0082] To address this issue, a system according to an example of this disclosure can determine the priority of tensors processed by a specific neural processing unit and transmitted via a bus associated with multiple neural processing units by based on 1) the memory access operation time of the bus and 2) the computation operation time of each tensor at the neural processing unit, thereby reducing the data starvation period of the neural processing unit. In other words, when the transmission time of multiple tensors corresponding to multiple neural processing units requesting bus access overlaps, the system according to an example of this disclosure can determine the priority of competing tensors, thereby reducing the data starvation time of the computation circuitry of a specific neural processing unit in the system. In this way, data starvation of the neural processing unit caused by memory bandwidth limitations and memory latency during read and write operations can be mitigated or eliminated.

[0083] In the following, the neural processing unit according to the examples of this disclosure may also be referred to as a processing core. For example, a neural processing unit may refer to a semiconductor chip formed on a substrate including at least one processing core. In other words, multiple processing cores may be configured as part of a neural processing unit. As described herein, a processing core may refer to computational circuitry configured to process operations of a neural network model. In various examples of this disclosure, neural processing units and processing cores may be substantially equivalent to each other.

[0084] In some examples, the first controller 1100 and the second controller 100 may be integrated to form a controller. The controller may be referred to as a control circuit.

[0085] Figure 2 This is a schematic diagram illustrating a system for controlling a processing core according to an example of this disclosure. Figure 2 A neural processing unit comprising multiple processing cores is illustrated, along with multiple peripheral devices for computing the neural processing unit. Therefore, the neural processing unit and the multiple peripheral devices can be referred to as a system. At least some components of this system may comprise a system-on-a-chip (SoC).

[0086] Reference Figure 2The neural processing unit (NPU) 1000 of system 1000 may include multiple processing cores 1000-1, ..., 1000-n, and may be configured to communicate with central processing unit (CPU) 2000, memory 3000, image sensor 4000, and / or decoder 5000 to execute various neural network inference functions. Furthermore, each processing core of the neural processing unit 1000 may be configured to be controlled via a corresponding first controller 1100.

[0087] Each of the neural processing unit 1000, central processing unit 2000, memory 3000, image sensor 4000, decoder 5000 and / or bus 6000 according to an example of this disclosure may be formed as a separate semiconductor circuit, or at least a portion thereof may be integrated within a package, and this disclosure is not limited thereto.

[0088] According to various examples, the neural processing unit 1000 of system 1000 can be patterned on the same semiconductor die as the central processing unit 2000.

[0089] According to various examples, the neural processing unit 1000, central processing unit 2000, and memory 3000 of system 10000 can be patterned on the same semiconductor die.

[0090] According to various examples, the neural processing unit 1000 of system 1000 may include a semiconductor die connected to the central processing unit 2000 via chiplet technology. When chiplet technology is applied, an inserter may be further included.

[0091] According to various examples, a system 10000, including a neural processing unit 1000, a central processing unit 2000, and a memory 3000, can be composed of semiconductor dies connected via chiplet technology.

[0092] Each of the aforementioned components is characterized by its operational function, and each component can be implemented in a circuit board, silicon substrate, resistor, transistor, etc. Therefore, each component can be a semiconductor circuit with many transistors connected to it, some of which may be difficult to identify and distinguish with the naked eye, and may be identifiable only by their operation. Figure 2 Each component can be called a corresponding circuit unit.

[0093] Each of the central processing unit 2000, memory 3000, image sensor 4000, and decoder 5000 can communicate via bus 6000 to send and receive data to and from each of the processing cores 1000-1, ..., 1000-n. According to one example of this disclosure, bus 6000 can be an Advanced Scalable Interface (AXI) bus. However, each processing core 1000-1, ..., 1000-n can be configured to be directly coupled to at least one of the aforementioned components, and is not limited thereto.

[0094] The neural processing unit 1000 can be defined as a processor specifically designed for operations on the neural network model. In particular, the neural processing unit 1000 can be specifically designed for matrix multiplication or convolution operations, which account for the majority of computations in the neural network model.

[0095] Neural network models are based on neural networks, which are artificial neural networks that receive multiple inputs or stimuli, multiply them by their respective weights, sum them together, and then transform and transmit the resulting biases through an activation function. Neural network models trained in this way can be used to output inference results from input data. These inference results can be used for object detection, image classification, event detection, pose estimation, token generation, natural language generation, image generation, and more.

[0096] The neural processing unit 1000 can be a semiconductor implemented as an electrical / electronic circuit. An electrical / electronic circuit system may refer to a system comprising multiple electronic components (e.g., transistors, capacitors).

[0097] In the case of a neural network model based on a converter and / or CNN, the neural processing unit 1000 can selectively process matrix multiplication operations, convolution operations, etc., according to the neural network architecture.

[0098] For example, in each layer of a convolutional neural network (CNN), the input feature map corresponding to the input data and the kernel corresponding to the weights can be matrices comprising multiple channels. Convolution operations can be performed on the input feature maps and kernels, and convolution operations can be generated on each channel and the pooled output feature map. Activation functions can be applied to the output feature maps to generate activation maps for the corresponding channels. Pooling can then be applied to the activation maps. In this paper, activation maps can be collectively referred to as output feature maps, and each feature map and weight can be referred to as a tensor.

[0099] However, this disclosure is not limited to this; the output feature map can be subjected to matrix multiplication, convolution, and other operations.

[0100] Furthermore, the output feature map according to the example of this disclosure will be fully interpreted. For example, the output feature map may be the result of matrix multiplication or convolution operations. Therefore, the plurality of processing elements (PEs) included in the processing element array 400 may be modified to further include processing circuit units for additional algorithms.

[0101] The neural processing unit 1000 can be configured to include multiple processing elements (PEs) for processing convolutions and matrix multiplications required for neural network operations.

[0102] The neural processing unit 1000 can be configured to include corresponding computing circuits optimized for matrix multiplication, convolution, activation function, pooling, stride, batch normalization, skip connection, concatenation, quantization, pruning, padding, and attention operations required for neural network operations.

[0103] For example, the neural processing unit 1000 may be configured to include a special function unit (SFU) 500 for processing at least one of the above algorithms: activation function operation, pooling operation, stride operation, batch normalization operation, jump connection operation, splicing operation, quantization operation, pruning operation, and padding operation.

[0104] Multiple tensors transmitted to the neural processing unit 1000 via bus 6000 can be configured to be controlled by a first controller 1100. Specifically, the first controller 1100 can be configured to determine which tensors cause data starvation based on the cycle time of memory access operations and computation operations (i.e., the number of clock cycles spent processing the tensor) before each processing core 1000-1, ..., 1000-n directly accesses memory 3000 to perform reads and / or writes, and to determine a priority order between the identified tensors and subsequent tensors. Bus 6000 can be configured to process each tensor sequentially based on the determined priority. Therefore, the first controller 1100 can be configured to ensure that higher-priority tensors yield bus 6000 bandwidth to lower-priority tensors, thereby preventing data starvation. Each tensor has its own data size, and the first controller 1100 can calculate the number of clock cycles based on the size of each tensor transmitted on bus 6000. Furthermore, the compiler can pre-calculate the number of clock cycles required to process the corresponding tensors on the processing core when compiling the corresponding neural network model. Therefore, the first controller 1100 can obtain cycle information for each tensor computed at compile time. As will be further described, the clock cycle for memory access operations of each tensor can be referred to as a memory cycle, and the clock cycle for computation operations can be referred to as a computation cycle. Moreover, when the pre-calculated computation clock cycles are processed by the neural processing unit, their number is unlikely to change. This is the case when the neural processing unit is a dedicated AI accelerator configured to process neural network models. Conversely, the number of pre-calculated memory cycles can be a minimum number and can have dynamic characteristics that may increase beyond the minimum number due to various reasons such as bandwidth contention on the bus, low priority in an ordered queue, etc. Therefore, the pre-acquired memory cycles can refer to the minimum number of memory cycles.

[0105] Specifically, the neural processing unit 1000 may include a controller 100, a direct memory access (DMA) 200, internal memory 300, a processing element array 400, and a special function unit 500. However, in describing the neural processing unit 1000, the following description will be limited to one processing core 1000-1. However, this is merely for ease of description and can be applied substantially equivalently to any processing core included in the neural processing unit 1000.

[0106] The components of the processing core 1000-1 are distinguished by their operational functions, and each component can be formed using at least one of a substrate, a resistive element, and a transistor. Therefore, each component can be a semiconductor circuit with many transistors connected thereon, some of which may be difficult to identify and distinguish with the naked eye and may only be identifiable by their operation. Thus, each functional unit of the processing core 1000-1 can be called a circuit unit.

[0107] The second controller 100 can be configured to control operations associated with each computational neural network model in the DMA 200, internal memory 300, processing element array 400, and special function unit 500. The second controller 100 can be directly or indirectly coupled to each of the DMA 200, internal memory 300, processing element array 400, and special function unit 500 to communicate with each other. For example, the second controller 100 can adjust the cache size of each tensor stored in the internal memory 300 for each computational step based on the capacity of the internal memory 300. The second controller 100 can be configured to control the processing core 1000-1 based on the machine code (e.g., binary code) of the compiled neural network model.

[0108] For example, a compiler can generate machine code that determines the read / write order of neural network model data, as well as information about the processing order of neural network layers, the order of convolution multiplication operations, the order of matrix multiplication operations, and the order of DMA data read / write operations. These orders are determined based on the hardware characteristics of the processing core 1000-1 (e.g., the number of processing elements, memory capacity, function circuit units within special function units, and the presence of post-processing units). Therefore, the second controller 100 can control the processing core 1000-1 based on the machine code. Machine code can be referred to as binary code, executable code, etc.

[0109] The second controller 100 can obtain scheduling information based on the directed acyclic graph (DAG) of the neural network model compiled by the compiler to schedule the order of operations of the neural network patterns to be executed by the processing core 1000-1. A computational step can be processed in a tensor unit. Here, the compiler can determine the operation scheduling that can accelerate the operation of the neural network model by determining the number of processing elements (PEs) of the processing core 1000-1, the size of the internal memory 300, the parameter size of each layer of the neural network model, etc. According to the computational schedule, the second controller 100 can be configured to control the number of processing elements (PEs) required for each computational step and control the read and write operations of the required parameters in the internal memory 300 for each computational step. The compiler can efficiently schedule computational steps based on its understanding of the hardware architecture and capacity of the processing core 1000-1. The compiler can determine the data order required for the computation of the neural network model based on the order of operations of the layers, convolutions, and / or matrix multiplications of the neural network and can generate compiled machine code. The parameters input to the neural processing unit in a computational step can be called input tensors, while the parameters output from the neural processing unit in a computational step can be called output tensors.

[0110] In some examples, processing core 1000-1 may be configured to include an embedded compiler. Based on the above configuration, processing core 1000-1 may be configured to generate machine code upon receiving one or more file inputs in various AI software framework formats. For example, AI software frameworks may include TensorFlow, PyTorch, Keras, XGBoost, mxnet, DARKNET, ONNX, etc. However, the examples disclosed herein are not limited to any specific AI software framework.

[0111] DMA 200 can be configured to access memory 3000 via bus 6000 and request reads and / or writes to memory 3000. Processing core 1000-1 can receive various data associated with the neural network model from memory 3000 via DMA 200. Memory 3000 can be included in a system-on-chip (SoC) or configured as a separate memory device.

[0112] Internal memory 300 can be memory located in the on-chip area of ​​processing core 1000-1, and can be memory used to cache or store data processed in the on-chip area. That is to say, internal memory 300 can also be called cache memory.

[0113] Furthermore, internal memory 300 can read from memory 3000 and store at least some data required for computing the neural network model. This at least some data may be referred to as tensors. Internal memory 300 can be configured to store all or part of the neural network model based on the storage capacity setting for each parameter and the data size of each layer of the neural network model. Parameters of representative data processed for the operation of the neural network model may include at least one of attention parameters, KV cache parameters, activation map parameters, input feature map parameters, output feature map parameters, and weight parameters.

[0114] Specifically, internal memory 300 can read and store parameters corresponding to the input data from memory 3000. Furthermore, internal memory 300 can read and store parameters corresponding to the output data from processing element array 400. As further described below, the parameters included in the neural network model may include input values ​​and weights. Input or output values ​​read or written by internal memory 300 may include at least one of activation parameters, feature map parameters, KV cache parameters, attention parameters, etc.

[0115] Internal memory 300 may include at least one of the following types of memory: register file, ROM, SRAM, DRAM, resistive RAM, magnetoresistive RAM, phase-change RAM, ferroelectric RAM, flash memory, HBM, etc. According to one example of this disclosure, internal memory 300 may be SRAM and configured such that SRAM is advantageous in terms of computational processing speed. Furthermore, internal memory 300 may be organized into at least one storage cell (e.g., a storage repository, etc.). Internal memory 300 may include homogeneous memory or heterogeneous memory.

[0116] Furthermore, the data stored in the storage units of internal memory 300 (e.g., parameters of the neural network model) is not fixed to one of attention, KV cache, activation map, input feature map, weights, and output feature map, but can be changed to another of attention, KV cache, activation map, input feature map, weights, and input feature map as needed. In other words, by changing the memory allocation of internal memory 300, the utilization efficiency of internal memory 300 can be improved; that is, the size of each tensor stored in internal memory 300 can vary for each computation step.

[0117] The processing element array 400 can be configured to include multiple processing elements that perform multiplication and accumulation (MAC) operations.

[0118] Each element of the processing element array 400 can be configured to perform an operation by receiving input (e.g., an input feature map corresponding to the input data and / or a kernel corresponding to the weights of the neural network).

[0119] The processing element can be configured to perform functions such as addition, multiplication, and accumulation required to process the neural network model. To this end, each processing element may include at least one of the following: MAC (multiplication and accumulation) operator, addition tree, and ALU (arithmetic logic unit) operator.

[0120] For example, a processing element can receive an input feature map and weights, perform convolution calculations, and output an output feature map. Furthermore, the array of processing elements 400 or the processing elements themselves can be referred to as an artificial intelligence (AI) computing unit.

[0121] In another example, a processing element (PE) can use the input feature map and weights as input to perform a generalized matrix multiplication (GEMM) operation, or matrix multiplication, to output an output feature map. More specifically, the processing element (PE) can multiply the input feature map in matrix form with a weight matrix, and then add a bias to the matrix to output an output feature map in matrix form. In particular, matrix multiplication can be performed at high speed through parallel processing in a neural processing unit, enabling efficient handling of matrix multiplication operations.

[0122] As another example, the processing element PE may include a circuit system designed to receive only integer type parameters as input. In this case, the input parameters of the processing element PE can be converted into integers of a specific width and stored in internal memory 300. According to the above configuration, power consumption can be effectively reduced compared to processors that support floating-point parameters, and it can be efficiently implemented on the device.

[0123] Special function unit 500 can process multiple activation functions to impart nonlinearity to the output feature map.

[0124] The activation function processed by the special function unit 500 may include, but is not limited to, the SiLU function, the Softmax function, the sigmoid function, the hyperbolic tangent (tanh) function, the ReLU function, the Leaky-ReLU function, the Maxout function, or the ELU function (which produces a non-linear output value relative to the input value).

[0125] On the other hand, it may be technically difficult to support all activation functions in the processing core 1000-1. Therefore, the processing core 1000-1 can approximate various activation functions using piecewise linear function approximation algorithms and piecewise linear function processing circuitry. These activation functions can be selectively applied after MAC operations. The calculated values ​​after applying activation functions can be called an activation graph.

[0126] In addition, the special function unit 500 can be configured to include floating-point multiplier circuitry to perform decimal point operations.

[0127] As another example, the special function unit 500 may be configured to communicate with the processing element PE and may include a circuit system designed to receive integer parameters output from the processing element PE. In this case, the special function unit 500 may also be configured to include dequantization circuitry configured to convert integer parameters to floating-point parameters. Furthermore, the special function unit 500 may be configured to process activation function operations using floating-point parameters. Additionally, the special function unit 500 may be configured to include quantization circuitry configured to convert floating-point parameters to integer parameters at the end of the activation function operation. According to the above configuration, the special function unit 500 may be configured to process floating-point operations by dequantizing integer parameters when floating-point operations are required, and to requantize the result. In other words, a neural processing unit according to an example of this disclosure may include a processing element circuitry configured to process integer parameters and a special function circuitry pipelined thereto, and the special function circuitry may include quantization and inverse quantization circuitry, and may be configured to process activation function operations with floating-point parameters. According to the above configuration, the special function unit 500 effectively communicates with the processing element PE, which only supports integer parameters, and has the effect of directly converting and processing parameter types even without external circuitry. That is, the neural processing unit 1000 is configured to receive integer-format tensors via bus 6000 according to a request from DMA 200, and store the integer-format tensors in internal memory 300. The processing element PE can be configured to compute the integer-format tensors. The special function unit 500 can be configured to receive the integer-format tensors computed by the processing element PE as input, convert them to floating-point tensors, process the result of at least one special function, convert them back to integer-format tensors, and store them in internal memory 300. The neural processing unit 1000 can send the results stored in internal memory 300 to memory 3000 via bus 6000 according to a request from DMA 200.

[0128] Now refer to Figure 3 The detailed configuration of the processing element is described below. Figure 3 This is a schematic diagram illustrating a processing element according to an example of the present disclosure. The processing element PE 410 may be configured to include a multiplier 411, an adder 412, an accumulator 413, and a bit quantization unit 414. However, the examples according to the present disclosure are not limited to this architecture, and the array of processing elements may be modified to take into account the computational characteristics of the target neural network model.

[0129] Multiplier 411 multiplies the input (N)-bit data with (M)-bit data. The result of the multiplier 411 is output as (N+M)-bit data, where N and M are integers greater than zero. A first input can be configured to receive (N)-bit data, and a second input can be configured to receive (M)-bit data, where the first input can be configured to receive an activation value, and the second input can be configured to receive a weight value. The second controller 100 can control the internal memory 300 to reuse parameters stored in the internal memory 300 according to machine code. Parameter reuse means that parameters stored in the internal memory 300 are not deleted or otherwise copied or moved to memory 3000, but are reused in subsequent operations. According to the above configuration, as... Figure 5 As shown, it has the effect of reducing power consumption based on the operation of memory 3000. Figure 5 In this context, a 32-bit SRAM read refers to the energy required to read one bit of data from internal memory 300, while a 32-bit DRAM read refers to the energy required to read one bit of data from memory 3000 via bus 6000. It also has the effect of eliminating the latency that occurs when the neural processing unit 1000 sends data to and receives data from memory 3000 via bus 6000.

[0130] In other words, the second controller 100 can obtain reusable variable parameters and reusable constant parameters based on the machine code of the compiled neural network model. Therefore, the second controller 100 can be configured to control the internal memory 300 to reuse the parameters stored in the internal memory 300.

[0131] The processing element can constrain the operation of multiplier 411 so that when zero is input at one of the first and second input terminals of multiplier 411, multiplier 411 may not perform the operation, because if zero is multiplied by any number, the result of the operation will be zero even if the operation is not performed.

[0132] For example, when zero is input to one of the first and second inputs of multiplier 411, multiplier 411 can be configured to operate in a zero-jumping manner. To perform the zero-jumping, each processing element PE included in the processing element array 400 can be individually enabled or disabled. The second controller 100 can be configured to provide an enable or disable signal to each processing element PE clockwise. When a processing element PE is disabled, multiplier 411 can be configured to be disabled. Therefore, the power consumed by multiplier 411 operations can be reduced. For example, refer to... Figure 5 Information about the multiplier's power consumption is provided. Adder 412 can also be configured to be disabled when processing element PE is disabled. Therefore, the power consumed by adder 412 during operation can be reduced. For example, refer to... Figure 5Provides information about the adder's power consumption.

[0133] In some examples, each processing element PE may be designed to receive a corresponding control signal from the second controller 100 for controlling (i.e., enabling or disabling) the zero-jumping operation.

[0134] In some examples, each multiplier 411 of each processing element PE can be designed to receive a corresponding control signal from the first controller 100 for controlling zero-jumping operations. According to the above configuration, the power consumption of the multipliers can be reduced by zero-jumping.

[0135] In some examples, each adder 412 of each processing element PE can be designed to receive a corresponding control signal from the second controller 100 for controlling zero-jumping operations. According to the above configuration, zero-jumping can reduce the power consumption of the adders.

[0136] In some examples, each of the multipliers 411 and adders 412 of each processing element PE can be designed to simultaneously receive a corresponding control signal from the second controller 100 for controlling zero-jumping operations. According to the above configuration, the power consumption of the multipliers and adders can be reduced by zero-jumping.

[0137] In some examples, the weights are constant parameters trained on, and the machine code on which the neural network model containing the weights is compiled can be programmed to input corresponding control signals to each processing element PE that receives a zero weight value to control the zero-jump operation.

[0138] The number of bits of data input to the first and second input terminals can be determined based on the quantization of the node data and weight data of the corresponding layers in the neural network model. For example, the node data of the first layer can be quantized to 5 bits, while the weight data of the first layer can be quantized to 7 bits. In this case, the first input terminal can be configured to receive 5 bits of data, while the second input terminal can be set to receive 7 bits of data; that is, the number of bits of data input to each input terminal can be different.

[0139] The processing element PE can be configured to receive quantization information of the data input to each input. Neural network data locality information can include quantization information of both the input and output data of the processing element PE.

[0140] In some examples, the processing core 1000-1 can be controlled so that when quantization bit width information is input to the input of the processing element PE, the quantized data stored in the internal memory 300 is dynamically converted. That is, different tensors can have different quantization bit widths, and the processing element PE can be configured to generate input data by receiving bit width information from the processing core 1000-1 in real time as the bit width of the incoming data is converted.

[0141] Accumulator 413 uses adder 412 to perform multiple (L) loops to accumulate the operation values ​​of multiplier 411 and accumulator 413. Therefore, the number of data bits at the output and input terminals of accumulator 413 can be output as (N+M+log2(L)) bits, where L is a positive integer.

[0142] Once the accumulator 413 has finished accumulating, it can receive an initialization reset signal to initialize the data stored in the accumulator 413 to zero. However, the examples according to this disclosure are not limited thereto.

[0143] Accumulator 413 is configured to store the accumulated value even when zero-jumping is enabled in the corresponding processing element PE. Therefore, subsequent values ​​can be accumulated even when zero-jumping is enabled.

[0144] The bit quantization unit 414 can reduce the bit width of the data output from the accumulator 413. The bit quantization unit 414 can be controlled by the second controller 100. The bit width of the quantized data can be output as (X) bits, where X is a positive integer. According to the above configuration, the processing element array is configured to perform MAC operations, and the processing element array has the effect of quantizing and outputting the MAC operation results. In particular, as the number of (L) cycles increases, this quantization has the effect of further reducing power consumption. Reducing power consumption also has the effect of reducing heat generation in edge devices. In particular, reducing heat generation has the effect of reducing the probability of malfunction caused by high temperature of the processing core 1000-1.

[0145] The output data (X) bits of the bit quantization unit 414 can be equal to or different from the (N) bits and / or (M) bits. For example, the (X) bits can be set to a bit width such that no overflow of the output data (X) bits occurs based on the maximum value that can be accumulated in the accumulator 413. For example, the (X) bits can be 16 bits, 24 bits, or 32 bits.

[0146] According to an example of this disclosure, the processing element array of a processing core 1000-1 may include a multiplier 411, an adder 412, an accumulator 413, and a bit quantization unit 414. The bit quantization unit 414 can reduce the number of data bits (N+M+log2(L)) output from the accumulator 413 to (X) bits. A second controller 100 can control the bit quantization unit 414 to reduce the number of bits in the output data from the least significant bit (LSB) to a predetermined number of the most significant bit (MSB).

[0147] In some examples, the quantization level can be determined separately for each tensor of the neural network model.

[0148] Based on the processing element PE, by adjusting the number of bits of the (N)-bit data and (M)-bit data of the multiplier 411, and by determining the number of bits of the operation value (X)-bit by the bit quantization unit 414, the processing element array has the effect of preventing MAC operation overflow.

[0149] Figure 4 This is a schematic diagram illustrating an example neural network. An exemplary convolutional neural network can be a combination of one or more convolutional layers, pooling layers, and fully connected layers. Convolutional neural networks have a structure suitable for training and inference on two-dimensional data and can be trained via the backpropagation algorithm.

[0150] In one example of this disclosure, the convolutional neural network has a kernel for each channel, which extracts features from the input image for that channel. The kernels can be organized as a two-dimensional matrix and perform convolution operations as they traverse the input data. The size of the kernel can be arbitrary, and the stride of the kernel traversing the input data can also be arbitrary. The convolution result of the entire input data for each kernel can be called a feature map or activation map.

[0151] As used in this article, a kernel can include a single set of weights or multiple sets of weights. The number of kernels per layer can be referred to as the number of channels.

[0152] Therefore, since convolution is a combination of input data and a kernel, an activation function can be subsequently applied to add non-linearity. When an activation function is applied to a feature map that is the result of a convolution operation, it can be called an activation map.

[0153] Specific reference Figure 4 A convolutional neural network can include at least one convolutional layer, at least one pooling layer, and at least one fully connected layer. For example, a convolution can be defined by two main parameters: the size of the input data (typically a 1x1, 3x3, or 5x5 matrix) and the depth of the output feature map (the number of kernels). These key parameters can be computed using convolution. These convolutions can start at a depth of 32, continue to a depth of 64, and terminate at a depth of 128 or 256. A convolution operation can be understood as sliding a 3x3 or 5x5 kernel across the input data, an input image matrix, multiplying each weight of the kernel by each element of the overlapping input image matrix, and then summing them together.

[0154] Activation functions can be applied to the resulting output feature map to obtain the final output activation map. Pooling layers can perform pooling operations to downsample the output data (i.e., the activation map) to reduce the size of the feature map. For example, pooling operations can include, but are not limited to, max pooling and / or average pooling.

[0155] Max pooling uses a kernel and outputs the maximum value in a region of the feature map where the feature map and kernel slide and overlap. Average pooling outputs the average value in a region of the feature map where the feature map and kernel slide and overlap. Because these pooling operations reduce the size of the feature map, they also reduce the number of parameters in the feature map.

[0156] Fully connected layers can classify the data output from pooling layers into multiple categories (i.e., estimates) and output the classified categories and their scores. The data output from pooling layers is in the form of a three-dimensional feature map, which can be converted into a one-dimensional vector and input into the fully connected layer.

[0157] In one example, further reference Figure 2 According to this disclosure, an example of a neural network model processed by processing core 1000-1 may be related to image classification and object detection.

[0158] For example, the input data of the processing element array 400 of the neural processing unit 1000 in the aforementioned neural network model can be image data, and the output data of the processing element array 400 can be multiple bounding box data of the input image. Each of the multiple bounding box data can include bounding box coordinate data and category data. The bounding box coordinate data can include position confidence scores, height data, width data, X-coordinate data, and Y-coordinate data. Assuming the shape of the bounding box is rectangular, the bounding box coordinate data can include the height data, width data, X-coordinate data, and Y-coordinate data as described above. However, the shape of the bounding box is not limited to a square, but can be transformed into a pentagon or more polygons or a circle; therefore, the number and type of bounding box coordinate data can vary depending on the shape of the bounding box. In addition, the category data can include multiple categories classified as existing within the bounding box and their scores.

[0159] Figure 5 This is a table showing the energy consumption per unit operation of a neural processing unit according to an example of this disclosure. Power consumption reduction techniques for the internal memory 300 of the processing core 1000-1 will be described. Figure 5This representation symbolically explains the energy consumed per unit operation of the core 1000-1. Energy consumption can be divided into memory access, addition, and multiplication operations. "8b Add" refers to the 8-bit integer addition operation of adder 412. An 8-bit integer addition operation consumes 0.03 pj of energy. "16b Add" refers to the 16-bit integer addition operation of adder 412. A 16-bit integer addition operation consumes 0.05 pj of energy. "32b Add" refers to the 32-bit integer addition operation of adder 412. A 32-bit integer addition operation consumes 0.1 pj of energy. "16b FP Add" refers to the 16-bit floating-point addition operation of adder 412. A 16-bit floating-point addition operation consumes 0.4 pj of energy. "32b FP Add" refers to the 32-bit floating-point addition operation of adder 412. A 32-bit floating-point addition operation consumes 0.9 pj of energy. "8b Mult" refers to the 8-bit integer multiplication operation of multiplier 411. An 8-bit integer multiplication operation consumes 0.2 pJ of energy. "32b Mult" refers to the 32-bit integer multiplication operation of multiplier 411. A 32-bit integer multiplication operation consumes 3.1 pJ of energy. "16b FP Mult" refers to the 16-bit floating-point multiplication operation of multiplier 411. A 16-bit floating-point multiplication operation consumes 1.1 pJ of energy. "32b FP Mult" refers to the 32-bit floating-point multiplication operation of multiplier 411. A 32-bit floating-point multiplication operation consumes 3.7 pJ of energy. "32b SRAM read" refers to the read access of 32-bit data when internal memory 300 is static random access memory (SRAM). Reading 32-bit data from internal memory 300 consumes 5 pJ of energy. "32b DRAM read" refers to the read access of 32-bit data when main memory 3000 is DRAM. Reading 32 bits of data from memory 3000 to internal memory 300 consumes 640 pJ of energy. The unit of energy is picojoule (pJ).

[0160] When processing core 1000-1 performs 32-bit floating-point multiplication versus 8-bit integer multiplication, the energy consumption per unit of operation differs by approximately 18.5 times. When reading 32-bit data from memory 3000 configured as DRAM versus reading 32-bit data from internal memory 300 configured as SRAM, the energy consumption per unit of operation differs by approximately 128 times.

[0161] In other words, from a power consumption perspective, power consumption increases with the median of the data. Furthermore, floating-point operations consume more power than integer operations. Additionally, reading data from DRAM significantly increases power consumption.

[0162] Therefore, the internal memory 300 of the processing core 1000-1 according to one example of this disclosure may be configured to include a high-speed static memory such as an SRAM transistor, but not DRAM. However, the neural network processing unit according to the example of this invention is not limited to SRAM.

[0163] For example, internal memory 300 may not include DRAM, and internal memory 300 may be configured to include static memory that is configured to have relatively higher read and write speeds and consume relatively less power than memory 3000.

[0164] Therefore, compared to memory 300, the internal memory 300 of the processing core 1000-1 according to an example of this disclosure can be configured to have relatively high read and write speeds and relatively low power consumption for neural network model inference operations.

[0165] Static memory, such as SRAM, MRAM, STT-MRAM, eMRAM, and OST-RAM, can be driven at high speeds. Furthermore, MRAM, STT-MRAM, eMRAM, and OST-RAM are static memories and possess non-volatile characteristics. Therefore, static memories capable of high-speed computation (such as SRAM) can have the effect that, upon restarting after a power failure, there is no need to redundantly provide neural network models from memory 3000. However, the examples according to this disclosure are not limited to this.

[0166] According to the above configuration, the processing core 1000-1 has the effect of significantly reducing DRAM power consumption during inference operations of the neural network model. Furthermore, the SRAM storage cell of the internal memory 300 may include, for example, four to six transistors to store one bit of data. However, the examples according to this disclosure are not limited thereto. Additionally, the MRAM storage cell of the internal memory 300 may include, for example, a magnetic tunnel junction (MTJ) and a transistor to store one bit of data.

[0167] Figure 6A and Figure 6B This is a diagram illustrating an exemplary operation for reducing latency in a tensor according to a first example of this disclosure, wherein bus congestion occurs in a system for controlling the processing core.

[0168] Reference Figure 6A and Figure 6B When processing tensor n of a neural network model in a neural processing unit, it is preferable to prefetch the data required for the operation of subsequent tensor n+1 via DMA (using DMA to transfer tensor n+1 is referred to as "DMAn+1" below). n T refers to the time it takes for a neural processing unit to perform computations on tensor n. dThis refers to the time spent by DMA transferring tensor n+1 to the internal memory of the neural processing unit. DMA transfer of tensor n+1 can be performed, for example, as a prefetch operation. If the prefetch of DMA n+1 for calculating tensor n+1 is completed before the neural processing unit completes the calculation of tensor n, the neural processing unit can process the calculation of tensor n+1 without data starvation. Conversely, if the prefetch of DMA n+1 is not completed when the operation on tensor n is completed, the calculation of tensor n+1 may be delayed.

[0169] like Figure 6A As shown in case 1, when calculating the tensor n of T in the neural processing unit... n Short T-shirt d During the DMA prefetching process n+1, computational operations on tensor n+1 can begin without delay. That is, when T... n During the computation of tensor n in the neural processing unit, DMA 200 can use tensor n+1 to transfer parameters for performing computational operations in internal memory 300 via bus 6000. However, in case 2, if the DMA n+1 time increases to T′... d The computation operation using tensor n+1 can be performed when tensor n terminates and T w It will begin after the expiration date.

[0170] Therefore, according to an example system of this disclosure, the order and / or timing of operations performed on each tensor can be adjusted, such as... Figure 6B As shown, the QoS of DMA is adjusted to reduce T. w In other words, latency (the time during which computational circuitry is not running) remains constant, even during bus congestion in DMA. DMA's QoS can be represented, for example, by parameters indicating memory request priority or urgency, parameters associated with bus bandwidth (e.g., guaranteed bandwidth, maximum bandwidth, percentage of total bandwidth), parameters indicating tolerable latency in data transfer, cache parameters, parameters indicating jitter, and parameters indicating packet loss. As described below, one or more of these QoS parameters can be controlled on a tensor basis so that neural processing units can prefetch tensors efficiently and timely for computational operations.

[0171] The time T for performing tensor n computation n This can be determined at compile time or monitored in real time. Furthermore, the amount of data to be DMA n+1 within that time period can be determined. However, with T... n Similarly, the operation time T of DMA n+1 dThis can be difficult to calculate or predict because the bandwidth that can be allocated to DMA varies depending on bus conditions. Furthermore, the system bus can be allocated in real-time for transferring data between various circuits (e.g., CPU, PCIe), rather than between memory and neural processing units. Therefore, the theoretical time T... d In practice, the number of buses can be increased according to the actual situation of the bus.

[0172] The interval at which the neural processing unit computes tensor n, i.e., the clock cycle for computing tensor n, can be called T. n On the other hand, the neural processing unit needs to use DMA to compute the time interval of tensor n+1, i.e., the memory cycle of DMA n+1, which can be called T. d However, T d It can be increased variably depending on the situation on the bus.

[0173] Compare T n and T d In each period, if T n Much greater than T d ,like Figure 6B In scenario 1, the prefetch completion time of DMA n+1 is relatively faster than the computation completion time of tensor n, and even considering various dynamic conditions on the bus, the system can determine T. d Less than T n The probability is high because DMA has sufficient time margin.

[0174] On the other hand, if T n Not significantly greater than T d (like Figure 6B In scenario 2), the prefetch completion time of DMA n+1 is relatively faster than the computation completion time of tensor n. Therefore, if bus congestion causes insufficient time for data transfer using DMA, T d Become greater than T n The probability of that increases.

[0175] In other words, according to one example, the system can be based on T n With T d The ratio, in the value When the value is equal to or greater than 1, the DMA n+1 transmitted via the bus is prioritized. That is, according to an example of this disclosure, the system can be configured based on T... n With T d The ratio, based on Prioritize DMA n+1 transmitted on the bus based on values ​​and / or preset thresholds.

[0176] Furthermore, according to one example, the system can be based on T n With Td The ratio and bus congestion level are used to increase the priority of DMA n+1 sent through the bus, where The value is compared with a predetermined threshold. The degree of congestion in the bus can be determined based on the bandwidth sharing of various additional circuits connected to the bus. The higher the degree of bus congestion, the higher the T value. d The greater the probability, the higher the chance of it increasing.

[0177] The threshold can be set to, for example, 0.9. Therefore, when In such cases, the system can be configured to determine that the transmission of tensor n+1 may be delayed and cause bus congestion. Therefore, the bus transmission priority of DMA n+1 is increased to prevent delayed transmission of tensor n+1. However, the aforementioned threshold can be appropriately determined according to the degree of bus congestion, and this disclosure is not limited thereto.

[0178] As an alternative or additional solution, the modified formula The threshold can be set to, for example, 1.1. Therefore, when In such cases, the system can be configured to determine that the transmission of tensor n+1 may be delayed and cause bus congestion. Therefore, the bus transmission priority of DMAn+1 is increased to prevent delayed transmission of tensor n+1. However, the aforementioned threshold can be appropriately determined according to the degree of bus congestion, and this disclosure is not limited thereto.

[0179] As mentioned above, by using Adjusting the DMA's QoS parameters to prioritize bus bandwidth for tensor n+1 transfers can effectively utilize the DMA bus to reduce the latency T associated with processing tensor n+1. w For example, if If the value is less than a preset threshold, the QoS parameter can be reduced, thereby lowering the priority associated with accessing the bus; if If the value is higher than the preset threshold, the QoS parameter can be increased, thereby improving the priority associated with accessing the bus.

[0180] Therefore, the control system can improve the performance of each neural processing unit by increasing the priority of DMA n+1 tensors that may experience delays in transmission to the neural processing unit due to, for example, bus congestion.

[0181] QoS mechanisms can be used to prioritize traffic on the bus, manage bandwidth allocation, and reduce latency, jitter, and packet loss to improve overall system performance. QoS parameters can be controlled or adjusted to achieve, in particular, the following objectives:

[0182] Bandwidth allocation: Control the bus to ensure that each processing core has sufficient bus bandwidth to receive and transmit data through the bus for its operation.

[0183] Priority: Each tensor can be assigned a specific priority. Based on the priority, bus bandwidth can be allocated differently and / or the order of data requests in the ordered queue on the bus can be adjusted. The bus may include additional ordered queue memory.

[0184] Traffic shaping: It can control data flow to improve or ensure performance, reduce latency, and ensure bandwidth.

[0185] Resource reservation: High-priority circuit units (such as neural processing units) can be reserved to maintain bus performance.

[0186] Figure 7 This is a flowchart illustrating control at the processing core according to a first example of this disclosure. This control method can be executed by a first controller 1100 that controls the neural processing unit 1000. (Refer to...) Figure 7 The first controller 1100 can perform S110: determining at least one data starvation period of the neural processing unit 1000 based on memory 3000 access operations for each tensor of the neural processing unit 1000 and computation operations on the data.

[0187] In this configuration, computational operations and memory access operations for each tensor can be performed within a given bus bandwidth so that each neural processing unit 1000 can communicate with the memory 3000.

[0188] To determine data starvation periods, the first controller 1100 can compare the computation cycle and memory cycle of each tensor in the neural processing unit 1000. For this purpose, the first controller 1100 can receive or monitor computation cycle and memory cycle information for each tensor.

[0189] Specifically, the first controller 1100 can compare the first processing time (i.e., computation cycle) spent completing a computational operation on a specific tensor with the second processing time (i.e., memory cycle) spent completing a memory access operation on the next tensor to follow the specific tensor, and identify the difference between the first and second processing times as a data starvation period. The first and second processing times are unique characteristics of the tensor determined based on the size of the tensor parameters and the complexity of the computational algorithm in the neural network model. Therefore, the first and second processing times can be pre-analyzed during the compilation phase of the neural network model.

[0190] Next, the first controller 1100 performs S120: prioritizing the memory access operations of each tensor of the neural processing unit 1000 such that data starvation periods do not occur or are reduced. The first controller 1100 can be configured to determine the priority of the memory access operations of the neural processing unit 1000 for tensors in which at least one data starvation period occurs. The first controller 1100 can be configured to control the bus based on the determined priority ordering.

[0191] When the first controller 1100 determines that a neural processing unit performing tensor computation operations may suffer at least one data starvation period, the first controller 1100 may adjust QoS parameters to eliminate or reduce the identified data starvation period. That is, if a data starvation period occurs because the second processing time (i.e., memory cycle) for completing a memory access operation is relatively longer than the first processing time (i.e., computation cycle), the first controller 1100 may assign higher priority to memory access operations of tensors to be read during the second processing time. Conversely, if the first processing time for completing a computation operation is sufficiently longer than the second processing time, the first controller 1100 may be configured to yield bus bandwidth by assigning lower priority to the corresponding memory access operations.

[0192] If the first processing time is longer than the second processing time, the first controller 1100 can assign a lower priority to the neural processing unit requesting a memory access operation with a relatively low probability of data starvation, so that the bus preferentially allocates bus bandwidth to circuits other than the neural processing unit (e.g., other neural processing units, other processing cores, CPU, decoder, image sensor, etc.).

[0193] In one aspect, if the second processing time is longer than the first processing time, the first controller 1100 may increase the priority of neural processing units requesting memory access operations to reduce or eliminate data starvation periods of neural processing units, and the bus may process memory access request operations of neural processing units first to further utilize additional available bus bandwidth.

[0194] In some examples, the first controller 1100 may grant a relatively higher bandwidth on the bus to a particular neural processing unit than to others, based on a first processing time and a second processing time requested by each of the plurality of neural processing units for each tensor. The bandwidth allocation on the bus can be dynamically adjusted to reduce the data starvation period associated with each tensor. Therefore, the data starvation period of the plurality of neural processing units included in system 10000 can be reduced or eliminated.

[0195] In other words, even if the second processing time to complete the memory access operation of a specific neural processing unit increases slightly, as long as the main processing time to complete the computation operation is long enough, it can still give up at least a portion of the bus bandwidth to other neural processing units (i.e., for neural processing units with overlapping memory access operations).

[0196] On the other hand, if the second processing time for completing a memory access operation of a specific neural processing unit is long enough compared to the first processing time for completing a computational operation, more memory access opportunities can be obtained by acquiring bus bandwidth from one or more other neural processing units (whose memory access operations overlap with those of the specific neural processing unit in time), thereby reducing the time when the computational circuitry of the specific neural processing unit is not running, and thus completing the memory access operation faster.

[0197] Figure 8 This is a diagram illustrating an exemplary method for determining the priority of a control processing core according to a first example of this disclosure. C(n) represents a first processing time, which is a computation cycle for completing a computational operation on a specific tensor, while D(n+1) represents a second processing time, which is a memory cycle for completing a memory access operation on the next tensor after that specific tensor. When the second processing time is longer than the first processing time, i.e., when the data hunger level (e.g., D(n+1) / C(n)) is greater than a first threshold Th1 (e.g., Th1 is set to 1), the first controller 1100 can assign a higher priority to the memory access request corresponding to D(n+1). Therefore, by obtaining bus bandwidth from another neural processing unit, the operation of D(n+1) of a specific neural processing unit can be accelerated. Thus, the total time for processing data is reduced because the time when the computing circuit is not running (i.e., the data hunger period) is reduced.

[0198] Furthermore, if the second processing time is less than the first processing time, i.e., if the data hunger level is less than a second threshold Th2 (e.g., Th2 is set to 1), the first controller 1100 can allocate a lower priority to the memory access request corresponding to D(n+1) because the first processing time is longer. Therefore, at least a portion of the bandwidth allocated to the D(n+1) operation can be at least partially supplied to one or more other neural processing units. As a result, the data hunger period of one or more other neural processing units is reduced or eliminated, thereby reducing the total time spent by all neural processing units processing data.

[0199] On the other hand, when the first processing time and the second processing time are equal, that is, when the data hunger level is equal to the third threshold (e.g., 1), since this corresponds to the case where there is no data hunger period, the first controller 1100 can assign normal priority to the data hunger period.

[0200] In other words, the example system can calculate the data hunger level for a specific tensor and determine that the tensor has high priority by comparing the data hunger level with a first threshold. Furthermore, the system can determine that the tensor has low priority by comparing the data hunger level with a second threshold. Additionally, the system can maintain the tensor's priority when the data hunger level and a third threshold are equal. The first and second thresholds can be equal. Furthermore, the second and third thresholds can be equal.

[0201] In some examples, the first threshold can be greater than the third threshold. The second threshold can be less than the third threshold. The third threshold can be a range between the first and second thresholds. Specifically, for example, the first threshold can be 1. When the data hunger level is 1, the corresponding tensor theoretically has no data hunger, but due to various overhead and bandwidth contention on the bus, data hunger is considered likely to occur, even if it is temporary, and therefore its priority should be increased. The second threshold can be 0.8. If the data hunger level is 0.8, even if various overhead and bandwidth contention occur on the bus, the corresponding tensor is unlikely to experience data hunger, but there may be sufficient bus bandwidth to yield, and its priority can be reduced. The third threshold can be a range between the first and second thresholds. When the data hunger level is between 0.7 and 1, even considering various overhead and bandwidth contention on the bus, data hunger can be considered unlikely to occur, but there may not be enough bus bandwidth to yield. In other words, the system can be configured to calculate the data hunger level for each tensor, increase the tensor's priority based on the first threshold, decrease the tensor's priority based on the second threshold (which is different from the first threshold), and maintain the tensor's priority based on a value between the first and second thresholds (i.e., the third threshold).

[0202] Figure 9 This is a diagram illustrating how, according to a first example of this disclosure, data processing speed can be improved by prioritizing the reduction of data starvation periods. Figure 9 The diagram illustrates the memory cycle (MEM) and computation cycle (COMP) for each tensor processed by each Neural Processing Unit (NPU). Each NPU can store the necessary parameters in its internal memory within the corresponding memory cycle for each tensor, and then use the parameters stored in the internal memory to process the neural network model's operations within the corresponding computation cycle. In other words, for an NPU to process a tensor, the NPU's DMA first commands the bus to transfer the tensor to the NPU's internal memory by sending a memory operation request, and then the NPU's processing element performs computations using the tensor stored in the internal memory during the computation cycle.

[0203] Reference Figure 9NPU0 refers to a neural processing unit. An NPU0 can include a single processing core or multiple processing cores. For example, an NPU0 can correspond to... Figure 2 Neural processing unit 1000 or Figure 2 The processing core is 1000-1. NPU1 refers to another neural processing unit. For example, NPU1 can correspond to... Figure 2 The processing core is 1000-n.

[0204] Reference Figure 9 As shown in (a), the first processing time (i.e., computation cycle) of the data operation in the first tensor n+1 processed by NPU0 is shorter than the second processing time (i.e., memory cycle) of the data memory access operation in the second tensor n+2. Therefore, a data starvation period occurs between the computation cycles of the first tensor n+1 and the second tensor n+2 of NPU0 until the memory cycle of the second tensor n+2 of NPU0 is completed. The bus bandwidth of the memory cycle of the second tensor m+2 of NPU1 can be utilized, which overlaps at least partially with the memory cycle of the second tensor n+2 of NPU0 on the time axis. That is, since the memory cycle of the second tensor of NPU1 is completed before the computation cycle of the first tensor m+1 of NPU1 is completed, there is a bandwidth margin in the bus bandwidth of the memory cycle of the second tensor m+2 of NPU1 before the computation cycle of the second tensor m+2 of NPU1 begins. Therefore, relinquishing at least a portion of the bus bandwidth allocated to the memory cycle of the second tensor m+2 of NPU1 to NPU0 can essentially eliminate data starvation between the computation cycles of the first tensor m+1 and the second tensor m+2 of NPU1.

[0205] In other words, based on an example system, the memory cycles and computation cycles of continuously processed tensors can be compared to determine one or more data starvation periods or one or more bandwidth retention periods.

[0206] In other words, the system according to the first example can determine data starvation periods between consecutive tensors, each of which is processed in a first neural processing unit among multiple neural processing units. Furthermore, the system according to the first example of this disclosure can be configured to, among tensors processed in a second neural processing unit of the multiple neural processing units, identify a tensor whose reserved bus transmission bandwidth can be at least partially relinquished to another tensor whose extended transmission on the bus may have or has already caused a data starvation period. Therefore, the system according to one example can reallocate bus bandwidth previously allocated to a neural processing unit with sufficient bus bandwidth to another neural processing unit that is experiencing or may experience a data starvation period.

[0207] For example, as shown in (b), by assigning a high priority to memory access operations of NPU0's second tensor n+2 and a low priority to memory access operations of NPU1's second tensor m+2, at least a portion of the bus bandwidth allocated to NPU1 can be reallocated to NPU0, thereby reducing the data starvation period of NPU0. In this case, NPU1 may essentially not experience data starvation in its second tensor m+2 because NPU1 is in a bandwidth reservation period.

[0208] On the other hand, as shown in (a), the memory cycle of NPU1's third tensor m+3 is longer than the computation cycle of NPU1's second tensor m+2. Therefore, a data starvation period occurs between the computation cycle of NPU1's second tensor m+2 and NPU1's third tensor m+3 before the memory cycle of NPU1's third tensor m+3 is completed. Since the computation cycle of NPU0's second tensor n+2 is longer than the memory cycle of NPU0's third tensor n+3, there is bandwidth slack in the memory cycle of NPU0's third tensor n+3. Therefore, at least a portion of the bus bandwidth allocated to transmit the memory cycle of NPU0's third tensor n+3 can be reallocated to facilitate the transmission of NPU1's third tensor m+3 via the bus.

[0209] Therefore, the controller of the system in the first example (e.g., the first controller) can adjust the priority of memory access operations of the third tensor n+3 of NPU0 and the third tensor m+3 of NPU1, respectively. Thus, as shown in (b), by assigning a high priority to the memory access operations of the third tensor m+3 of NPU1 and a low priority to the third tensor n+3 of NPU0, at least a portion of the bus bandwidth of NPU0 can be reallocated to NPU1 to reduce the data starvation period of the third tensor m+3 of NPU1.

[0210] In summary, the system according to the first example can be configured to determine the data hunger period for each tensor of a neural processing unit, determine the bus bandwidth reservation period along the time axis of tensors of another neural processing unit that overlaps with the data hunger period, and reduce the data hunger period by differentiating the priorities for each of the tensors with data hunger periods and the tensors with bandwidth reservation periods. (See reference...) Figure 9 In (a) and (b), when the memory cycles of tensors with data starvation periods and tensors with bandwidth retention periods overlap by at least a portion, the processing time of NPU0 and NPU1 can be reduced by adjusting the priority of memory access operations for each tensor, respectively.

[0211] In the following, second and third examples of this disclosure will be described, wherein an operation to check the state information of each neural processing unit in real time is performed without comparing the computation cycle and memory cycle of each tensor of the neural processing unit 1000, thereby allowing each tensor to be given an appropriate priority to reduce data starvation.

[0212] Figure 10 This is a diagram illustrating an example of a priority determination method in a processing core control method according to a second example of this disclosure. A first controller 1100 can receive status information from a second controller 100 for each processing core 1000-1, ..., 1000-n, and can perform S210: determining whether each processing core 1000-1, ..., 1000-n is in a busy state based on the received status information. The busy state indicates whether the processing core is in use or currently processing. The status information can be updated in real time.

[0213] Next, the priority of each processing core 1000-1, ..., 1000-n can be determined according to the determination in step S210. Specifically, in step S221, high priority is assigned to processing cores that are not currently busy; and in step S222, low priority is assigned to processing cores that are currently busy. A busy state for a processing core indicates that it is currently executing a computational operation on a specific tensor, therefore it is given a lower priority because the memory access operation for the next tensor does not require fast execution. A non-busy state for a processing core indicates a data starvation state where the computational operation for the next tensor has not yet been executed, therefore it is given a high priority because the memory access operation for the next tensor requires fast execution.

[0214] The ordered queues on bus 6000 can be reordered according to the adjusted priorities. The reordered ordered queues can be stored in an ordered queue memory or reordered. According to a second example of this disclosure, the first controller 1100 can determine the busy state of each NPU and reorder the ordered queues on bus 6000 according to priority. However, this disclosure is not limited thereto; the ordered queues on bus 6000 can also be configured to be reordered by at least one of the second controller 100, CPU 2000, DMA 200, or bus 6000.

[0215] Figure 11 This is a diagram illustrating an example of a data hunger signal generated during the runtime of the processing core according to the second example of this disclosure. Figure 11 Similar to Figure 9 Furthermore, for the sake of brevity, redundant descriptions have been omitted here.

[0216] First, the second controller 100 for each processing core 1000-1, ..., 1000-n can send its status information in the form of data hunger signals 11 and 12. A data hunger signal is a signal indicating that the processing element PE is in an idle state during processing core operation. A data hunger signal can indicate that the computational operation of the processing core has stopped. The status information for each processing core 1000-1, ..., 1000-n can be generated by checking the status of the components controlled by each second controller 100. Each data hunger signal can be an independent signal. As will be further described below, signals indicating idle state information can be generated by the second controller 100.

[0217] Reference Figure 11 For NPU0, as shown in (a), NPU0 disables the first data hunger signal 11 while in a busy state processing the computation operation COMP of the first tensor n. For example, the disable signal can be a first-level signal, while the activation signal can be a second-level signal. Alternatively, the disable signal can be a second-level signal, while the activation signal can be a first-level signal. The first level can be indicated by a low-level voltage signal, while the second level can be represented by a high-level voltage signal with a voltage higher than the low-level voltage signal. Furthermore, NPU0 can disable the first data hunger signal 11 while in a busy state processing the computation operation COMP of the first tensor n+1. Between the 0th tensor n and the first tensor n+1 of NPU0, no data hunger period caused by the memory access operation MEM will occur. Then, NPU0 activates the first data hunger signal 11 in a data hunger state in which the computation operation COMP of the first tensor n+1 has been completed, but the computation operation COMP of the second tensor n+2 has not been executed.

[0218] Therefore, as shown in (b), the first controller 1100 can be configured to assign a higher priority to the processing core that enables the first data hunger signal 11. This increases the bus bandwidth of the memory access operation MEM of the high-priority tensor n+2, thus reducing the data hunger period shown in (b) compared to (a).

[0219] Furthermore, in some examples, when the first data hunger signal 11 is activated while the second data hunger signal 12 of another processing core is deactivated, the first controller 1100 can assign a lower priority to the other processing core. In this case, the first data hunger period shown in (b) can be further reduced.

[0220] Meanwhile, for NPU1, as described in (a), NPU1 can disable the second data hunger signal 12 while in a busy state during the computation operation COMP processing the first tensor m. Subsequently, NPU1 can also disable the second data hunger signal 12 while in a busy state during the computation operation COMP processing the first tensor m+1. Therefore, there is no data hunger period caused by the memory access operation MEM between the first tensor m and the second tensor m+1 of NPU1.

[0221] NPU1 can deactivate the second data hunger signal 12 while the second tensor m+2 computation operation COMP is in a busy state. NPU1 can activate the second data hunger signal 12 while the second tensor m+2 computation operation COMP has been completed but the third tensor m+3 computation operation COMP has not been executed.

[0222] Therefore, as shown in (b), the first controller 1100 can be configured to give higher priority to the processing core that enables the second data hunger signal 12. The bus bandwidth of the memory access operation MEM of the high-priority tensor m+3 can be increased, and thus the data hunger period shown in (b) can be reduced compared to (a).

[0223] Furthermore, in some examples, when the second data hunger signal 12 is enabled and the first data hunger signal 11 of another processing core is disabled, the first controller 1100 can be configured to give the other processing core a lower priority. In this case, the second data hunger period shown in (b) can be further reduced.

[0224] As described above, the first controller 1100 can be configured to perform real-time priority sorting by checking the busy status in real time based on data hunger signals 11 and 12 received from each processing core 1000-1, ..., 1000-n, rather than comparing the period of each tensor processed by multiple processing cores 1000-1, ..., 1000n. Furthermore, by displaying the priority of each processing core based on whether it is busy, causing each processing core 1000.1, ..., 1000-n to yield bus bandwidth, data hunger periods can be reduced, enabling the computing circuitry to operate quickly.

[0225] Figure 12 This is a diagram illustrating an example method, according to a third example of this disclosure, for determining priorities to reduce delays identified by the count values ​​of a counter (e.g., a counter circuit). For brevity, details related to... Figure 10 and Figure 11 Repeated Figure 12 Explanation.

[0226] According to the third example, a counter may be provided at a specific location within system 10000. The counter may be included in one of the first controller 1100, neural processing unit 1000, bus 6000, and CPU 2000, and this disclosure is not limited to the location of the counter. Figure 2 In the example, counter 110 is shown as being included in the first controller 1100. (See reference...) Figure 12 Each processing core 1000-1, ..., 1000-n may include a counter and may be configured to perform counting when a memory access operation is performed based on a counter threshold of the memory access operation. In this case, the number of clock cycles in each memory access operation MEM can be pre-calculated based on the tensor size, and the maximum value of the counter can be the sum of the pre-calculated number of clock cycles based on the tensor size (e.g., 0 to t clock cycles) plus a certain number of clock cycles (e.g., 0 to 100 clock cycles).

[0227] On the other hand, a counter threshold for memory access operations can be preset, and each processing core 1000-1, ..., 1000-n can count based on this threshold.

[0228] When the counter value is lower than the counter threshold, each processing core 1000-1, ..., 1000-n determines that no data starvation period has occurred and disables the data starvation signal for the remaining interval.

[0229] If the counter value of each processing core 1000-1, ..., 1000-n exceeds a preset threshold, a data starvation period is determined to have occurred, and a data starvation signal is activated during the period A exceeding the preset maximum counter value. This activation can remain active until the memory access operation is completed. Therefore, the interval at which the data starvation signal is activated is assigned a high priority by the first controller 1100. Based on the higher priority of the memory access operation caused by the activation of the data starvation signal, the bus bandwidth of the memory access operation increases, which leads to a reduction in the data starvation period. When the memory access instruction is completed, the counter can be reset, and the priority can be reduced again.

[0230] For example, such as Figure 12 As shown, when the counter threshold is set to t+100 clock cycles, each processing core 1000-1, ..., 1000-n can disable the data starvation signal during intervals when the counter value is below the counter threshold, and enable the data starvation signal during intervals when the counter value exceeds the counter threshold. The threshold can be appropriately determined by considering the characteristics of various communication networks; that is, when the counter threshold is exceeded, a bottleneck is determined to have occurred on the bus.

[0231] As described above, by having a counter to perform counting for each processing core 1000-1, ..., 1000-n, and by allowing the first controller 1100 to determine its busy state via a correspondingly activated data hunger signal, the first controller 1100 can dynamically adjust its priority. Furthermore, by allowing each processing core 1000-1, ..., 1000-n to yield bus bandwidth by prioritizing it according to whether it is busy, computing circuitry with reduced data hunger periods can operate.

[0232] The following describes a fourth example that applies the first and second / third examples of this disclosure. Figure 13 This is a diagram illustrating the prioritization process according to a fourth example of this disclosure. The first controller 1100 may perform S310: comparing the clock cycles of computational operations and memory access operations for each tensor of the neural processing unit 1000 to identify at least one data starvation period. Information regarding the duration of each computational cycle and each memory cycle for each tensor of the neural network model may be included in the neural network model to be processed by the neural processing unit. In step S310, the first controller 1100 may be configured to compare a first processing time T1 (e.g., computational cycle) for completing a computational operation at a specific tensor with a second processing time T2 for a memory access operation (e.g., memory cycle) for reading data required for a computational operation at a subsequent tensor after completing that specific tensor, and determine that an interval where the second processing time T2 is longer than the first processing time T1 is a potential data starvation period. If the first processing time is not greater than the second processing time, i.e., if it can be determined that the computational cycle will complete faster than the memory cycle, the system may be configured to determine that a data starvation period is unlikely to occur. Therefore, the first processing time T1 and the second processing time T2 for each tensor may be compared. The first processing time T1 and the second processing time T2 are included in the neural network model and can be provided to the neural processing unit in advance. If the first processing time T1 is greater than the second processing time T2, the tensor is determined to be within the computationally limited CB interval. If the first processing time T1 is shorter than the second processing time T2, the tensor is determined to be within the memory-limited MB interval. The first and second processing time information can be included in the corresponding neural network model and are initial values ​​determined based on the tensor size. The second processing time can change in real time according to the real-time bandwidth allocation of the bus.

[0233] Next, based on the data hunger signal generated by the second controller 100, it can be determined in real time whether the NPU is in a busy state. Specifically, for memory-constrained tensors, based on the data hunger signal received from the second controller 100 for each processing core 1000-1, ..., 1000-n, the second controller 200 can perform S321: determine whether the corresponding processing core is in a busy state. If the computation cycle completes faster than the memory cycle, the start of the computation cycle for subsequent tensors may be delayed until the memory cycle is completed; that is, a data hunger period may occur.

[0234] In step S310, the memory cycles allocated to the tensor can be either a default priority or a high priority, and these memory cycles are predicted to encounter data starvation periods determined based on the first processing time T1 and the second processing time T2.

[0235] If the processing core is busy in step S321, the first controller 1100 can proceed to S331: assigning the processing core a default priority. That is, although a data starvation period was predicted to occur in step S310, confirming that the NPU is busy confirms that a data starvation period has not actually occurred. Therefore, in step S331, the processing core can retain its default priority based on the fact that it is still performing computational operations.

[0236] If the corresponding processing core is not busy in step S321, the first controller 1100 can proceed to S332: assigning high priority to the corresponding processing core, that is, predicting in step S310 that a data starvation period may occur on the corresponding processing core, and confirming in step S321 that the computation on the corresponding processing core has actually stopped. Therefore, the corresponding processing core can be regarded as being in a data starvation state DS and can be configured to receive high bus bandwidth with a high priority setting.

[0237] On the other hand, for computationally constrained tensors, based on the data hunger signals received from the second controller 100 of each processing core 1000-1, ..., 1000-n, the second controller 200 can perform S322: determine whether the corresponding processing core is busy. If the computation cycle is later than the memory cycle, the start of the next tensor's computation cycle is unlikely to be delayed until the memory cycle is completed, i.e., a data hunger period is unlikely to occur.

[0238] In step S310, the memory cycle allocated to the tensor can be either low or high priority, and the memory cycle is predicted not to experience a data starvation period determined based on the first processing time and the second processing time.

[0239] If the corresponding processing core is busy in step S322, the first controller 1100 can proceed to S333: assign a low priority. That is, in step S310, it is predicted that a data starvation period is unlikely to occur, and confirming that the NPU is busy confirms that a data starvation period is unlikely to occur. Therefore, in step S322, the first controller 1100 can determine that a data starvation period is highly unlikely to occur based on the reason why the processing core is still computing, and can adjust the priority to a low priority.

[0240] If the processing core is not busy in step S322, the first controller 1100 can proceed to S332: assigning high priority to the processing core. In other words, in step S310, it was predicted that a data starvation period was unlikely to occur in the processing core, but contrary to the prediction, it was confirmed in step S322 that the processing core actually stopped working. Therefore, the corresponding processing core can be identified as being in a data starvation state DS and can be configured to have high priority and be provided with high bus bandwidth. That is, the computation cycle and memory cycle of each tensor can be compared to first determine the probability of a data starvation period occurring, and then determine in real time whether a data starvation period has actually occurred. Furthermore, as the second processing time becomes relatively longer than the first processing time, the probability of occurrence and the data starvation period may become longer. Therefore, when a data starvation period occurs, the system can allocate higher bus bandwidth in a priority manner, and then can allocate bus bandwidth differently based on the characteristics of the first and second processing times according to the default priority or lower priority. The first and second processing times are unique characteristics determined based on the size of the tensor parameters of the neural network model and the complexity of the computation algorithm. Therefore, the first and second processing times can be pre-analyzed during the compilation phase of the neural network model. In other words, the fourth example of this disclosure can provide the effect of optimal bus bandwidth allocation by taking into account pre-analyzed static computation scheduling information and real-time bandwidth contention of various data communications occupying the actual bus.

[0241] In other words, the fourth example could allow for finer-grained prioritization, which would allow bus bandwidth to be reallocated to allow neural processing units to operate more efficiently.

[0242] Figure 14 This is a graph illustrating how to improve data processing speed by prioritizing the reduction of data starvation periods, based on the fourth example. (See reference...) Figure 14In (a), if the control method according to the fourth example of this disclosure is not applied, data starvation periods may occur in some tensors of NPU0. Specifically, a first data starvation period DS1 may occur from the completion of the computation cycle COMP of the first tensor n+1 of NPU0 to the completion of the memory cycle MEM of the second tensor n+2. Then, a second data starvation period DS2 may occur from the completion time of the computation cycle COMP of the second tensor n+2 of NPU0 to the completion time of the memory cycle MEM of the third tensor n+3, that is, tensors with memory-limited characteristics may have data starvation periods. On the other hand, as shown in (a), even if the control method according to the fourth example of this disclosure is not applied, data starvation periods may not occur in NPU1. Specifically, the computation cycle of all tensors in NPU1 is longer than the memory cycle. In this case, data starvation periods may not occur on NPU1, that is, tensors with computationally limited characteristics may not experience data starvation periods.

[0243] Reference Figure 13 and Figure 14 (a) According to the fourth example of this disclosure, the first controller can compare information about the computation cycle and memory cycle for each tensor processed on NPU0 and NPU1. That is, a first processing time and a second processing time for each tensor to be processed by each NPU can be compared. The processing time information may be part of the scheduling information of the neural network model. The processing time information may be information included in the respective neural network models and is an initial value determined based on the tensor size. The second processing time can vary in real time according to the real-time bandwidth of the bus. Tensors with a first processing time greater than the second processing time can be identified as computationally limited CB intervals. Tensors with a first processing time shorter than the second processing time can be identified as memory-limited MB intervals. Specifically, since the computation cycle COMP of the first tensor n on NPU0 is completed before the memory cycle MEM of the first tensor n+1, it is identified as a computationally limited CB interval. Here, as an example, all tensors on NPU1 are also identified as computationally limited CB intervals. Since the computation cycle COMP of the first tensor n+1 and the second tensor n+2 of NPU0 is completed before the memory cycle MEM of the second tensor n+2 and the third tensor n+3 of NPU0, the aforementioned tensors are identified as memory-limited MB intervals, which corresponds to Figure 13 Step S310.

[0244] Reference Figure 13 and Figure 14(b) According to the fourth example of this disclosure, the system can be configured to generate data hunger signals in real time. A first controller can determine in real time whether each processing core is in a data hunger state based on the data hunger signal IDLE generated by the second controller. The data hunger signal IDLE can be enabled when the NPU is not busy and disabled when the NPU is busy. The first controller can be configured to dynamically check the activation of the data hunger signal IDLE when processing computationally limited CB tensors. The first controller can be configured to dynamically check the activation of the data hunger signal IDLE when processing memory-limited MB tensors. Specifically, the data hunger signal IDLE for the 0th tensor n and the first tensor n+1 interval, determined to be the NPU0 computationally limited CB interval, is disabled. The data hunger signal IDLE for the first tensor n+1 and the second tensor n+2, determined to be the NPU0 memory-limited MB interval, is disabled and then enabled after the computation cycle COMP of the first tensor n+1 is completed. The data hunger signal IDLE for the second tensor n+2 and the third tensor n+3, determined to be the NPU0 memory-limited MB interval, is disabled and then enabled after the computation cycle COMP of the second tensor n+2 is completed. The data starvation signal IDLE for all tensors m, m+1, m+2, and m+3 identified as NPU1 computationally limiting the CB interval is disabled. This corresponds to... Figure 13 Steps S321 and S322.

[0245] Reference Figure 13 and Figure 14 (b) When the data hunger signal IDLE is disabled, the system according to the fourth example can set the bus priority of the tensor of the memory-limited BM characteristic to the default priority D, which can correspond to Figure 13 Step S331. When the data hunger signal IDLE is disabled, the system according to the fourth example can set the priority of the bus for computing the tensor that limits the CM characteristics to a low priority L, which can correspond to... Figure 13 Step S333. When the data hunger signal IDLE is activated, the system according to the fourth example can set the priority of the bus of the corresponding tensor to high priority H. If the data hunger signal IDLE is activated, the first controller can assign high priority H to the tensor corresponding to the data hunger signal IDLE, regardless of memory or computational limitations, which can correspond to Figure 13 Step S332. Refer to the above. Figure 11 It describes the data hunger signal.

[0246] Reference Figure 14 (b) describes the benefits according to the fourth example of this disclosure. Figure 14 (a) describes the situation prior to the application of this disclosure. Figure 14(b) describes the situation after applying the fourth example of this disclosure.

[0247] According to the fourth example of this disclosure, when multiple tensors compete for bandwidth on the system bus, the bus can be configured to allocate relatively higher bandwidth to tensors with relatively higher priority. For example, the bus can allocate higher bandwidth to memory-limited tensors than to computation-limited tensors. Therefore, if low-priority tensors and normal-priority tensors compete on the bus, the bus can be configured to process the normal-priority tensors first. For example, the bus can allocate higher bandwidth to memory-limited tensors than to computation-limited tensors. Therefore, if low-priority tensors and normal-priority tensors compete on the bus, the bus can be configured to process the normal-priority tensors first. For example, if memory-limited tensors and computation-limited tensors compete on the bus, the bus can reorder the ordered queues to process the memory-limited tensors first. Furthermore, since the computation cycle COMP is shorter than the corresponding memory cycle MEM, it can be determined that memory-limited (MB) tensors are insufficient in terms of memory bandwidth. Since the computation cycle COMP is longer than the corresponding memory cycle MEM, it can be determined that computation-limited (CB) tensors are idle in terms of memory bandwidth. For some tensors with increased bandwidth, the duration of a memory cycle can be reduced; conversely, for some tensors with decreased bandwidth, the duration of a memory cycle can be increased.

[0248] Reference Figure 14 (a) and Figure 14 In (b), the memory cycle MEM interval of the first tensor n+1 of NPU0 is identified as a computationally restricted CB interval, and the first controller assigns a low priority L to the memory cycle MEM of the first tensor n+1 based on the data hunger signal IDLE. The memory cycle MEM interval of the first tensor m+1 of NPU1, which is competing with the aforementioned tensor, is identified as a computationally restricted CB interval, and the first controller assigns a low priority L to the memory cycle MEM of the first tensor m+1 based on the data hunger signal IDLE. In this case, since the bus priorities of the memory access operations of the first tensor n+1 of NPU0 and the first tensor m+1 of NPU1 are equal, in (b) of the fourth example according to this disclosure, compared to (a), NPU0 and NPU1 do not mutually yield or receive bus bandwidth, and therefore the memory cycles of NPU0 and NPU1 may not change substantially.

[0249] The memory cycle MEM interval of the second tensor n+2 of NPU0 is identified as a memory-limited MB interval, and the first controller assigns a default priority D to the memory cycle MEM of the second tensor n+2 based on the data hunger signal IDLE, and then assigns a high priority H when the data hunger signal IDLE is activated. The memory cycle MEM interval of the second tensor m+2 of NPU1, which competes with the above tensor, is identified as a computation-limited CB interval, and the first controller assigns a low priority L to the memory cycle MEM of the second tensor m+2 based on the data hunger signal IDLE. The priority of the memory cycle MEM of the second tensor n+2 of NPU0 is the default priority D, while the priority of the memory cycle MEM of the second tensor m+2 of NPU1 is the low priority L. Therefore, in (b) of the fourth example according to this disclosure, compared with (a), NPU0 can yield a predetermined bus bandwidth to NPU1, or the ordered queue of the bus can be reordered according to the priorities of NPU0 and NPU1. In this scenario, the duration of the memory cycle MEM of NPU0's second tensor n+2 decreases, while the duration of the memory cycle MEM of NPU1's second tensor m+2 increases. Furthermore, when the computation cycle COMP of NPU0's first tensor n+1 completes, the data hunger signal IDLE is activated in NPU0, and the priority of the memory cycle MEM of NPU0's second tensor n+2 is changed to high priority H. Therefore, in example (b) according to the fourth example of this disclosure, compared to (a), NPU0 yields a greater amount of bus bandwidth to NPU1. In this case, the duration of the memory cycle MEM of NPU0's second tensor n+2 further decreases, while the duration of the memory cycle MEM of NPU1's second tensor m+2 further increases.

[0250] Therefore, the first data hunger period DS1 of NPU0 in (a) is reduced to the first data hunger period DS1' in (b), and the processing speed of NPU0 is improved. Simultaneously, the duration of the memory cycle MEM of the second tensor m+2 of NPU1 is increased, and the first interval M1 with sufficient bandwidth in (a) is reduced to the first interval M1' with sufficient bandwidth in (b). Even with the increased interval of the memory cycle MEM of the second tensor m+2 of NPU1, the computation time of NPU1 will not be delayed because there is sufficient bandwidth to be freed up. Therefore, freeing up some bus bandwidth allocated to NPU1 for NPU0 has the effect of maintaining the computation speed of NPU1.

[0251] The memory cycle MEM interval of NPU0's third tensor n+3 is identified as a memory-limited MB interval, and the first controller assigns a default priority D to the memory cycle MEMS of the third tensor n+3 based on the data hunger signal IDLE, and then assigns a high priority H when the data hunger signal IDLE is activated. The memory cycle MEM interval of NPU1's third tensor m+3, which competes with the above tensor, is identified as a computation-limited CB interval, and the first controller assigns a low priority L to the memory cycle MEM of the third tensor m+3 based on the data hunger signal IDLE. That is, the priority of the memory cycle MEM of NPU0's third tensor n+3 is the default priority D, while the priority of the memory cycle MEM of NPU1's third tensor m+3 is the low priority L. Therefore, in (b) of the fourth example according to this disclosure, compared with (a), NPU0 can yield a predetermined bus bandwidth to NPU1, or the ordered queue of the bus can be reordered according to the priorities of NPU0 and NPU1. In this scenario, the duration of the memory cycle MEM of the third tensor n+3 of NPU0 is reduced, while the duration of the memory cycle MEM of the third tensor m+3 of NPU1 is increased. Here, when the computation cycle COMP of the second tensor n+2 of NPU0 is completed, the data hunger signal IDLE in NPU0 is activated, and the priority of the memory cycle MEM of the third tensor n+3 of NPU0 is changed to high priority H. Therefore, in (b) of the fourth example according to this disclosure, compared to (a), NPU0 yields a larger amount of bus bandwidth to NPU1. In this case, the interval of the memory cycle MEM of the third tensor n+3 of NPU0 is further reduced, while the interval of the memory cycle MEM of the third tensor m+3 of NPU1 is further increased. Therefore, the second data hunger period DS2 of NPU0 in (a) is reduced to the second data hunger period DS2' in (b). Therefore, the processing speed of NPU0 can be improved. Simultaneously, the duration of the memory cycle MEM of NPU1's third tensor m+3 increases, and the second interval M2 with sufficient bandwidth (a) decreases to a second interval M2' with sufficient bandwidth (b). Even with the increased interval of the memory cycle MEM of NPU1's third tensor m+3, the computation time of NPU1 will not be delayed because there is sufficient bandwidth to make way. Therefore, even if the bus bandwidth allocated to NPU1 is partially given up to NPU0, this has the effect of maintaining the computation speed of NPU1.

[0252] In other words, the system according to the fourth example can assign one of a first priority (e.g., low priority L) or a second priority (e.g., default priority D) to each tensor of the neural network model based on information in the neural network model, and also assign a third priority based on a first signal (e.g., a data starvation signal) generated by the neural processing unit processing the corresponding tensor. The third priority is a higher priority than the first or second priority, and the bus can send tensor data of the third priority before tensor data of the first or second priority. The second priority is a higher priority than the first priority, and the bus can send tensor data of the second priority before tensor data of the first priority. The first and second priorities can be determined based on pre-obtained information, and the third priority can be determined dynamically based on a dynamically generated first signal. Therefore, the system according to the fourth example of this disclosure can be configured to adjust the priority of the bus for a specific interval of each tensor processed in real time to reduce the data starvation period of the neural network model processed on at least one of a plurality of processing cores.

[0253] According to the fourth example, data starvation periods can be reduced more effectively by first comparing the cycles of computational operations and memory access operations of each tensor of the neural processing unit 1000 to identify data starvation periods (e.g., corresponding to the first example), and then further identifying the data starvation state of each processing core and dynamically prioritizing them (e.g., corresponding to the second and third examples).

[0254] According to the above disclosure, based on a control system including multiple neural processing units, a QoS (Quality of Service) can be applied to prioritize read and write operations for accessing the memory of each neural processing unit, so as to achieve efficient operation between read and write operations during DMA operation.

[0255] Furthermore, according to this disclosure, when a data starvation period is predicted because the time required to complete a memory access operation is shorter or longer than the time required to complete a computation operation, the bus bandwidth can be reallocated according to the priority-ordered QoS so that the computing circuit can operate without a data starvation period, thereby improving data processing performance and reducing power consumption.

[0256] Furthermore, according to this disclosure, by applying a high priority to NPU0 and a low priority to NPU1 at a certain point in time, the bus bandwidth of NPU1 can be yielded to NPU0 to reduce the duration of data starvation.

[0257] According to one example of this disclosure, a system for controlling a processing core can be provided. The system may include: at least one processing core for processing computations of at least one neural network model containing at least one tensor; at least one memory circuit configured to store at least one tensor; a bus circuit electrically coupled to the at least one processing core and the at least one memory circuit for transmitting at least one tensor based on memory access operation instructions; and a controller that controls the priority of memory access operations for each tensor of the at least one processing core.

[0258] The bus circuit can be configured to transfer at least one tensor from at least one memory to at least one processing core.

[0259] The controller can be configured to determine the memory cycles and computation cycles of at least one tensor allocated to at least one processing core.

[0260] The controller can be configured to compare the computation cycle of at least one tensor with the memory cycle of a subsequent tensor of at least one tensor.

[0261] At least one processing core may include a first processing core and a second processing core. The controller may be configured to increase the bus bandwidth of the first processing core by reducing the bus bandwidth of the second processing core when a data starvation occurs in the first processing core.

[0262] At least one processing core may include a first processing core and a second processing core. The controller may be configured to increase the priority of tensors of the first processing core in order to increase the bus bandwidth of tensors assigned to the first processing core for processing when a data starvation occurs on the first processing core.

[0263] The controller can be configured to control the bus circuitry to relinquish at least a portion of the bus bandwidth allocated to at least one processing core when a tensor being processed by at least one processing core is within a computationally constrained interval.

[0264] The controller can be configured to determine the priority order of at least one processing core based on data hunger signals.

[0265] The system may include a counter for determining a data starvation state. The system can be configured to determine a data starvation state by comparing the counter's value to a preset counter threshold.

[0266] At least one processing core may include multiple processing elements (PEs). The multiple PEs may include at least one of the multiplication and accumulation (MAC) operator, addition tree, or arithmetic logic unit (ALU) operator.

[0267] According to one example of this disclosure, a method for controlling a processing core processed by a device can be provided. The method may include: determining a data hunger period for at least one tensor based on the computation cycle of at least one tensor processed by at least one processing core and the memory cycle of subsequent tensors; and controlling the priority of memory access operations of at least one tensor on a bus based on the data hunger period.

[0268] Computational operations and memory access operations on at least one tensor can be performed based on the bus bandwidth provided for communication between at least one processing core and memory.

[0269] Determining data starvation periods may include: determining the memory cycle of at least one tensor; and determining the computation cycle of subsequent tensors.

[0270] Priorities on the control bus can include adjusting the bus bandwidth for memory cycles.

[0271] At least one processing core may include a first processing core and a second processing core. Priority on the control bus may include relinquishing at least a portion of the bus bandwidth allocated to the first processing core to the second processing core.

[0272] Priority control may include prioritizing at least one tensor based on a data hunger signal.

[0273] Determining data hunger can be configured by comparing the counter value of a counter with a preset counter threshold.

[0274] The priority ordering of memory access operations for at least one tensor on the control bus can be configured to increase the bus bandwidth of the memory-limited tensor.

[0275] Prioritization of the control bus may include prioritizing memory access operations on at least one tensor. Prioritization may include first to third priorities. A second priority may be a higher priority than the first priority. A third priority may be a higher priority than the first and second priorities.

[0276] According to one example of this disclosure, a system may be provided. The system may include: at least one neural processing unit configured to process computations of at least one neural network model comprising at least one tensor; at least one memory circuit configured to store at least one tensor; a bus circuit electrically coupled to the at least one neural processing unit and the at least one memory circuit, configured to transfer at least one tensor based on memory access operation instructions; and a controller configured to control the priority of memory access operations for each tensor of the at least one neural processing unit.

[0277] The examples of this disclosure disclosed herein and in the accompanying drawings are for illustrative purposes only and to facilitate understanding of the technical content of this disclosure, and are not intended to limit the scope of this disclosure.

[0278] [National R&D projects supporting this invention]

[0279] [Project Identifier] 1711193247

[0280] [Task ID] 2022-0-00248-002

[0281] [Department Name] Department of Science and Information Technology

[0282] [Name of Task Management (Specialized) Institution] Institute of Information and Communication Technology Planning and Evaluation

[0283] [Research Project Title] Development (Design) of PIM Artificial Intelligence Semiconductor Core Technologies

[0284] [Research Task Title] Development of CXL-based Multi-DRAM Module PIM Semiconductor Technology Considering Memory Coherence

[0285] [Name of the organization performing the task] DeepX Ltd.

[0286] [Research Period] January 1, 2023 – December 31, 2023

Claims

1. A system comprising: At least one processing core configured to perform computational operations on at least one neural network model associated with the tensor; at least one memory circuit configured to store the tensor; A bus circuit operatively coupled to the at least one processing core and the at least one memory circuit, the bus circuit being configured to send the tensor from the at least one memory circuit to the at least one processing core in response to receiving a request for a memory access operation; as well as A controller, operably coupled to the bus circuitry, is configured to determine the priority of memory access operations for each of the tensors and control the bus circuitry according to the determined priority to send each of the tensors to the at least one processing core.

2. The system of claim 1, wherein the controller is configured to determine the priority of the memory access operation of each of the tensors by comparing the memory cycle length using the bus circuit and the computation cycle length of the tensor.

3. The system of claim 2, wherein the controller is further configured to control one or more quality of service parameters associated with the bus circuitry based on a determined priority of the memory access operation for each of the tensors.

4. The system of claim 1, wherein the controller is configured to determine the priority of the memory access operation of each of the tensors by comparing at least the computation cycle of a first tensor in the tensor at the processing core with the memory cycle of a subsequent second tensor in the tensor.

5. The system of claim 1, wherein the controller is configured to increase the bus bandwidth allocated to the first processing core by reducing the bus bandwidth allocated to the second processing core of the at least one processing core in response to determining that a data starvation period is predicted to occur or has occurred in the first processing core of the at least one processing core.

6. The system of claim 1, wherein the controller is configured to, in response to determining that a data starvation period is predicted to occur or has occurred in the first processing core, increase the priority of sending tensors of the first processing core to the at least one processing core, thereby increasing the bus bandwidth allocated to the first processing core.

7. The system of claim 1, wherein the controller is configured to control the bus circuit to yield at least a portion of the bandwidth of the bus circuit assigned to the at least one processing core in response to determining that the at least one processing core is in a compute-limited state.

8. The system of claim 1, wherein the controller is configured to determine the priority of sending tensors to the at least one processing core via the bus circuit in response to receiving a data hunger signal.

9. The system of claim 1, further comprising a counter for determining a data starvation state, wherein determining whether the at least one processing core is in the data starvation state is done by comparing the counter value of the counter with a preset counter threshold.

10. The system of claim 1, wherein each of the at least one processing core comprises a plurality of processing elements (PEs), wherein the plurality of PEs includes at least one of a multiplication and accumulation (MAC) operator, an addition tree, or an arithmetic logic unit (ALU) operator.

11. A method comprising: Determine whether data starvation has occurred or is predicted to occur in at least one processing core, the processing core being configured to perform computational operations on at least one neural network model associated with a tensor; In response to determining whether the data starvation has occurred or is predicted to occur in the at least one processing core, the priority of memory access operations for each of the tensors via the bus circuitry is determined. Based on the determined priority, bandwidth is allocated for sending each of the tensors to the bus circuitry of the at least one processing core; as well as Each tensor is sent to the at least one processing core via the bus circuit according to the bandwidth allocated to the bus circuit that sends each tensor to the at least one processing core.

12. The method of claim 11, wherein determining whether the data starvation has occurred or is predicted to occur involves comparing a computation cycle in at least one processing core containing a first tensor to be processed by the at least one processing core with a memory cycle for sending a second tensor to the at least one processing core via a bus circuit, the second tensor being processed by the at least one processing core subsequently after the first tensor.

13. The method of claim 11, wherein the allocation of bandwidth includes controlling one or more quality of service parameters associated with the bus circuitry based on a determined priority of the memory access operation for each of the tensors.

14. The method of claim 11, wherein determining whether the data starvation has occurred or may occur comprises determining a memory cycle of a first tensor in the tensor and determining a computation cycle of a second tensor in the tensor that follows the first tensor.

15. The method of claim 11, wherein the allocation of bandwidth comprises reallocating at least a portion of the bus bandwidth assigned to a first processing core of the at least one processing core to a second processing core of the at least one processing core.

16. The method of claim 11, wherein determining the priority comprises prioritizing the transmission of at least one of the tensors based on a data hunger signal indicating that a processing element in the at least one processing core is idle.

17. The method of claim 11, wherein determining whether the data starvation has occurred or may occur comprises comparing the counter value of a counter with a preset counter threshold.

18. The method of claim 11, wherein allocating the bandwidth of the bus circuitry comprises increasing the bus bandwidth to transmit tensors that are memory-limited by the at least one processing core.

19. The method of claim 11, wherein the priority of the memory operation of each tensor is one of a first priority, a second priority which is higher than the first priority, or a third priority which is higher than the second priority.

20. A system comprising: At least one neural processing unit is configured to perform computational operations on at least one neural network model associated with a tensor; At least one memory circuit is configured to store the tensor; A bus circuit operatively coupled to the at least one neural processing unit and the at least one memory circuit, the bus circuit being configured to send the tensor from the at least one memory circuit to the at least one neural processing unit in response to receiving a request for a memory access operation; as well as A controller, operably coupled to the bus circuitry, is configured to determine the priority of memory access operations for each of the tensors and, based on the determined priority, control the bus circuitry to send each of the tensors to the at least one neural processing unit.

Images