Neural processing device

Abstract

The present invention discloses a neural processing device. The neural processing device comprises a processing unit that performs operations, an L0 memory that receives data from the processing unit and provides data to the processing unit, and a Load / Store Unit (LSU) that performs load and store operations of the data. The LSU comprises a neural core load unit that issues a load instruction for the data, a neural core store unit that issues a store instruction for transmitting and storing the data, and a sink ID logic that provides a sink ID to the neural core load unit and the neural core store unit to generate a synchronization signal for each sink ID.

Description

Technology Field

[0001] The present invention relates to a neural processing device. Specifically, the present invention relates to a neural processing device that minimizes waiting time for instruction issues. Background Technology

[0003] Over the past few years, Artificial Intelligence (AI) has been cited globally as the most promising technology and a core component of the Fourth Industrial Revolution. The biggest challenge facing this AI technology is computing performance. For AI technology to realize human learning, reasoning, perception, and natural language processing capabilities, rapidly processing large amounts of data is paramount.

[0004] Although the central processing unit (CPU) or graphics processing unit (GPU) of conventional computers was used for deep learning training and inference in early artificial intelligence, there are limitations to deep learning training and inference tasks that have high workloads, so neural processing units (NPUs) that are structurally specialized for deep learning tasks are gaining attention.

[0005] A synchronization signal refers to a signal that indicates that each hardware component has completed an operation and can perform the next operation. The transmission of such synchronization signals must be performed sequentially, and since the next operation can only be executed when one operation is completed, waiting time may inevitably occur.

[0006] Therefore, it is time to consider various measures to minimize such waiting times. Prior art literature

[0008] Registered Patent Publication No. 10-2258566 The problem to be solved

[0009] The objective of the present invention is to provide a neural processing device that groups load and store instructions and performs them in parallel.

[0010] The objects of the present invention are not limited to those mentioned above, and other unmentioned objects and advantages of the present invention may be understood from the following description and will be more clearly understood by the embodiments of the present invention. Furthermore, it will be readily apparent that the objects and advantages of the present invention can be realized by the means and combinations thereof set forth in the claims. means of solving the problem

[0012] A neural processing device according to some embodiments of the present invention for solving the above problem comprises a processing unit that performs operations, an L0 memory that receives data from the processing unit and provides data to the processing unit, and a Load / Store Unit (LSU) that performs load and store operations of the data. The LSU includes a neural core load unit that issues a load instruction for the data, a neural core store unit that issues a store instruction for transmitting and storing the data, and a sink ID logic that provides a sink ID to the neural core load unit and the neural core store unit to generate a synchronization signal for each sink ID.

[0013] Additionally, the instruction includes the load instruction and the store instruction, and the instruction includes at least one group, and each of the at least one group may include at least one operation instruction and a sync signal indicating the completion of an issue of the operation instruction.

[0014] Additionally, the sink ID logic may include a current sink ID register that stores the sink IDs of the groups of the instructions currently being issued, a sink ID MO counter that counts MOs according to the number of issues of the operation instructions and interconnection returns for the operation instructions for each sink ID, and a sink queue that stores the sink signals for each group and sequentially outputs the sink signals when the MO becomes 0.

[0015] Additionally, the above at least one group includes a first group and a second group, and the sink ID MO counter may include a first MO counter for counting the MO of the first group and a second MO counter for counting the MO of the second group.

[0016] In addition, the first MO counter can be increased by 1 each time an operation instruction of the first group is issued.

[0017] In addition, the first MO counter may decrease by 1 each time an interconnection return for the operation instruction of the first group is received.

[0018] Additionally, the sink queue can store the sink signal of the first group when it is issued, store it sequentially following the sink signal of the first group when it is issued, output the sink signal of the first group when the first MO counter becomes 0, and output the sink signal of the second group when the second MO counter becomes 0.

[0019] In addition, the sink queue can sequentially output the sink signal of the first group and the sink signal of the second group.

[0020] Additionally, the above at least one group further includes a third group, the first group and the second group have dependencies on each other, and the third group may be independent of the first and second groups.

[0021] Additionally, the first time between the issue end time of the operation instruction of the first group and the issue start time of the operation instruction of the second group may be the same as the second time between the issue end time of the operation instruction of the second group and the issue start time of the operation instruction of the third group.

[0022] In addition, the above sink queue may be a FIFO (First In First Out) structure.

[0023] In addition, the current sink ID register can change and store the sink ID of the group corresponding to the sink signal after a preset fixed cycle when the sink signal is issued.

[0024] Additionally, it further includes a local interconnection that receives and transmits data from the LSU, and when the instruction is issued, an interconnection request according to the instruction is transmitted to the local interconnection, and the time difference between the transmission of the interconnection request and the issuance of the instruction may be a predetermined fixed cycle.

[0025] Additionally, the local interconnection transmits an interconnection return corresponding to the interconnection request to the LSU, and the timing of receiving the interconnection return and transmitting the interconnection request may differ by an unspecified, non-fixed cycle.

[0026] A neural processing device according to some embodiments of the present invention for solving the above problem comprises at least one neural core, an L2 shared memory shared by the at least one neural core, an L2 interconnection for transmitting data between the at least one neural core and the L2 shared memory, and an L2 sync path for transmitting a synchronization signal between the at least one neural core and the L2 shared memory, wherein each of the at least one neural core comprises a processing unit for performing operations, an L0 memory for storing data input to and output to the processing unit, and an LSU for generating the synchronization signal by performing load and / or store operations of the data, wherein the LSU issues a plurality of groups of operation instructions, and a first time between the time points of issuance of the plurality of operation instructions is the same.

[0027] In addition, the size of the first time can be pre-set.

[0028] In addition, the above instructions may include load instructions and store instructions.

[0029] Additionally, the plurality of groups may include at least one operation instruction and a sync signal indicating that the issue of the operation instruction has ended.

[0030] In addition, the above LSU may include a sink ID logic that provides a sink ID for each of the plurality of groups.

[0031] Additionally, the above sink ID logic may include an MO counter that counts MOs by group according to the issue of the operation instruction and the interconnection return for the operation instruction. Effects of the invention

[0033] The neural processing device of the present invention can issue and transmit synchronization signals without waiting time, thereby maximizing the speed of the device.

[0034] In addition, through group synchronization, instructions can be executed in a pre-set order without waiting time.

[0035] In addition to the above, the specific effects of the present invention are described together with the specific details for implementing the invention below. Brief explanation of the drawing

[0037] FIG. 1 is a block diagram illustrating a neural processing system according to some embodiments of the present invention. Figure 2 is a block diagram for explaining the neural processing unit of Figure 1 in detail. Figure 3 is a block diagram for explaining the neural core SoC of Figure 2 in detail. Figure 4 is a structural diagram for explaining the global interconnection of Figure 3 in detail. Figure 5 is a block diagram for explaining the neural processor of Figure 3 in detail. FIG. 6 is a diagram illustrating the hierarchical structure of a neural processing device according to some embodiments of the present invention. Figure 7 is a block diagram for explaining the neural core of Figure 5 in detail. Figure 8 is a block diagram for explaining the LSU of Figure 7 in detail. FIG. 9 is a block diagram for explaining the sync ID logic of FIG. 8 in detail. Figure 10 is a block diagram for explaining in detail the structure of the sink ID MO counter of Figure 9. FIG. 11 is a diagram illustrating a load instruction stream of a neural processing device according to some embodiments of the present invention. Figure 12 is a time diagram to explain the operation of blocking a load sink signal. FIG. 13 is a time diagram for explaining the operation of non-blocking a load synchronization signal of a neural processing device according to some embodiments of the present invention. FIG. 14 is a diagram illustrating a store instruction stream of a neural processing device according to some embodiments of the present invention. FIG. 15 is a time diagram for explaining the operation of non-blocking a store synchronization signal of a neural processing device according to some embodiments of the present invention. Figure 16 is a conceptual diagram illustrating the waiting time for an in-order execution pipeline. FIG. 17 is a conceptual diagram illustrating the waiting time of an in-order execution pipeline of a neural processing device according to some embodiments of the present invention. Figure 18 is a conceptual diagram illustrating the waiting time for an out-of-order execution pipeline. FIG. 19 is a conceptual diagram illustrating the waiting time of an out-of-order execution pipeline of a neural processing device according to some embodiments of the present invention. FIG. 20 is a block diagram for explaining the processing unit of FIG. 7 in detail. FIG. 21 is a block diagram for explaining the L0 memory of FIG. 20 in detail. FIG. 22 is a block diagram for explaining the local memory bank of FIG. 21 in detail. FIG. 23 is a block diagram for explaining in detail the structure of the neural processing device of FIG. 1. FIG. 24 is a block diagram illustrating the memory reconfiguration of the neural processing system of FIG. 1. FIG. 25 is a block diagram showing an example of memory reorganization of the neural processing system of FIG. 1. FIG. 26 is an enlarged block diagram of part A of FIG. 24. FIG. 27 is a drawing for explaining the first memory bank of FIG. 26 in detail. FIG. 28 is a block diagram illustrating the software layer structure of the neural processing device of FIG. 1. FIG. 29 is a conceptual diagram for explaining deep learning operations performed by the neural processing unit of FIG. 1. FIG. 30 is a conceptual diagram illustrating the learning and inference operations of the neural network of the neural processing device of FIG. 1. FIG. 31 is a flowchart illustrating the operation of an in-order execution pipeline of a neural processing device according to some embodiments of the present invention. FIG. 32 is a flowchart illustrating the operation of an out-of-order execution pipeline of a neural processing device according to some embodiments of the present invention. Specific details for implementing the invention

[0038] Terms and words used in this specification and claims shall not be interpreted as being limited to their general or dictionary meanings. In accordance with the principle that an inventor may define the concept of a term or word to best describe their invention, they shall be interpreted in a meaning and concept consistent with the technical spirit of the invention. Furthermore, since the embodiments described in this specification and the configurations illustrated in the drawings are merely one embodiment of the invention and do not represent the entire technical spirit of the invention, it should be understood that various equivalents, modifications, and applicable examples capable of replacing them may exist at the time of filing this application.

[0039] The terms first, second, A, B, etc., as used in this specification and claims may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0040] The terms used in this specification and claims are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" should be understood as not precluding the existence or addition of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification.

[0041] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which this invention pertains.

[0042] Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0043] In addition, each component, process, procedure, or method included in each embodiment of the present invention may be shared within a scope that is not technically contradictory to one another.

[0045] Hereinafter, a neural processing apparatus according to several embodiments of the present invention will be described with reference to FIGS. 1 to 30.

[0046] FIG. 1 is a block diagram illustrating a neural processing system according to some embodiments of the present invention.

[0047] Referring to FIG. 1, a neural processing system (NPS) according to some embodiments of the present invention may include a first neural processing device (1), a second neural processing device (2), and an external interface (3).

[0048] The first neural processing device (1) may be a device that performs calculations using an artificial neural network. The first neural processing device (1) may be, for example, a device specialized in performing deep learning calculation tasks. However, the present embodiment is not limited thereto.

[0049] The second neural processing device (2) may be a device having the same or similar configuration as the first neural processing device (1). The first neural processing device (1) and the second neural processing device (2) may be connected to each other through an external interface (3) to share data and control signals.

[0050] Although two neural processing devices are illustrated in FIG. 1, the neural processing system (NPS) according to some embodiments of the present invention is not limited thereto. That is, the neural processing system (NPS) according to some embodiments of the present invention may have three or more neural processing devices connected to each other through an external interface (3). Alternatively, the neural processing system (NPS) according to some embodiments of the present invention may include only one neural processing device.

[0051] Figure 2 is a block diagram for explaining the neural processing unit of Figure 1 in detail.

[0052] Referring to FIG. 2, the first neural processing device (1) may include a neural core SoC (10), a CPU (20), an off-chip memory (30), a first non-volatile memory interface (40), a first volatile memory interface (50), a second non-volatile memory interface (60), and a second volatile memory interface (70).

[0053] The neural core SoC (10) may be a System on Chip device. The neural core SoC (10) may be an accelerator as an artificial intelligence computing unit. The neural core SoC (10) may be, for example, any one of a GPU (graphics processing unit), an FPGA (field programmable gate array), and an ASIC (application-specific integrated circuit). However, the present embodiment is not limited thereto.

[0054] The neural core SoC (10) can exchange data with other external computation units through an external interface (3). Additionally, the neural core SoC (10) can be connected to a non-volatile memory (31) and a volatile memory (32), respectively, through a first non-volatile memory interface (40) and a first volatile memory interface (50).

[0055] The CPU (20) may be a control unit that controls the system of the first neural processing unit (1) and executes program operations. As a general-purpose computing unit, the CPU (20) may have low efficiency in performing parallel simple operations that are frequently used in deep learning. Therefore, the neural core SoC (10) can have high efficiency by performing operations on deep learning inference and learning tasks.

[0056] The CPU (20) can exchange data with other external computing units through the external interface (3). Additionally, the CPU (20) can be connected to the non-volatile memory (31) and the volatile memory (32), respectively, through the second non-volatile memory interface (60) and the second volatile memory interface (70).

[0057] The off-chip memory (30) may be memory placed outside the chip of the neural core SoC (10). The off-chip memory (30) may include non-volatile memory (31) and volatile memory (32).

[0058] Non-volatile memory (31) may be a memory that retains stored information even when power is not supplied. Non-volatile memory (31) is, for example, ROM (Read-Only Memory), PROM (Programmable Read-Only Memory), EAROM (Erasable Alterable ROM), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory) (e.g., NAND Flash memory, NOR Flash memory), UVEPROM (Ultra-Violet Erasable Programmable Read-Only Memory), FeRAM (Ferroelectric Random Access Memory), MRAM (Magnetoresistive Random Access Memory), PRAM (Phase-change Random Access Memory), SONOS (silicon-oxide-nitride-oxide-silicon), RRAM (Resistive Random Access Memory), NRAM (Nanotube Random Access Memory), magnetic computer memory devices (e.g., hard disk, floppy disk drive, magnetic tape), optical disc drive and 3D crosspoint It may include at least one of a memory (3D XPoint memory). However, the present embodiment is not limited thereto.

[0059] Volatile memory (32), unlike non-volatile memory (31), may be a memory that continuously requires power to maintain stored information. Volatile memory (32) may include, for example, at least one of DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory), SDRAM (Synchronous Dynamic Random Access Memory), and DDR SDRAM (Double Data Rate SDRAM). However, the present embodiment is not limited thereto.

[0060] The first non-volatile memory interface (40) and the second non-volatile memory interface (60) may each include, for example, at least one of PATA (Parallel Advanced Technology Attachment), SCSI (Small Computer System Interface), SAS (Serial Attached SCSI), SATA (Serial Advanced Technology Attachment), and PCIe (PCI Express). However, the present embodiment is not limited thereto.

[0061] The first volatile memory interface (50) and the second volatile memory interface (70) may each be, for example, at least one of SDR (Single Data Rate), DDR (Double Data Rate), QDR (Quad Data Rate), and XDR (eXtreme Data Rate, Octal Data Rate). However, the present embodiment is not limited thereto.

[0062] Figure 3 is a block diagram for explaining the neural core SoC of Figure 2 in detail.

[0063] Referring to FIGS. 2 and 3, the neural core SoC (10) may include at least one neural processor (1000), shared memory (2000), Direct Memory Access (DMA) (3000), a non-volatile memory controller (4000), a volatile memory controller (5000), and a global interconnection (5000).

[0064] A neural processor (1000) may be a computational unit that directly performs computational tasks. If there are multiple neural processors (1000), computational tasks may be assigned to each neural processor (1000). Each neural processor (1000) may be connected to each other through a global interconnection (5000).

[0065] The shared memory (2000) may be a memory shared by multiple neural processors (1000). The shared memory (2000) may store data from each neural processor (1000). Additionally, the shared memory (2000) may receive data from the off-chip memory (30), temporarily store it, and transfer it to each neural processor (1000). Conversely, the shared memory (2000) may receive data from the neural processor (1000), temporarily store it, and transfer it to the off-chip memory (30) of FIG. 2.

[0066] The shared memory (2000) may require a relatively fast memory. Accordingly, the shared memory (2000) may include, for example, SRAM. However, the present embodiment is not limited thereto. That is, the shared memory (2000) may include DRAM.

[0067] The shared memory (2000) may be a memory corresponding to the SoC level, i.e., L3 (level 3). Therefore, the shared memory (2000) may be defined as an L3 shared memory.

[0068] The DMA (3000) can directly control the movement of data without the need for the neural processor (1000) to control the input and output of data. Accordingly, the DMA (3000) can control the movement of data between memories to minimize the number of interrupts of the neural processor (1000).

[0069] The DMA (3000) can control data movement between the shared memory (2000) and the off-chip memory (30). Through the authority of the DMA (3000), the non-volatile memory controller (4000) and the volatile memory controller (5000) can perform data movement.

[0070] The non-volatile memory controller (4000) can control read or write operations on the non-volatile memory (31). The non-volatile memory controller (4000) can control the non-volatile memory (31) through the first non-volatile memory interface (40).

[0071] The volatile memory controller (5000) can control read or write operations on the volatile memory (32). Additionally, the volatile memory controller (5000) can perform refresh operations on the volatile memory (32). The volatile memory controller (5000) can control the non-volatile memory (31) through the first volatile memory interface (50).

[0072] A global interconnection (5000) can connect at least one neural processor (1000), shared memory (2000), DMA (3000), non-volatile memory controller (4000), and volatile memory controller (5000) to each other. Additionally, an external interface (3) can also be connected to the global interconnection (5000). The global interconnection (5000) may be a path through which data travels between at least one neural processor (1000), shared memory (2000), DMA (3000), non-volatile memory controller (4000), volatile memory controller (5000), and the external interface (3).

[0073] The global interconnection (5000) can transmit signals for the transmission and synchronization of control signals as well as data. That is, in some embodiments of the present invention, the neural processing device does not have a separate control processor managing the synchronization signals, but rather each neural processor (1000) can directly transmit and receive the synchronization signals. Accordingly, the latency of the synchronization signals generated by the control processor can be blocked.

[0074] That is, when there are multiple neural processors (1000), there may be dependencies of individual tasks such that the next neural processor (1000) can start a new task only after the task of one neural processor (1000) has been completed. The completion and start of these individual tasks can be verified through a synchronization signal, and in conventional technology, the control processor performed the reception of such synchronization signal and the instruction to start a new task.

[0075] However, as the number of neural processors (1000) increases and the dependencies of the tasks become more complex, the number of requests and instructions for these synchronization tasks increases exponentially. Consequently, the latency associated with each request and instruction can significantly reduce the efficiency of the task.

[0076] Accordingly, in some embodiments of the present invention, the neural processing device may have each neural processor (1000) directly transmit a synchronization signal to another neural processor (1000) according to the dependency of the task, instead of a control processor. In this case, compared to a method managed by a control processor, multiple neural processors (1000) can perform synchronization tasks in parallel, thereby minimizing the latency associated with synchronization.

[0077] In addition, the control processor must perform task scheduling of neural processors (1000) according to task dependencies, and the overhead of such scheduling can increase significantly as the number of neural processors (1000) increases. Therefore, in some embodiments of the present invention, the neural processing device can have the scheduling task performed by individual neural processors (1000), and thus the scheduling burden is eliminated, thereby improving the performance of the device.

[0078] Figure 4 is a structural diagram for explaining the global interconnection of Figure 3 in detail.

[0079] Referring to FIG. 4, the global interconnection (5000) may include a data channel (5100), a control channel (5200), and an L3 sink channel (5300).

[0080] The data channel (5100) may be a dedicated channel for transmitting data. Through the data channel (5100), at least one neural processor (1000), shared memory (2000), DMA (3000), non-volatile memory controller (4000), volatile memory controller (5000), and external interface (3) can exchange data with each other.

[0081] The control channel (5200) may be a dedicated channel for transmitting control signals. Through the control channel (5200), at least one neural processor (1000), shared memory (2000), DMA (3000), non-volatile memory controller (4000), volatile memory controller (5000), and external interface (3) can exchange control signals with each other.

[0082] The L3 sink channel (5300) may be a dedicated channel for transmitting synchronization signals. Through the L3 sink channel (5300), at least one neural processor (1000), shared memory (2000), DMA (3000), non-volatile memory controller (4000), volatile memory controller (5000), and external interface (3) can exchange synchronization signals with each other.

[0083] The L3 sink channel (5300) is set as a dedicated channel within the global interconnection (5000) so that it can rapidly transmit synchronization signals without overlapping with other channels. Accordingly, the neural processing device according to some embodiments of the present invention does not require new wiring work and can smoothly perform synchronization work using the existing global interconnection (5000).

[0084] Figure 5 is a block diagram for explaining the neural processor of Figure 3 in detail.

[0085] Referring to FIGS. 3 to 5, the neural processor (1000) may include at least one neural core (100), an L2 shared memory (400), a local interconnection (200), and an L2 sink path (300).

[0086] At least one neural core (100) can share and perform the work of the neural processor (1000). For example, there may be eight neural cores (100). However, the present embodiment is not limited thereto. Although FIGS. 3 and 5 show that multiple neural cores (100) are included in the neural processor (1000), the present embodiment is not limited thereto. That is, the neural processor (1000) can be configured with only one neural core (100).

[0087] The L2 shared memory (400) may be a memory shared by each neural core (100) within the neural processor (1000). The L2 shared memory (400) may store data of each neural core (100). Additionally, the L2 shared memory (400) may receive data from the shared memory (2000) of FIG. 4, temporarily store it, and transfer it to each neural core (100). Conversely, the L2 shared memory (400) may receive data from the neural core (100), temporarily store it, and transfer it to the shared memory (2000) of FIG. 3.

[0088] L2 shared memory (400) may be memory corresponding to the neural processor level, i.e., L2 (level 2). L3 shared memory, i.e., shared memory (2000), may be shared by the neural processor (1000), and L2 shared memory (400) may be shared by the neural core (100).

[0089] A local interconnection (200) can connect at least one neural core (100) and an L2 shared memory (400) to each other. The local interconnection (200) may be a path for data to travel between at least one neural core (100) and an L2 shared memory (400). The local interconnection (200) can be connected to the global interconnection (5000) of FIG. 3 to transmit data.

[0090] The L2 sink path (300) can connect at least one neural core (100) and an L2 shared memory (400) to each other. The L2 sink path (300) may be a path through which a synchronization signal of at least one neural core (100) and an L2 shared memory (400) travels.

[0091] The L2 sync path (300) can be physically formed separately from the local interconnection (200). Unlike the global interconnection (5000), the local interconnection (200) may not have sufficient internal channels. In such cases, the L2 sync path (300) can be formed separately to perform the transmission of synchronization signals quickly and without delay. The L2 sync path (300) can be used for synchronization performed at a level one step lower than the L3 sync channel (5300) of the global interconnection (5000).

[0092] FIG. 6 is a diagram illustrating the hierarchical structure of a neural processing device according to some embodiments of the present invention.

[0093] Referring to FIG. 6, the neural core SoC (10) may include at least one neural processor (1000). Each neural processor (1000) can transmit data to each other through a global interconnection (5000).

[0094] Each neural processor (1000) may include at least one neural core (100). The neural core (100) may be a processing unit optimized for deep learning computation tasks. The neural core (100) may be a processing unit corresponding to one operation of a deep learning computation task. That is, a deep learning computation task may be expressed as a sequential or parallel combination of multiple operations. Each neural core (100) may be a processing unit capable of processing one operation, and may be the minimum computation unit that can be considered for scheduling from the perspective of a compiler.

[0095] The neural processing device according to the present embodiment can facilitate fast and efficient scheduling and computational operations by configuring the minimum computational unit considered from the perspective of compiler scheduling and the hardware processing unit to have the same scale.

[0096] In other words, if the hardware's divisible processing unit is excessively large compared to the computational task, inefficiency in the operation of the processing unit may occur. Conversely, scheduling a processing unit smaller than an operation, which is the minimum scheduling unit of a compiler, every time is not appropriate because it can lead to scheduling inefficiency and increase hardware design costs.

[0097] Therefore, this embodiment can simultaneously achieve fast scheduling of computational tasks and efficient execution of computational tasks without wasting hardware resources by coordinating the scale of the compiler's scheduling unit and the hardware processing unit to be similar.

[0098] Figure 7 is a block diagram for explaining the neural core of Figure 5 in detail.

[0099] Referring to FIG. 7, the neural core (100) may include an LSU (Load / Store Unit) (110), an L0 memory (120), a weight buffer (130), an activation LSU (140), an activation buffer (150), and a processing unit (160).

[0100] The LSU (110) can receive at least one of data, control signals, and synchronization signals from the outside through the local interconnection (200) and the L2 sink path (300). The LSU (110) can transmit at least one of the received data, control signals, and synchronization signals to the L0 memory (120). Similarly, the LSU (110) can transmit at least one of the data, control signals, and synchronization signals to the outside through the local interconnection (200) and the L2 sink path (300).

[0101] Figure 8 is a block diagram for explaining the LSU of Figure 7 in detail.

[0102] Referring to FIG. 8, the LSU (110) may include a local memory load unit (111a), a local memory store unit (111b), a neural core load unit (112a), a neural core store unit (112b), a load buffer (LB), a store buffer (SB), a load engine (113a), a store engine (113b), a conversion index buffer (114), and a sink ID logic (SID_L).

[0103] The local memory load unit (111a) can fetch a load instruction for L0 memory (120) and issue a load instruction. When the local memory load unit (111a) provides the issued load instruction to the load buffer (LB), the load buffer (LB) can sequentially send memory access requests to the load engine (113a) according to the order in which they were input.

[0104] Additionally, the local memory store unit (111b) can fetch store instructions for L0 memory (120) and issue store instructions. When the local memory store unit (111b) provides the issued store instructions to the store buffer (SB), the store buffer (SB) can sequentially send memory access requests to the store engine (113b) according to the order in which they were input.

[0105] The neural core load unit (112a) can fetch load instructions for the neural core (100) and issue load instructions. When the neural core load unit (112a) provides the issued load instructions to the load buffer (LB), the load buffer (LB) can sequentially send memory access requests to the load engine (113a) according to the order in which they were input.

[0106] Additionally, the neural core store unit (112b) can fetch store instructions for the neural core (100) and issue store instructions. When the neural core store unit (112b) provides the issued store instructions to the store buffer (SB), the store buffer (SB) can sequentially send memory access requests to the store engine (113b) according to the order in which they were input.

[0107] The load engine (113a) can receive a memory access request and retrieve data through the local interconnection (200). At this time, the load engine (113a) can quickly find data by using the translation table of recently used virtual and physical addresses in the translation index buffer (114). If the virtual address of the load engine (113a) is not in the translation index buffer (114), address translation information can be found in another memory.

[0108] The store engine (113b) can receive a memory access request and retrieve data through the local interconnection (200). At this time, the store engine (113b) can quickly find the data by using the translation table of recently used virtual and physical addresses in the translation index buffer (114). If the virtual address of the store engine (113b) is not in the translation index buffer (114), the address translation information can be found in another memory.

[0109] The load engine (113a) and the store engine (113b) can send a synchronization signal to the L2 sink path (300). At this time, the synchronization signal may indicate that the operation has been completed.

[0110] The sync ID logic (SID_L) may provide a sync ID to the neural core load unit (112a) and the neural core store unit (112b). The sync ID logic (SID_L) may not provide a sync ID to the local memory load unit (111a) and the local memory store unit (111b). However, the present embodiment is not limited thereto. That is, the sync ID logic (SID_L) may also provide a sync ID to the local memory load unit (111a) and the local memory store unit (111b).

[0111] The sink ID can be an ID for group identification of load instructions and store instructions. That is, load instructions and store instructions can be identified by group based on the sink ID.

[0112] FIG. 9 is a block diagram for explaining the sync ID logic of FIG. 8 in detail.

[0113] Referring to FIG. 9, the sink ID logic (SID_L) may include a current sink ID register (CSID), a sink ID MO counter (SMO), and a sink queue (SQ).

[0114] The Current Sink ID Register (CSID) can store the sink ID of the issued instruction. In this case, the instruction may include load instructions and store instructions.

[0115] An instruction may be in the form of a series of streams. Accordingly, an instruction may be defined as an instruction stream. An instruction stream may include at least one group. One group may include operation instructions and sink signals for related data. In this case, the operation instructions may include load operation instructions for load operations and store operation instructions for store operations.

[0116] An operation instruction may be an instruction for a load operation or a store operation on data. There may be a single operation instruction, but there may also be multiple. That is, it may include various data load or store operations to be performed sequentially for the required operation.

[0117] A sink signal can be a signal indicating that a group has terminated. That is, multiple groups in an instruction stream may issue a sink signal at the end of their respective issues. This may mean that the issues of the corresponding group are now closed.

[0118] Accordingly, the Current Sync ID Register (CSID) stores the current sync ID, which may be information about the currently issued group.

[0119] The Sink ID MO Counter (SMO) can count the Maximum Outstanding (MO) for each sink ID. In this case, MO may represent the maximum number of requests that a single hardware component can transmit to prevent bottlenecks in the interconnection. That is, since requests exceeding a preset MO cannot be transmitted through the interconnection, the number of current requests can be counted and controlled. This embodiment minimizes the waiting time for request transmission by counting MOs per sink ID rather than counting MOs for the entire network.

[0120] The sync queue (SQ) may be a memory with a FIFO (First In First Out) structure. The sync queue (SQ) may receive and store at least one group of sync signals and sequentially output them. The sync signals output from the sync queue (SQ) may be transmitted to the neural core load unit (112a) or the neural core store unit (112b). The neural core load unit (112a) or the neural core store unit (112b) may receive the sync signals and generate a sync out signal accordingly. At this time, the sync out signal may be generated separately for each group. The sync out signal may indicate that the operation of the corresponding group has been completely terminated. At this time, the sync out signal may refer to the synchronization signal described above. The synchronization signal may be transmitted through the L3 sync channel (5300), L2 sync path (300), and L1 sync path (L1 Sync Path). However, the present embodiment is not limited thereto.

[0121] Figure 10 is a block diagram for explaining in detail the structure of the sink ID MO counter of Figure 9.

[0122] Referring to FIG. 10, the sync ID MO counter (SMO) may include a first MO counter (ct1), a second MO counter (ct2), and an nth MO counter (ctn). That is, the sync ID MO counter (SMO) may include the first to nth MO counters (ct1~ctn).

[0123] Assuming that the instruction includes a first group through an nth group, the first through nth MO counters (ct1 to ctn) can each correspond to the first through nth groups. That is, the first MO counter (ct1) can count the MO of the first group, and the second MO counter (ct2) can count the MO of the second group. The nth MO counter (ctn) can count the MO of the nth group. In this case, if the instruction is a load instruction, the first through nth groups may be load groups, and if the instruction is a store instruction, the first through nth groups may be store groups.

[0124] FIG. 11 is a diagram illustrating a load instruction stream of a neural processing device according to some embodiments of the present invention, and FIG. 12 is a time diagram illustrating an operation of blocking a load sink signal.

[0125] Referring to FIGS. 11 and 12, the load instruction stream (I_LD) includes at least one operation instruction, i.e., load instructions (LD0~LD 2N It may include ). In this case, load instructions (LD0~LD 2N ) can have multiple load groups (GL1~GL3), and each load group (GL1~GL3) can group associated load instructions. That is, in FIG. 11, load instructions (LD0~LD N-1 ) is the first load group (GL1), and other load groups (GL2, GL3) may also sequentially include multiple load instructions. Although only three load groups are shown in FIG. 11, the present embodiment is not limited thereto.

[0126] The sync signals (SYNC0(SEND), SYNC1(SEND)) can indicate the end of each load group (GL1~GL3). That is, the end of each load group can be indicated by the sync signals (SYNC0(SEND), SYNC1(SEND)). The sync signals (SYNC0(SEND), SYNC1(SEND)) can define the issue unit of the operation instruction.

[0127] At this time, generally, operation instructions are not continuously issued as in Fig. 12, but are issued in a specific number, and when the return of that operation instruction comes back, the next operation instruction is executed to prevent delay in operation.

[0128] In other words, generally, requests are transmitted to the interconnection based on the issuance of operation instructions; however, if the number of such requests is not limited, the interconnection may become paralyzed, and the speed of the entire device may decrease. Therefore, when a specific number of operation instructions are issued, the next operation instruction may be issued after all corresponding returns have been received. The sync signals (SYNC0(SEND), SYNC1(SEND)) can define the unit of these continuously issued operation instructions.

[0129] In another aspect, operation instructions for a specific task can all be related to each other and can be executed sequentially. Since there may be dependencies with other tasks, they may need to be executed first. Therefore, these related operation instructions form a group and are issued sequentially, and can be executed before the next issue via sync signals (SYNC0(SEND), SYNC1(SEND)).

[0130] Specifically, referring to FIGS. 10 through 12, load instructions can be fetched and issued for each load group. The first load group (GL1) in FIG. 12 has load instructions (LD0~LD N-1) may be. In addition, subsequent load instructions may be sequentially included in the second load group (GL2) and the third load group (GL3).

[0131] At this time, a delay of a preset clock cycle occurs after the issue, and an interconnection request (Intercon. Req.) can be transmitted through the local interconnection (200). At this time, the delay may be a preset fixed cycle (Fc).

[0132] The issue of the sync signal (SYNC0) for the first load group (GL1) is not terminated immediately, but can be terminated only after all interconnection returns (Intercon. Retn.), which are responses to interconnection requests (Intercon. Req.), are received.

[0133] The interconnection return (Intercon. Retn.) can be received via the local interconnection (200). The interconnection return (Intercon. Retn.) may have a delay of an unset non-fixed cycle (Fc) from the interconnection request (Intercon. Req.).

[0134] At this time, the MO counter can perform counting regardless of the group. Counting can be performed by an increment signal (Icr) that rises when a load instruction is issued and a decrement signal (Dcr) that falls when an interconnection return (Intercon. Retn.) is received. That is, when a load instruction is issued, the MO counter increases by 1, and when an interconnection return (Intercon. Retn.) is received, the MO counter decreases by 1.

[0135] Accordingly, when the MO counter finally becomes 0, the response to the load instruction of the first load group (GL1), i.e., the interconnection return (Intercon. Retn.), is fully received, so the MO counter can transmit a completion signal (Ds), and accordingly, the issue of the sync signal (SYNC0) for the first load group (GL1) can be terminated. In this case, a sync out (SYNC out) signal can be issued after a fixed cycle (Fc).

[0136] Accordingly, the sync signal (SYNC0) for the first load group (GL1) may have an additional waiting time (Wt) in addition to the first time (t1), which is the actual delay during which the actual MO counter becomes 0 and the sync out (SYNC out) signal is issued by a fixed cycle. This additional waiting time may be generated by blocking new issues until all returns of the first load group (GL1) arrive.

[0137] FIG. 13 is a time diagram for explaining the operation of non-blocking a load synchronization signal of a neural processing device according to some embodiments of the present invention.

[0138] Referring to FIGS. 9 to 11 and FIG. 13, unlike FIG. 12, load instructions can be issued in a non-blocking manner.

[0139] First, the issue of the operation instruction, that is, the load instruction, can be performed in the same way as in FIG. 12. In FIG. 12, the load instruction (LD0~LD N-1 ) forms the first load group (GL1), and the load instruction (LD N ~LD 2N-1 ) can form a second group.

[0140] Load instructions (LD0~LD) of the first load group (GL1) N-1When ) is issued, SYNC ID 0 may be issued (SYNC ID (SID) of Issue). The SYNC ID may be stored in the Current SYNC ID Register (CSID). Subsequently, when the SYNC signal (SYNC0) of the first load group (GL1) is issued, an increment signal (Icr) is generated, and the SYNC ID may be increased by 1.

[0141] The interconnection request (Intercon. Req.) is the load instruction (LD0~LD) of the first load group (GL1). N-1 For each, it can be generated after a delay of a fixed cycle (Fc). An interconnection request (Intercon. Req.) can be transmitted through a local interconnection (200). An interconnection request (Intercon. Req.) can be transmitted through a local interconnection (200), a global interconnection (5000), and a first volatile memory interface (50).

[0142] An interconnection return (Intercon. Retn.) may be sent in response to an interconnection request (Intercon. Req.). The timing of the response of the interconnection return (Intercon. Retn.) may vary depending on various factors, such as the state of the interconnection and the location and state of the responding element. Therefore, the interconnection return (Intercon. Retn.) may be sent after an unspecified, non-fixed cycle (Fc) has elapsed since the interconnection request (Intercon. Req.).

[0143] A SYNC ID (SID) of Return may be generated for such an interconnection return. The SYNC ID of Return may occur simultaneously with the reception of the interconnection return.

[0144] The first MO counter (ct1) is the load instruction (LD0~LD) of the first load group (GL1). N-1The number can be counted whenever ) is issued. That is, the load instructions (LD0~LD) of the first load group (GL1) N-1 Each can be incremented by 1 by the increment signal (Icr) whenever an issue occurs. Conversely, the first MO counter (ct1) can be incremented by 1 by the decrement signal (Dcr) whenever a sink ID return for the first load group (GL1) is generated. Therefore, the load instructions (LD0~LD of the first load group (GL1) N-1 If all interconnection returns for ) have come in, the first MO counter (ct1) may be 0.

[0145] The second MO counter (ct2) is the load instruction (LD) of the second group. N ~LD 2N-1 The number can be counted whenever ) is issued. That is, the load instructions of the second group (LD N ~LD 2N-1 Each can be incremented by 1 by the increment signal (Icr) whenever an issue occurs. Conversely, the second MO counter (ct2) can be incremented by 2 by the decrement signal (Dcr) whenever a sink ID return for the second group is generated. Therefore, the load instruction (LD of the second group) N ~LD 2N-1 If all interconnection returns for ) have come in, the second MO counter (ct2) may be 0.

[0146] A SYNC queue can be a signal stored in the SYNC queue (SQ). The SYNC queue is a SYNC signal (SYNC 0, SYNC 1, Depending on the issue with SYNC2), the sync signal (SYNC 0, SYNC 1, SYNC2) can be saved sequentially.

[0147] Sync Signal (SYNC) 0, SYNC1, The issue of SYNC2) and storage in the sync queue (SYNC queue) can occur with a delay of a preset fixed cycle (Fc). The sync signal (SYNC0) for the first load group (GL1) stored in the sync queue (SQ) can be converted into the sync signal (SYNC1) for the second group by the issue of the sync signal (SYNC1) for the second group and the completion signal (Ds) generated when the first MO counter (ct1) becomes 0. That is, it can be converted into the sync signal (SYNC1) for the second group only when both the reception of the completion signal (Ds) and the issue of the sync signal (SYNC1) for the second group are satisfied. When the sync queue (SQ) is converted, the sync out signal (SYNC) after the fixed cycle (Fc) 0, SYNC 1, SYNC2) can be generated.

[0148] SYNC signal of this embodiment 0, SYNC 1, The issue with SYNC2) is that only the essential first time (t1) is required, and additional waiting time may not be necessary. This may be due to the fact that a sync ID is generated for each group, and a separate MO counter exists for each sync ID.

[0149] That is, the issue of load instructions can proceed even before the first load group (GL1) receives an interconnection return. Nevertheless, since the completion status of each is checked in parallel by a separate MO counter, each instruction can be executed sequentially without issue. Accordingly, the additional waiting time (Wt) required in FIG. 12 may not be required at all in this embodiment due to the non-blocking operation. As a result, the overall speed and performance of the device can be significantly improved.

[0150] FIG. 14 is a diagram illustrating a store instruction stream of a neural processing device according to some embodiments of the present invention.

[0151] Referring to FIG. 14, the store instruction stream (I_ST) contains at least one operation instruction, i.e., store instructions (ST0~ST 2N It may include ). In this case, store instructions (ST0~ST 2N ) can have multiple groups, and each group can group associated store instructions. That is, in FIG. 14, store instructions (ST0~ST N-1 ) can be a single group.

[0152] The sync signals (SYNC0(SEND), SYNC1(SEND)) can indicate the end of a group, just like load instructions. That is, the sync signals (SYNC0(SEND), SYNC1(SEND)) can define the issue unit of operation instructions.

[0153] FIG. 15 is a time diagram for explaining the operation of non-blocking a store synchronization signal of a neural processing device according to some embodiments of the present invention.

[0154] Referring to FIGS. 9 to 11 and FIG. 15, unlike FIG. 12, store instructions can be issued in a non-blocking manner.

[0155] First, the issue of the operation instruction, that is, the store instruction, can be performed in the same way as in FIG. 12. The store instructions (ST0~ST) of the first store group (GS1) N-1 When this is issued, SYNC ID 0 may be issued (SYNC ID (SID) of Issue). The SYNC ID may be stored in the Current SYNC ID Register (CSID). Subsequently, when the SYNC signal (SYNC0) of the first store group (GS1) is issued, an increment signal (Icr) is generated, and the SYNC ID may be increased by 1.

[0156] The interconnection request (Intercon. Req.) is the store instruction (ST0~ST) of the first store group (GS1). N-1 For each, it can be generated after a delay of a fixed cycle (Fc). An interconnection request (Intercon. Req.) can be transmitted through a local interconnection (200). An interconnection request (Intercon. Req.) can be transmitted through a local interconnection (200), a global interconnection (5000), and a first volatile memory interface (50).

[0157] An interconnection return (Intercon. Retn.) may be sent in response to an interconnection request (Intercon. Req.). The timing of the response of the interconnection return (Intercon. Retn.) may vary depending on various factors, such as the state of the interconnection and the location and state of the responding element. Therefore, the interconnection return (Intercon. Retn.) may be sent after an unspecified, non-fixed cycle (Fc) has elapsed since the interconnection request (Intercon. Req.).

[0158] A SYNC ID (SID) of Return may be generated for such an interconnection return. The SYNC ID of Return may occur simultaneously with the reception of the interconnection return.

[0159] The first MO counter (ct1) is the store instruction (ST0~ST) of the first store group (GS1). N-1 The number can be counted whenever ) is issued. That is, the store instructions (ST0~ST of the first store group (GS1) N-1Each can be incremented by 1 by the increment signal (Icr) whenever an issue occurs. Conversely, the first MO counter (ct1) can be incremented by 1 by the decrement signal (Dcr) whenever a sink ID return for the first store group (GS1) is generated. Therefore, the load instructions (LD0~LD of the first store group (GS1) N-1 If all interconnection returns for ) have come in, the first MO counter (ct1) may be 0.

[0160] The second MO counter (ct2) is the store instruction (ST) of the second group N ~ST 2N-1 The number can be counted whenever ) is issued. That is, the store instructions of group 2 (ST N ~ST 2N-1 Each ) can be incremented by 1 by the increment signal (Icr) whenever an issue occurs. Conversely, the second MO counter (ct2) can be incremented by 2 by the decrement signal (Dcr) whenever a sink ID return for the second group is generated. Therefore, the store instruction (ST of the second group N ~ST 2N-1 If all interconnection returns for ) have come in, the second MO counter (ct2) may be 0.

[0161] A SYNC queue can be a signal stored in the SYNC queue (SQ). The SYNC queue is a SYNC signal (SYNC 0, SYNC 1, Depending on the issue with SYNC2), the sync signal (SYNC 0, SYNC 1, SYNC2) can be saved sequentially.

[0162] Sync Signal (SYNC) 0, SYNC 1,The issue of SYNC2) and storage in the SYNC queue can occur with a delay equal to a preset fixed cycle (Fc). The sync signal (SYNC0) for the first store group (GS1) stored in the sync queue (SQ) can be converted into the sync signal (SYNC1) for the second group by the issue of the sync signal (SYNC1) for the second group and the completion signal (Ds) generated when the first MO counter (ct1) becomes 0. That is, it can be converted into the sync signal (SYNC1) for the second group only when both the reception of the completion signal (Ds) and the issue of the sync signal (SYNC1) for the second group are satisfied. When the sync queue (SQ) is converted, the sync out signal (SYNC) after the fixed cycle (Fc) 0, SYNC 1, SYNC2) can be generated.

[0163] That is, in this embodiment, even in the case of a store operation, just as in the case of a load operation, only a delay of the first time (t1) may be required without additional waiting time (Wt). Therefore, the waiting time associated with the load / store operation is minimized, and the overall efficiency of data transmission can be improved.

[0164] FIG. 16 is a conceptual diagram illustrating the waiting time in the case of an in-order execution pipeline, and FIG. 17 is a conceptual diagram illustrating the waiting time of an in-order execution pipeline of a neural processing device according to some embodiments of the present invention.

[0165] Referring to Fig. 16, a stall may occur as shown in Fig. 16 when proceeding with a blocking operation. That is, when there is one issue pipeline, each load / store operation can proceed through a single dependency chain. In other words, since the temporal sequence of each load / store operation must be satisfied, a waiting time is required until the load / store operation of the first load group (GL1) is completed, and after the waiting, a sync-out signal (SYNC0) can be issued and sent. That is, the issuance of the sync signal and the transmission of the sync-out signal may require additional waiting time (Wt) as shown in Fig. 12. Subsequently, the instruction issuance of the second group can proceed.

[0166] Conversely, referring to FIG. 17, the present embodiment allows the issue of the sync signal (SYNC0) of the first load group (GL1) to be immediately followed by the issue of the load / store instruction of the next second group with minimal delay. Instead, the sync signal (SYNC0) of the first load group (GL1) can be stored in the sync queue (SQ) immediately before being sent as a sync out signal.

[0167] Subsequently, the sync signal (SYNC1) of the second load group (GL2) can be stored in the sync queue (SQ). Subsequently, when the interconnection return of the first load group (GL1) is completed, the sync signal (SYNC0) of the first load group (GL1) in the sync queue (SQ) can be issued as a sync-out signal. In this way, additional time may be required between the issuance of the sync signal (SYNC0) of the same group and the issuance of the sync-out signal. At this time, the sync queue (SQ) is a FIFO structure, so the sync signal (SYNC0) of the first load group (GL1) that was issued first can be sent out first as a sync-out signal.

[0168] However, since load / store instructions continue without interruption during this process, maximum transmission efficiency can be achieved without any sections where the interconnection does not operate.

[0169] FIG. 18 is a conceptual diagram illustrating the waiting time in the case of an out-of-order execution pipeline, and FIG. 19 is a conceptual diagram illustrating the waiting time of an out-of-order execution pipeline of a neural processing device according to some embodiments of the present invention.

[0170] Referring to FIG. 18, it can be assumed that the load / store instructions of the first load group (GL1) and the second load group (GL2) have dependencies on each other, while the third load group (GL3) and the fourth load group (GL4) have no dependencies. In this case, the instruction issues of the first load group (GL1) and the second load group (GL2) must proceed in order, but the issues of the third load group (GL3) and the fourth load group (GL4) do not require such order.

[0171] Accordingly, according to the blocking operation, the first issue pipeline (PL1) has an additional waiting time (Wt) between the issue of the sink signal of the first load group (GL1) and the transmission of the sink-out signal of the first load group (GL1), as shown in FIG. 12, and then the instruction issue of the second load group (GL2) can be performed.

[0172] The second issue pipeline (PL2) can immediately issue the load / store instruction of the third load group (GL3) without additional waiting when the sync signal (SYNC0) of the first load group (GL1) becomes an issue. However, the issue of the third load group (GL3) itself may require additional waiting time. Similarly, the third issue pipeline (PL3) can immediately issue the load / store instruction of the fourth load group (GL4) without additional waiting when the sync signal (SYNC2) of the third load group (GL3) becomes an issue. However, the issue of the fourth load group (GL4) itself requires additional waiting time, after which the sync signal (SYNC3) of the fourth load group (GL4) may become an issue.

[0173] In cases of out-of-order execution like this, there may be a slight reduction in waiting time compared to in-order execution due to the overlap. However, depending on the number of pipelines and dependency status, waiting time may still exist, and hardware optimization may not be performed.

[0174] Conversely, referring to FIG. 19, the present embodiment does not require waiting time for the first issue pipeline (PL1), the second issue pipeline (PL2), and the third issue pipeline (PL3), so sequential processing can be performed in the same way as in-order execution. That is, in the first issue pipeline (PL1), the sync signal (SYNC0) of the first load group (GL1) and the sync signal (SYNC1) of the second load group (GL2) can be issued in sequence. Subsequently, in the second issue pipeline (PL2), the sync signal (SYNC2) of the third load group (GL3) can be issued, and subsequently, in the third issue pipeline (PL3), the sync signal (SYNC3) of the fourth load group (GL4) can be issued.

[0175] Referring again to FIG. 7, L0 memory (120) is a memory located inside the neural core (100) that can receive all input data required for the operation from the outside and temporarily store it. Additionally, L0 memory (120) can temporarily store output data computed by the neural core (100) to transmit it to the outside. L0 memory (120) can perform the role of a cache memory for the neural core (100).

[0176] The L0 memory (120) can transmit an input activation (Act_In) to an activation buffer (150) via an activation LSU (140) and receive an output activation (Act_Out). In addition to the activation LSU (140), the L0 memory (120) can also directly transmit and receive data with a processing unit (160). That is, the L0 memory (120) can exchange data with each of the PE array (163) and the vector unit (164).

[0177] L0 memory (120) may be memory corresponding to the neural core level, i.e., L1 (level 1). Therefore, L0 memory (120) may be defined as L1 memory. However, unlike L2 shared memory (400) and L3 shared memory, i.e., shared memory (2000), L1 memory may not be shared and may be private memory of the neural core.

[0178] The L0 memory (120) can transmit data such as activation or weight through a data path. The L0 memory (120) can send and receive synchronization signals through a separate dedicated path, the L1 Sync Path. The L0 memory (120) can send and receive synchronization signals through the L1 Sync Path with, for example, the LSU (110), the weight buffer (130), the activation LSU (140), and the processing unit (160).

[0179] The weight buffer (130) can receive weights from the L0 memory (120). The weight buffer (130) can transfer weights to the processing unit (160). The weight buffer (130) can temporarily store weights before transferring them.

[0180] Input activation (Act_In) and output activation (Act_Out) can refer to the input and output values ​​of a layer in a neural network. In this case, if there are multiple layers in the neural network, the output value of the previous layer becomes the input value of the next layer, so the output activation (Act_Out) of the previous layer can be used as the input activation (Act_In) of the next layer.

[0181] Weight can refer to a parameter that is multiplied by the input activation (Act_In) in each layer. Weight is adjusted and fixed during the deep learning training phase, and can be used to derive the output activation (Act_Out) through a fixed value during the inference phase.

[0182] The activation LSU (140) can transfer an input activation (Act_In) from the L0 memory (120) to the activation buffer (150) and transfer an output activation (Act_Out) from the activation buffer (150) to the on-chip buffer. That is, the activation LSU (140) can perform both the load and store operations of the activation.

[0183] The activation buffer (150) can provide an input activation (Act_In) to the processing unit (160) and receive an output activation (Act_Out) from the processing unit (160). The activation buffer (150) can temporarily store the input activation (Act_In) and the output activation (Act_Out).

[0184] The activation buffer (150) can quickly provide activation to a processing unit (160) with a large amount of computation, particularly a PE array (163), and can quickly receive activation to increase the computation speed of the neural core (100).

[0185] The processing unit (160) may be a module that performs operations. The processing unit (160) can perform not only one-dimensional operations but also two-dimensional matrix operations, i.e., convolution operations. The processing unit (160) can receive an input activation (Act_In), multiply it by a weight, and then add the results to generate an output activation (Act_Out).

[0186] FIG. 20 is a block diagram for explaining the processing unit of FIG. 7 in detail.

[0187] Referring to FIGS. 7 and FIGS. 20, the processing unit (160) may include a PE array (163), a vector unit (164), a column register (161), and a row register (162).

[0188] The PE array (163) can receive input activations (Act_In) and weights and perform multiplication. At this time, the input activations (Act_In) and weights can each be computed through convolution in the form of a matrix. Through this, the PE array (163) can generate output activations (Act_Out). However, the present embodiment is not limited thereto. The PE array (163) can generate any other type of output besides output activations (Act_Out).

[0189] The PE array (163) may include at least one processing element (PE). The processing elements (PE) may be aligned with each other and each may perform multiplication for one input activation (Act_In) and one weight (Weight).

[0190] The PE array (163) can generate partial sums by summing the values ​​for each multiplication. These partial sums can be used as output activations (Act_Out). Since the PE array (163) performs two-dimensional matrix multiplication, it may also be referred to as a two-dimensional matrix compute unit (2D matrix compute unit).

[0191] The vector unit (164) can primarily perform one-dimensional operations. The vector unit (164) can perform deep learning operations together with the PE array (163). Through this, the processing unit (160) can be specialized for the necessary operations. That is, the neural core (100) has a computation module that performs a large amount of two-dimensional matrix multiplication and one-dimensional operations, respectively, so that deep learning tasks can be performed efficiently.

[0192] The column register (161) can receive a first input (I1). The column register (161) can receive the first input (I1), divide it, and provide it to each column of the processing element (PE).

[0193] The row register (162) can receive the second input (I2). The row register (162) can receive the second input (I2), divide it, and provide it to each row of the processing element (PE).

[0194] The first input (I1) may be an input activation (Act_In) or a weight. The second input (I2) may be a value other than the first input (I1) among the input activation (Act_In) or the weight. Alternatively, the first input (I1) and the second input (I2) may be values ​​other than the input activation (Act_In) and the weight.

[0195] FIG. 21 is a block diagram for explaining the L0 memory of FIG. 7 in detail.

[0196] Referring to FIG. 21, the L0 memory (120) may include a scheduler (121) and at least one local memory bank (122).

[0197] When data is stored in L0 memory (120), the scheduler (121) can receive data from the load engine (113a). At this time, the data can be allocated to a local memory bank (122) in a round-robin manner. Accordingly, the data can be stored in at least one of the local memory banks (122).

[0198] Conversely, when data is loaded from L0 memory (120), the scheduler (121) can receive data from the local memory bank (122) and transfer it to the store engine (113b). The store engine (113b) can store the data externally through the local interconnection (200).

[0199] FIG. 22 is a block diagram for explaining the local memory bank of FIG. 21 in detail.

[0200] Referring to FIG. 22, the local memory bank (122) may include a local memory bank controller (122_1) and a local memory bank cell array (122_2).

[0201] The local memory bank controller (122_1) can manage read and write operations through the address of the data stored in the local memory bank (122). That is, the local memory bank controller (122_1) can manage the input and output of data overall.

[0202] The local memory bank cell array (122_2) may have a structure in which cells where data is directly stored are aligned in rows and columns. The local memory bank cell array (122_2) may be controlled by the local memory bank controller (122_1).

[0203] FIG. 13 is a block diagram for explaining in detail the structure of a neural processing device according to some embodiments of the present invention.

[0204] Referring to FIG. 23, the neural core (101) may be a CGRA structure, unlike the neural core (100). The neural core (101) may include an instruction memory (111_1), a CGRA L0 memory (111_2), a PE array (111_3), and an LSU (Load / Store Unit) (111_4).

[0205] The instruction memory (111_1) can receive and store instructions. The instruction memory (111_1) can sequentially store instructions internally and provide the stored instructions to the PE array (111_3). At this time, the instructions can direct the operation of the processing elements (113_3a) included in each PE array (111_3).

[0206] The CGRA L0 memory (111_2) is a memory located inside the neural core (101) that can receive all input data required for the operation from the outside and temporarily store it. Additionally, the CGRA L0 memory (111_2) can temporarily store output data computed by the neural core (101) to transmit it to the outside. The CGRA L0 memory (111_2) can serve as a cache memory for the neural core (101).

[0207] The CGRA L0 memory (111_2) can transmit and receive data with the PE array (111_3). The CGRA L0 memory (111_2) may be a memory corresponding to L0 (level 0), which is lower than L1. In this case, the L0 memory may be a private memory of the neural core (101) that is not shared, unlike the L1 memory (120) and L2 memory (300). The CGRA L0 memory (111_2) can transmit data such as activations or weights, programs, etc., to the PE array (111_3).

[0208] The PE array (111_3) may be a module that performs operations. The PE array (111_3) may perform not only one-dimensional operations but also matrix / tensor operations of two dimensions or more. The PE array (111_3) may include multiple processing elements (113_3a) and specific processing elements (113_3b) internally.

[0209] Processing elements (113_3a) and specific processing elements (113_3b) can be arranged in rows and columns. Processing elements (113_3a) and specific processing elements (113_3b) can be arranged in m columns. Additionally, processing elements (113_3a) can be arranged in n rows, and specific processing elements (113_3b) can be arranged in l rows. Accordingly, processing elements (113_3a) and specific processing elements (113_3b) can be arranged in (n+l) rows and m columns.

[0210] The LSU (111_4) can receive at least one of data, control signals, and synchronization signals from the outside through the L1 interconnect (130). The LSU (111_4) can transmit at least one of the received data, control signals, and synchronization signals to the CGRA L0 memory (111_2). Similarly, the LSU (111_4) can transmit at least one of the data, control signals, and synchronization signals to the outside through the L1 interconnect (130).

[0211] The neural core (101) may have a Coarse Grained Reconfigurable Architecture (CGRA) structure. Accordingly, the neural core (101) may have each processing element (113_3a) and a specific processing element (113_3b) of the PE array (111_3) connected to at least one of the CGRA L0 memory (111_2), instruction memory (111_1), and LSU (111_4). That is, the processing element (113_3a) and the specific processing element (113_3b) do not have to be connected to all of the CGRA L0 memory (111_2), instruction memory (111_1), and LSU (111_4), but may be connected to some of them.

[0212] Additionally, the processing element (113_3a) and the specific processing element (113_3b) may be different types of processing devices. Accordingly, among the CGRA L0 memory (111_2), instruction memory (111_1), and LSU (111_4), the device connected to the processing element (113_3a) and the device connected to the specific processing element (113_3b) may be different.

[0213] The neural core (101) of the present invention having a CGRA structure is capable of high-level parallel computation and can have low power consumption because direct data exchange between a processing element (113_3a) and a specific processing element (113_3b) is possible. In addition, optimization according to various computational tasks may be possible by including two or more types of processing elements (113_3a).

[0214] For example, if the processing element (113_3a) is a processing element that performs a two-dimensional operation, a specific processing element (113_3b) may be a processing element that performs a one-dimensional operation. However, the present embodiment is not limited thereto.

[0215] FIG. 24 is a block diagram illustrating memory reconfiguration of a neural processing system according to some embodiments of the present invention.

[0216] Referring to FIG. 24, the neural core SoC (10) may include first to eighth processing units (160a to 160h) and on-chip memory (OCM). FIG. 24 illustrates eight processing units as an example, but this is merely an example and the number of processing units can vary.

[0217] The on-chip memory (OCM) may include first to eighth L0 memories (120a to 120h) and a shared memory (2000).

[0218] The first to eighth L0 memories (120a to 120h) can each be used as dedicated memory for the first to eighth processing units (160a to 160h). That is, the first to eighth processing units (160a to 160h) and the first to eighth L0 memories (120a to 120h) can correspond to each other in a 1:1 ratio.

[0219] The shared memory (2000) may include first to eighth memory units (2100a to 2100h). The first to eighth memory units (2100a to 2100h) may correspond to the first to eighth processing units (160a to 160h) and the first to eighth L0 memories (120a to 120h), respectively. That is, the number of memory units may be eight, which is the same as the number of processing units and L0 memories.

[0220] The shared memory (2000) can operate as either of two types of on-chip memory formats. That is, the shared memory (2000) can operate as either an L0 memory format or a global memory format. That is, the shared memory (2000) can implement two types of logical memory with a single piece of hardware.

[0221] When the shared memory (2000) is implemented in the form of an L0 memory, the shared memory (2000) can operate as a private memory for each of the first to eighth processing units (160a to 160h), such as the first to eighth L0 memories (120a to 120h). The L0 memory can operate at a relatively high clock speed compared to the global memory, and the shared memory (2000) can also use a relatively faster clock speed when operating in the form of an L0 memory.

[0222] When the shared memory (2000) is implemented in the form of a global memory, the shared memory (2000) can operate as a common memory used by the first processing unit (100a) and the second processing unit (100b) together. At this time, the shared memory (2000) can be shared not only by the first to eighth processing units (160a to 160h) but also by the first to eighth L0 memories (120a to 120h).

[0223] Global memory can generally use a lower clock speed compared to L0 memory, but is not limited thereto. When the shared memory (2000) operates in the form of a global memory, the first to eighth processing units (160a to 160h) can share the shared memory (2000). At this time, the shared memory (2000) is connected to the volatile memory (32) of FIG. 2 through a global interconnection (5000) and may operate as a buffer for the volatile memory (32).

[0224] The shared memory (2000) may operate in an L0 memory format for at least a portion and in a global memory format for the remainder. That is, the entire shared memory (2000) may operate in an L0 memory format, or the entire shared memory (2000) may operate in a global memory format. Alternatively, a portion of the shared memory (2000) may operate in an L0 memory format, and the remainder may operate in a global memory format.

[0225] FIG. 25 is a block diagram illustrating an example of memory reconfiguration of a neural processing system according to some embodiments of the present invention.

[0226] Referring to FIGS. 24 and 25, the first, third, fifth, and seventh dedicated areas (AE1, AE3, AE5, AE7) of each of the first, third, fifth, and seventh processing units (100a, 100c, 100e, 100g) may each include only the first, third, fifth, and seventh L0 memories (120a, 120c, 120e, 120g). Additionally, the second, fourth, sixth, and eighth dedicated areas (AE2, AE4, AE6, AE8) of each of the second, fourth, sixth, and eighth processing units (100b, 100d, 100f, 100h) may each include the second, fourth, sixth, and eighth L0 memories (120b, 120d, 120f, 120h). Additionally, the second, fourth, sixth, and eighth dedicated areas (AE2, AE4, AE6, AE8) may include the second, fourth, sixth, and eighth memory units (2100b, 2100d, 2100f, 2100h). The first, third, fifth, and seventh memory units (2100a, 2100c, 2100e, 2100g) of the shared memory (2000) may be utilized as a common area (AC).

[0227] The shared area (AC) may be a memory shared by the first to eighth processing units (160a to 160h). The second private area (AE2) may include the second L0 memory (120b) and the second memory unit (2100b). The second private area (AE2) may be an area where the hardware-separated second L0 memory (120b) and the second memory unit (210b) operate in the same way to logically operate as a single L0 memory. The fourth, sixth, and eighth private areas (AE4, AE6, AE8) may also operate in the same way as the second private area (AE2).

[0228] The shared memory (2000) according to the present embodiment can use the area corresponding to each neural core by converting it into a logical L0 memory and a logical global memory at an optimized ratio. The shared memory (2000) can perform this ratio adjustment at runtime.

[0229] In other words, while each neural core may perform identical tasks, they may also perform different tasks. In this case, the capacity of L0 memory and global memory required for each neural core's task will inevitably vary each time. Consequently, if the configuration ratio of L0 memory to shared memory is fixed, as is the case with conventional on-chip memory, inefficiency may arise due to the computational tasks allocated to each neural core.

[0230] Accordingly, the shared memory (2000) of the neural processing device according to the present embodiment can set the optimal ratio of L0 memory and global memory according to the computational task during runtime, and can improve the efficiency and speed of the computation.

[0231] FIG. 26 is an enlarged block diagram of part A of FIG. 24.

[0232] Referring to FIGS. 24 and 26, the shared memory (2000) may include a first L0 memory controller (122_1a), a second L0 memory controller (122_1b), a fifth L0 memory controller (122_1e), a sixth L0 memory controller (122_1f), first to eighth memory units (2100a to 2100h), and a global controller (2200). Other L0 memory controllers not illustrated may also be included in this embodiment, but are omitted for convenience.

[0233] The first L0 memory controller (122_1a) can control the first L0 memory (120a). Additionally, the first L0 memory controller (122_1a) can control the first memory unit (2100a). Specifically, when the first memory unit (2100a) is implemented in a logical L0 memory format, control by the first L0 memory controller (122_1a) can be performed on the first memory unit (2100a).

[0234] The second L0 memory controller (122_1b) can control the second L0 memory (120b). Additionally, the second L0 memory controller (122_1b) can control the second memory unit (2100b). That is, when the second memory unit (2100b) is implemented in a logical L0 memory format, control by the first L0 memory controller (122_1a) can be performed on the second memory unit (2100b).

[0235] The fifth L0 memory controller (122_1e) can control the fifth L0 memory (120e). Additionally, the fifth L0 memory controller (122_1e) can control the fifth memory unit (2100e). That is, when the fifth memory unit (2100e) is implemented in a logical L0 memory format, control by the fifth L0 memory controller (122_1e) can be performed on the fifth memory unit (2100e).

[0236] The 6th L0 memory controller (122_1f) can control the 6th L0 memory (120f). Additionally, the 6th L0 memory controller (122_1f) can control the 6th memory unit (2100f). That is, when the 6th memory unit (2100f) is implemented in a logical L0 memory format, control by the 6th L0 memory controller (122_1f) can be performed on the 6th memory unit (2100f).

[0237] The global controller (2200) can control all of the first to eighth memory units (2100a to 2100h). Specifically, the global controller (2200) can control the first memory unit (2100a) to the eighth memory unit (2100h) when each of the first to eighth memory units (2100a to 2100h) operates logically in a global memory format (i.e., when they do not operate logically in an L0 memory format).

[0238] That is, depending on what type of memory the first to eighth memory units (2100a to 2100h) are logically implemented, they may be controlled by the first to eighth L0 memory controllers (122_1a to 122_1h) respectively or by the global controller (2200).

[0239] When an L0 memory controller including the first, second, fifth, and sixth L0 memory controllers (122_1a, 122_1b, 122_1e, 122_1f) controls the first to eighth memory units (2100a to 2100h), the first to eighth L0 memory controllers (122_1a to 141h) control the first to eighth memory units (2100a to 2100h) in the same way as the first to eighth L0 memories (120a to 120h), so they can be controlled as dedicated memory for the first to eighth processing units (160a to 160h). Accordingly, the first to eighth memory units (2100a to 2100h) can operate at a clock frequency corresponding to the clock frequency of the first to eighth processing units (160a to 160h).

[0240] The L0 memory controllers, including the first L0 memory controller (122_1a), the second L0 memory controller (122_1b), the fifth L0 memory controller (122_1e), and the sixth L0 memory controller (122_1f), may each include the LSU (110) of FIG. 7.

[0241] When the global controller (2200) controls at least one of the first to eighth memory units (2100a to 2100h), the global controller (2200) can control the first to eighth memory units (2100a to 2100h) as global memory of the first to eighth processing units (160a to 160h). Accordingly, at least one of the first to eighth memory units (2100a to 2100h) can operate at a clock frequency independent of the clock frequency of the first to eighth processing units (160a to 160h). However, the present embodiment is not limited thereto.

[0242] The global controller (2200) can connect the first to eighth memory units (2100a to 2100h) to the global interconnection (5000) of FIG. 3. The first to eighth memory units (2100a to 2100h) can exchange data with the off-chip memory (30) of FIG. 1 or exchange data with the first to eighth L0 memories (120a to 120h) respectively by the global controller (2200).

[0243] The first to eighth memory units (2100a to 2100h) may each include at least one memory bank. The first memory unit (2100a) may include at least one first memory bank (2110a). The first memory bank (2110a) may be an area in which the first memory unit (2100a) is divided into specific sizes. Each first memory bank (2110a) may be a memory element of the same size. However, the present embodiment is not limited thereto. In FIG. 15, four memory banks are shown included in one memory unit.

[0244] Similarly, the second, fifth, and sixth memory units (2100b, 2100e, 2100f) may each include at least one second, fifth, and sixth memory bank (2110b, 2110e, 2110f).

[0245] The following description is based on the first memory bank (2110a) and the fifth memory bank (2110e), which may be identical to other memory banks including the second and sixth memory banks (2110b, 2110f).

[0246] Each of the first memory banks (2110a) may logically operate in an L0 memory format or logically in a global memory format. At this time, the first memory bank (2110a) may operate independently of other memory banks within the first memory unit (2100a). However, the present embodiment is not limited thereto.

[0247] When each memory bank operates independently, the first memory unit (2100a) may include a first region that operates in the same manner as the first L0 memory (120a) and a second region that operates in a different manner from the first L0 memory (120a). In this case, the first region and the second region do not necessarily coexist, and either region may occupy the entire first memory unit (2100a).

[0248] Likewise, the second memory unit (2100b) may include a third region that operates in the same manner as the second L0 memory (120b) and a fourth region that operates in a different manner from the second L0 memory (120b). In this case, the third region and the fourth region do not necessarily coexist, and either region may occupy the entire first memory unit (2100a).

[0249] At this time, the ratio of the first region to the second region may differ from the ratio of the third region to the fourth region. However, the present embodiment is not limited thereto. Accordingly, the ratio of the first region to the second region may be the same as the ratio of the third region to the fourth region. That is, the memory configuration ratio in each memory unit can vary as much as desired.

[0250] Generally, in conventional system-on-chip (SOC) architectures, on-chip memory—excluding high-speed L0 memory—was often configured using high-density, low-power SRAM. This is because SRAM offers high efficiency in terms of chip area and power consumption relative to the required capacity. However, conventional on-chip memory inevitably suffered from significantly slower processing speeds for tasks requiring more data rapidly than the predetermined L0 memory capacity. Furthermore, inefficiencies arose because there was no way to utilize the remaining global memory even when its need was minimal.

[0251] In contrast, the shared memory (2000) according to some embodiments of the present invention may be selectively controlled by either of two controllers depending on the case. At this time, the shared memory (2000) is not controlled entirely by only one of the two controllers, but may be controlled independently at the memory unit level or the memory bank level.

[0252] Through this, the shared memory (2000) according to the present embodiment can obtain an optimal memory configuration ratio according to the computation task during runtime, thereby enabling faster and more efficient computation tasks. In the case of a processing unit specialized for artificial intelligence, the required size of L0 memory and global memory may vary depending on the specific application. Furthermore, even for the same application, if a deep learning network is used, the required size of L0 memory and global memory may vary for each layer. The shared memory (2000) according to the present embodiment can change the memory configuration ratio during runtime even with changes in the computation stage according to each layer, thereby enabling fast and efficient deep learning tasks.

[0253] FIG. 27 is a drawing for explaining the first memory bank of FIG. 26 in detail. FIG. 16 is illustrated with respect to the first memory bank (2110a), but other memory banks may have the same structure as the first memory bank (2110a).

[0254] Referring to FIG. 27, the first memory bank (2110a) may include a cell array (Ca), a bank controller (Bc), a first path unit (P1), and a second path unit (P2).

[0255] The cell array (Ca) may include a plurality of memory elements (Cells) internally. The cell array (Ca) may have a plurality of memory elements arranged in a grid structure. The cell array (Ca) may be, for example, a Static Random Access Memory (SRAM) cell array.

[0256] The bank controller (Bc) can control the cell array (Ca). The bank controller (Bc) can determine whether the cell array (Ca) will operate in L0 memory format or global memory format and control the cell array (Ca) accordingly.

[0257] Specifically, the bank controller (Bc) can determine whether to transmit and receive data in the direction of the first path unit (P1) or the direction of the second path unit (P2) during runtime. The bank controller (Bc) can determine the direction of data transmission and reception according to the path control signal (Spc).

[0258] The path control signal (Spc) can be generated by a pre-designed device driver or compiler. The path control signal (Spc) can be generated based on the characteristics of the operation. Alternatively, the path control signal (Spc) can be generated by input received from the user. That is, the user can directly apply input to the path control signal (Spc) to select the most optimal memory configuration ratio.

[0259] The bank controller (Bc) can determine the path for transmitting and receiving data stored in the cell array (Ca) through the path control signal (Spc). Depending on how the bank controller (Bc) determines the path for transmitting and receiving data, the data exchange interface may vary. That is, when the bank controller (Bc) exchanges data with the first path unit (P1), it may use the first interface, and when it exchanges data with the second path unit (P2), it may use the second interface. At this time, the first interface and the second interface may be different from each other.

[0260] In addition, the address scheme in which data is stored may also change. That is, if a specific interface is selected, read and write operations can be performed using the corresponding address scheme.

[0261] The bank controller (Bc) can operate at a specific clock frequency. For example, if the cell array (Ca) is an SRAM cell array, the bank controller (Bc) can operate at the operating clock frequency of a typical SRAM.

[0262] The first path unit (P1) can be connected to the bank controller (Bc). The first path unit (P1) can directly exchange data of the cell array (Ca) with the first processing unit (100a). Here, “directly” may mean that the data is exchanged without passing through the global interconnection (5000). That is, the first processing unit (100a) can directly exchange data with the first L0 memory (120a), and the first processing unit (100a) can exchange data through the first path unit (P1) when the shared memory (2000) is logically implemented in the L0 memory format. The first path unit (P1) may include an L0 memory controller including the first L0 memory controller (122_1a) and the second L0 memory controller (122_1b) of FIG. 14.

[0263] The first path unit (P1) can configure a multi-cycle sync-path. That is, the operating clock frequency of the first path unit (P1) can be the same as the operating clock frequency of the first processing unit (100a). The first L0 memory (120a) can rapidly exchange data at the same clock frequency as the operating clock frequency of the first processing unit (100a) in order to rapidly exchange data at the same speed as the operation of the first processing unit (100a). The first path unit (P1) can also likewise operate at the same clock frequency as the operating clock frequency of the first processing unit (100a).

[0264] At this time, the operating clock frequency of the first path unit (P1) may be a multiple of the operating clock frequency of the bank controller (Bc). In this case, a separate Clock Domain Crossing (CDC) operation for clock synchronization between the bank controller (Bc) and the first path unit (P1) is not required, and accordingly, no delay in data transmission may occur. Accordingly, faster and more efficient data exchange may be possible.

[0265] In FIG. 17, for example, the operating clock frequency of the first path unit (P1) may be 1.5 GHz. This may be twice the frequency of the bank controller (Bc) at 750 MHz. However, the present embodiment is not limited thereto, and any number of times may be possible as long as the first path unit (P1) operates at an integer multiple of the clock frequency of the bank controller (Bc).

[0266] The second path unit (P2) can be connected to the bank controller (Bc). The second path unit (P2) can exchange data from the cell array (Ca) through the global interconnection (5000) rather than directly with the first processing unit (100a). That is, the first processing unit (100a) can exchange data with the cell array (Ca) through the global interconnection (5000) and the second path unit (P2). At this time, the cell array (Ca) can exchange data not only with the first processing unit (100a) but also with other neural cores.

[0267] That is, the second path unit (P2) may be a data exchange path between the cell array (Ca) and all neural cores when the first memory bank (2110a) is logically implemented in a global memory format. The second path unit (P2) may include the global controller (2200) of FIG. 14.

[0268] The second path unit (P2) can configure an async-path. The operating clock frequency of the second path unit (P2) may be the same as the operating clock frequency of the global interconnection (5000). The second path unit (P2) may also operate at the same clock frequency as the operating clock frequency of the global interconnection (5000).

[0269] At this time, the operating clock frequency of the second path unit (P2) may not be synchronized with the operating clock frequency of the bank controller (Bc). In this case, a Clock Domain Crossing (CDC) operation may be required to synchronize the clock between the bank controller (Bc) and the second path unit (P2). If the operating clock frequency of the bank controller (Bc) and the operating clock frequency of the second path unit (P2) are not synchronized with each other, the degree of freedom in the design of the clock domain may be increased. Therefore, the difficulty of hardware design is lowered, making it easier to derive hardware operation.

[0270] The bank controller (Bc) may use different address schemes when exchanging data through the first path unit (P1) and when exchanging data through the second path unit (P2). That is, the bank controller (Bc) may use the first address scheme through the first path unit (P1) and the second address scheme through the second path unit (P2). At this time, the first address scheme and the second address scheme may be different from each other.

[0271] A bank controller (Bc) does not necessarily need to exist for each memory bank. That is, since the bank controller (Bc) serves the role of transmitting signals rather than being a part for scheduling, it is not an essential part for each memory bank having two ports. Therefore, a single bank controller (Bc) can control multiple memory banks. Multiple memory banks can operate independently even if they are controlled by a bank controller (Bc). However, the present embodiment is not limited thereto.

[0272] Of course, a bank controller (Bc) may exist for each memory bank. In this case, the bank controller (Bc) can control each memory bank individually.

[0273] Referring to FIGS. 26 and 27, the first memory unit (210a) may use a first address scheme when exchanging data through the first path unit (P1) and a second address scheme when exchanging data through the second path unit (P2). Similarly, the second memory unit (210b) may use a third address scheme when exchanging data through the first path unit (P1) and a second address scheme when exchanging data through the second path unit (P2). In this case, the first address scheme and the third address scheme may be identical. However, the present embodiment is not limited thereto.

[0274] The first address scheme and the third address scheme can be used exclusively for the first processing unit (100a) and the second processing unit (100b), respectively. The second address scheme can be applied commonly to the first processing unit (100a) and the second processing unit (100b).

[0275] In FIG. 15, for example, the operating clock frequency of the second path unit (P2) may operate at 1 GHz. This may be a frequency that is not synchronized with the operating clock frequency of the bank controller (Bc) at 750 MHz. That is, the operating clock frequency of the second path unit (P2) may be freely set without being dependent on the operating clock frequency of the bank controller (Bc) at all.

[0276] Conventional global memory inevitably experiences delays due to CDC operations by using a slow SRAM (e.g., 750 MHz) and a faster global interconnect (e.g., 1 GHz). In contrast, the shared memory (2000) according to some embodiments of the present invention can avoid delays due to CDC operations by utilizing the first path unit (P1) in addition to the second path unit (P2).

[0277] In addition, since a typical global memory utilizes a single global interconnection (5000) for multiple neural cores, a decrease in overall processing speed can easily occur when data transfer volumes occur simultaneously. In contrast, the shared memory (2000) according to some embodiments of the present invention has the potential to utilize a first path unit (P1) in addition to a second path unit (P2), thereby enabling the effect of appropriately distributing the data processing volume concentrated on the global controller (2200).

[0278] FIG. 28 is a block diagram illustrating the software layer structure of a neural processing device according to some embodiments of the present invention.

[0279] Referring to FIG. 28, the software layer structure of a neural processing device according to some embodiments of the present invention may include a DL framework (10000), a compiler stack (20000), and a backend module (30000).

[0280] The DL framework (10000) may refer to a framework for a deep learning model network used by the user. For example, a neural network that has been trained can be created using a program such as TensorFlow or PyTorch.

[0281] The compiler stack (20000) may include an adaptation layer (21000), a compute library (22000), a frontend compiler (23000), a backend compiler (24000), and a runtime driver (25000).

[0282] The adaptation layer (21000) may be a layer that interacts with the DL framework (10000). The adaptation layer (21000) can quantize the user's neural network model generated by the DL framework (10000) and modify the graph. Additionally, the adaptation layer (21000) can convert the model type to a required type.

[0283] The frontend compiler (23000) can convert various neural network models and graphs received from the adaptation layer (21000) into a certain intermediate representation (IR). The converted IR may be a pre-configured representation that is easy to handle later by the backend compiler (24000).

[0284] The IR of the frontend compiler (23000) can perform optimizations that can be done in advance at the graph level. Additionally, the frontend compiler (23000) can finally generate the IR by converting it into a layout optimized for hardware.

[0285] The backend compiler (24000) optimizes the IR converted by the frontend compiler (23000) and converts it into a binary file so that it can be used by a runtime driver. The backend compiler (24000) can generate optimized code by splitting the job into a scale that fits the hardware details.

[0286] The compute library (22000) can store template operations designed in a form suitable for hardware among various operations. The compute library (22000) provides various template operations required by hardware to the backend compiler (24000) so that optimized code can be generated.

[0287] The runtime driver (25000) can perform continuous monitoring during operation to operate the neural network device according to some embodiments of the present invention. Specifically, it can be responsible for the execution of the interface of the neural network device.

[0288] The backend module (30000) may include an Application Specific Integrated Circuit (ASIC) (31000), a Field Programmable Gate Array (FPGA) (32000), and a C-model (33000). The ASIC (31000) may refer to a hardware chip determined according to a predetermined design method. The FPGA (32000) may be a programmable hardware chip. The C-model (33000) may refer to a model implemented by simulating hardware in software.

[0289] The backend module (30000) can perform various tasks and produce results using binary code generated through the compiler stack (20000).

[0290] FIG. 29 is a conceptual diagram illustrating deep learning operations performed by a neural processing device according to some embodiments of the present invention.

[0291] Referring to FIG. 29, the artificial neural network model (40000) is a statistical learning algorithm implemented based on the structure of a biological neural network in machine learning technology and cognitive science, as an example of a machine learning model, or a structure that executes the algorithm.

[0292] An artificial neural network model (40000) can represent a machine learning model capable of problem-solving by learning to reduce the error between the correct output corresponding to a specific input and the inferred output through the repeated adjustment of synaptic weights by nodes, which are artificial neurons that form a network through the combination of synapses as in biological neural networks. For example, the artificial neural network model (40000) may include arbitrary probability models, neural network models, etc., used in artificial intelligence learning methods such as machine learning and deep learning.

[0293] A neural processing device according to some embodiments of the present invention can perform operations by implementing the form of such an artificial neural network model (40000). For example, the artificial neural network model (40000) can receive an input image and output information about at least a part of an object included in the input image.

[0294] The artificial neural network model (40000) is implemented as a multilayer perceptron (MLP) composed of multiple layers of nodes and connections between them. The artificial neural network model (40000) according to the present embodiment can be implemented using one of various artificial neural network model structures including an MLP. As shown in FIG. 19, the artificial neural network model (40000) is composed of an input layer (41000) that receives an input signal or data (40100) from the outside, an output layer (44000) that outputs an output signal or data (40200) corresponding to the input data, and n hidden layers (42000 to 43000) located between the input layer (41000) and the output layer (44000), which receive a signal from the input layer (41000), extract a feature, and transmit it to the output layer (44000) (where n is a positive integer). Here, the output layer (44000) receives a signal from the hidden layer (42000 to 43000) and outputs it to the outside.

[0295] The learning methods of the artificial neural network model (40000) include a supervised learning method that learns to be optimized for solving the problem by inputting a teacher signal (correct answer) and an unsupervised learning method that does not require a teacher signal.

[0296] The neural processing device can directly generate training data for training the artificial neural network model (40000) through simulation. In this way, a plurality of input variables and corresponding plurality of output variables are matched to the input layer (41000) and output layer (44000) of the artificial neural network model (40000), and the synapse values ​​between the nodes included in the input layer (41000), hidden layer (42000 to 43000), and output layer (44000) are adjusted so that the correct output corresponding to a specific input can be extracted. Through this learning process, characteristics hidden in the input variables of the artificial neural network model (40000) can be identified, and the synapse values ​​(or weights) between the nodes of the artificial neural network model (40000) can be adjusted so that the error between the output variable calculated based on the input variables and the target output is reduced.

[0297] FIG. 30 is a conceptual diagram illustrating the learning and inference operations of a neural network of a neural processing device according to some embodiments of the present invention.

[0298] Referring to Fig. 30, during the training phase, multiple training data (TD) can be forwarded to and then backwarded from the artificial neural network model (NN). Through this process, the weights and biases of each node of the artificial neural network model (NN) are adjusted, and training can be performed to derive increasingly accurate results. In this way, through the training phase, the artificial neural network model (NN) can be transformed into a trained neural network model (NN_T).

[0299] In the inference phase, new data (ND) can be input into the retrained neural network model (NN_T). The trained neural network model (NN_T) can derive result data (RD) using the new data (ND) as input and the already learned weights and biases. For this result data (RD), the training materials (TD) used during the training phase and the amount of TD utilized can be important.

[0300] Hereinafter, with reference to FIGS. 17 and FIGS. 31, the operation of an in-order pipeline of a neural processing device according to some embodiments of the present invention will be described. Descriptions that overlap with the embodiments described above will be brief or omitted.

[0301] FIG. 31 is a flowchart illustrating the operation of an in-order execution pipeline of a neural processing device according to some embodiments of the present invention.

[0302] Referring to FIG. 31, a load instruction for the first load group is issued (S100). Then, a first sink signal is issued and stored in the sink queue (S200). Then, a load instruction for the second load group is issued (S300).

[0303] Specifically, referring to FIG. 17, the issue of the sync signal (SYNC0) of the first load group (GL1) can be performed immediately after the issue of the load / store instruction of the next second group with minimal delay.

[0304] Again, referring to FIG. 31, the second sink signal is issued and stored in the sink queue (S400). That is, in the issue pipeline, the first load group (GL1) and the second load group (GL2) can be issued immediately without additional waiting time.

[0305] Meanwhile, in parallel, after step S200, it is determined whether the return of the first load group is complete (S500). If not, it may wait until the return of the first load group is complete. When the return of the first load group is complete, the first sink signal is issued as a sink-out signal from the sink queue (S600).

[0306] Specifically, referring to FIG. 17, when all interconnection returns of the first load group (GL1) are completed, the sync signal (SYNC0) of the first load group (GL1) that was in the sync queue (SQ) may be issued as a sync-out signal. In this way, additional time may be required between the issue of the sync signal (SYNC0) of the same group and the issue of the sync-out signal.

[0307] Referring again to FIG. 31, when both steps S400 and S600 are completed, it is determined whether the return of the second load group is complete (S700). If not, it may wait until the return of the first load group is complete. When the return of the second load group is complete, the second sink signal is issued as a sink-out signal from the sink queue (S800).

[0308] Specifically, referring to FIG. 17, when the interconnection return of the second load group (GL2) is completed, the sync signal (SYNC1) of the second load group (GL2) in the sync queue (SQ) can be issued as a sync-out signal. At this time, the sync queue (SQ) is a FIFO structure, so the sync signal (SYNC0) of the first load group (GL1) that was issued first can be sent out as a sync-out signal before the sync signal (SYNC1) of the second load group (GL2).

[0309] Hereinafter, with reference to FIGS. 19 and FIGS. 32, the operation of an in-order pipeline of a neural processing device according to some embodiments of the present invention will be described. Descriptions that overlap with the embodiments described above will be brief or omitted.

[0310] FIG. 32 is a flowchart illustrating the operation of an out-of-order execution pipeline of a neural processing device according to some embodiments of the present invention.

[0311] Referring to FIG. 32, the first and second sink signals of the first load group and the second load group are issued sequentially in the first issue pipeline (S1100).

[0312] Specifically, referring to FIG. 19, the sync signal (SYNC0) of the first load group (GL1) and the sync signal (SYNC1) of the second load group (GL2) can be issued sequentially in the first issue pipeline (PL1). The first load group (GL1) and the second load group (GL2) may have dependencies that must be processed in sequence. The load instruction issuance of the first load group (GL1) and the second load group (GL2) can proceed immediately without additional waiting time.

[0313] Again, referring to FIG. 32, the third sink signal of the third load group is issued in the second issue pipeline (S1200).

[0314] Specifically, referring to FIG. 19, a sync signal (SYNC2) of a third load group (GL3) can be issued in a second issue pipeline (PL2) different from the first issue pipeline (PL1). At this time, even though they are different issue pipelines, each load group can be executed sequentially without waiting time, just like a single pipeline.

[0315] Again, referring to FIG. 32, the fourth sink signal of the fourth load group is issued in the third issue pipeline (S1300).

[0316] Specifically, referring to FIG. 19, the sync signal (SYNC3) of the fourth load group (GL4) can be issued in the third issue pipeline (PL3), which is different from the first issue pipeline (PL1) and the second issue pipeline (PL2). At this time, even though they are different issue pipelines, each load group can be executed sequentially without waiting time, just like a single pipeline.

[0317] The above description is merely an illustrative explanation of the technical concept of the present embodiment, and a person skilled in the art to which the present embodiment belongs would be able to make various modifications and variations within the scope of the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain, not limit, the technical concept of the present embodiment, and the scope of the technical concept of the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the present embodiment.

Claims

Claim 1 A method of operating a neural processing device, comprising: a step of issuing an instruction included in at least one group, the instruction including a load instruction for data or a store instruction for transmitting and storing said data, wherein the load instruction is issued by a neural core load unit included in an LSU (Load / Store Unit) and the store instruction is issued by a neural core store unit included in the LSU; a step of providing said sink ID to the neural core load unit or said neural core store unit that issued said instruction so as to generate a sink signal for each of said at least one group by a sink ID for identifying each of said at least one group; and a step of storing said sink signal corresponding to each of said at least one group in a sink queue. Claim 2 A method of operation of a neural processing device according to claim 1, wherein each of the at least one group comprises at least one operation instruction corresponding to the load instruction or the store instruction, and the sink signal indicating the completion of an issue of the at least one operation instruction. Claim 3 A step of issuing an instruction including a load instruction for data or a store instruction for transmitting and storing said data - said load instruction is issued by a neural core load unit included in an LSU (Load / Store Unit), and said store instruction is issued by a neural core store unit included in the LSU -; a step of providing said sink ID to the neural core load unit or said neural core store unit that issued said instruction to generate a sink signal for each sink ID; A method of operation of a neural processing device, comprising the step of storing the sink signal in a sink queue, wherein the instruction comprises at least one group of instructions, and each of the at least one group of instructions comprises at least one operation instruction corresponding to the load instruction or the store instruction, and the sink signal signifying the completion of an issue of the at least one operation instruction; the step of storing the sink signal comprises: storing a sink ID for each group of the instructions being issued; updating a sink ID MO counter that counts the MO (Maximum Outstanding) according to the number of issues of the at least one operation instruction and the number of interconnection returns for the at least one operation instruction; and storing the sink signal in the sink queue, and further comprising the step of discharging the sink signal from the sink queue when the MO becomes 0. Claim 4 A method of operation of a neural processing device according to claim 3, wherein the instruction of at least one group comprises a first group instruction and a second group instruction, and the sink ID MO counter comprises a first MO counter for counting the MO of the first group and a second MO counter for counting the MO of the second group. Claim 5 A method of operation of a neural processing device according to claim 4, wherein the step of updating the sink ID MO counter includes the step of increasing the first MO counter by 1 whenever an operation instruction of the first group is issued. Claim 6 A method of operation of a neural processing device according to claim 4, wherein the step of updating the sink ID MO counter comprises the step of decreasing the first MO counter by 1 whenever an interconnection return for the operation instruction of the first group is received. Claim 7 In claim 4, the step of storing the sync signal comprises: storing the sync signal of the first group in the sync queue when the sync signal of the first group is issued; and sequentially storing the sync signal of the second group in the sync queue following the sync signal of the first group when the sync signal of the second group is issued, and the step of outputting the sync signal comprises: outputting the sync signal of the first group when the first MO counter becomes 0; and outputting the sync signal of the second group when the second MO counter becomes 0, the method of operation of a neural processing device. Claim 8 A method of operation of a neural processing device according to claim 7, wherein the sink signal of the first group and the sink signal of the second group are output sequentially. Claim 9 A method of operation of a neural processing device according to claim 4, wherein the instructions of at least one group further include instructions of a third group, the first group and the second group have dependencies on each other, and the third group is independent of the first group and the second group. Claim 10 A method of operation of a neural processing device according to claim 9, wherein the first time between the issue end time of the operation instruction of the first group and the issue start time of the operation instruction of the second group is the same as the second time between the issue end time of the operation instruction of the second group and the issue start time of the operation instruction of the third group. Claim 11 In claim 1, the above sink queue is a method of operation of a neural processing device having a FIFO (First In First Out) structure. Claim 12 A method of operation of a neural processing device according to claim 3, wherein the step of storing the sink ID includes the step of changing and storing the sink ID of a group corresponding to the sink signal after a preset fixed cycle when the sink signal is issued. Claim 13 A step of issuing an instruction including a load instruction for data or a store instruction for transmitting and storing said data - said load instruction is issued by a neural core load unit included in an LSU (Load / Store Unit), and said store instruction is issued by a neural core store unit included in the LSU -; a step of providing said sink ID to the neural core load unit or said neural core store unit that issued said instruction to generate a sink signal for each sink ID; A method of operation of a neural processing device, comprising the step of storing the sink signal in a sink queue, wherein the instruction comprises at least one group of instructions, and each of the at least one group of instructions comprises at least one operation instruction corresponding to the load instruction or the store instruction and the sink signal indicating the completion of the issue of the at least one operation instruction, and further comprising the step of transmitting an interconnection request according to the at least one operation instruction to a local interconnection when the at least one operation instruction is issued, wherein the time of transmission of the interconnection request and the time of issuance of the at least one operation instruction differ by a predetermined fixed cycle. Claim 14 A method of operation of a neural processing device according to claim 13, further comprising the step of receiving an interconnection return corresponding to the interconnection request from the local interconnection, wherein the time of receiving the interconnection return and the time of transmitting the interconnection request differ by an unpredetermined non-fixed cycle.

Images