Neural processing device and method for synchronizing the same

Abstract

The present invention relates to a neural processing device and a method for synchronizing the same. The neural processing device includes first and second neural processors, a shared memory shared by the first and second neural processors, first and second semaphore memories corresponding to the first and second neural processors, respectively, and receiving and storing an L3 sink target, where synchronization of the first and second neural processors is performed by the L3 sink target, and a global interconnection connecting the first and second neural processors and the shared memory, and including an L3 sink channel through which a synchronization signal from the L3 sink target is transmitted.

Description

[Technical Field]

[0001] The present invention relates to a neural processing device and a synchronization method thereof, and more particularly to a neural processing device and a synchronization method thereof in which synchronization is performed by each processor instead of by a central control processor. [Background technology]

[0002] In recent years, artificial intelligence (AI) technology has been attracting attention worldwide as one of the most promising technologies and a core technology of the Fourth Industrial Revolution. The biggest issue with AI technology is computing performance. AI technology realizes human learning, reasoning, perception, and natural language processing capabilities, so the most important thing is the ability to process large amounts of data quickly.

[0003] In the early days of artificial intelligence, deep learning learning and inference were performed using the central processing unit (CPU) and graphics processing unit (GPU) of conventional computers. However, due to limitations in the high workload of deep learning learning and inference tasks, neural processing units (NPUs), which are structurally specialized for deep learning tasks, are gaining attention.

[0004] Such neural processing devices contain multiple processing units and cores, and the synchronization of these modules must be clearly handled according to task dependencies. In conventional processing devices, a centralized control processor (CPC) controls such synchronization signals and manages procedural operations.

[0005] However, as neural processing devices include more and more processing units and cores, such an approach can introduce significant latency into synchronization processes and result in high overhead for the control processor. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Korean Patent Registration No. 10-2258566 Summary of the Invention [Problem to be solved by the invention]

[0007] SUMMARY OF THE INVENTION An object of the present invention is to provide a neural processing device capable of fast and efficient synchronization processing.

[0008] Another object of the present invention is to provide a method for synchronizing a neural processing device that allows for fast and efficient synchronization.

[0009] The objects of the present invention are not limited to those described above, and other unmentioned objects and advantages of the present invention can be understood from the following description and will become more clearly understood by the examples of the present invention. Furthermore, it will be readily apparent that the objects and advantages of the present invention can be realized by the means and combinations thereof set forth in the claims. [Means for solving the problem]

[0010] To solve the above problem, a neural processing device according to some embodiments of the present invention includes first and second neural processors, a shared memory shared by the first and second neural processors, first and second semaphore memories corresponding to the first and second neural processors, respectively, that receive and store an L3 sink target, whereby synchronization of the first and second neural processors is performed by the L3 sink target, and a global interconnection that connects the first and second neural processors to the shared memory and includes an L3 sink channel through which a synchronization signal from the L3 sink target is transmitted.

[0011] The global interconnection may also include the L3 sink channel, a data channel for transmitting data between the shared memory and the first and second neural processors, and a control channel for transmitting control signals to the first and second neural processors.

[0012] Additionally, the first semaphore memory may include first and second fields corresponding to the first and second neural processors, respectively.

[0013] The neural network may further include a first FIFO buffer for sequentially transmitting the value of the first field to the first neural processor.

[0014] The neural network may further include a second FIFO buffer for sequentially transmitting the values ​​of the second field to the first neural processor.

[0015] Additionally, the L3 sink targets may include first and second L3 sink targets, the first neural processor generating the first L3 sink target, and the second neural processor generating the second L3 sink target.

[0016] The L3 sink target may also include first and second sink target fields corresponding to the first and second neural processors, respectively, and the first and second sink target fields may include information regarding whether the first and second neural processors receive a synchronization signal from the L3 sink target.

[0017] Additionally, the first and second sink target fields may be sorted in the order of the virtual IDs of the first and second neural processors, respectively.

[0018] Additionally, the first neural processor may use the L3 sink target and a VPID table to identify the physical ID of the neural processor receiving the synchronization signal from the L3 sink target, and the VPID table may include information for converting between the virtual ID and the physical ID.

[0019] Furthermore, when the first and second neural processors execute the same program, the physical IDs of the first and second neural processors and the virtual IDs of the first and second neural processors may have a one-to-one correspondence with each other.

[0020] Additionally, the L3 sink target may be included in an instruction set architecture (ISA).

[0021] The neural processor may also include at least one neural core and a local interconnection for transmitting data between the at least one neural core.

[0022] The neural processor may further include an L2 sink path through which a synchronization signal is transmitted by an L2 sink target to perform synchronization between the at least one neural core.

[0023] Additionally, the L2 sink target may be included in the command set structure.

[0024] Each of the at least one neural core may also include a processing unit that receives input activations and weights, performs deep learning operations, and outputs output activations, and a local memory that temporarily stores the input activations, the weights, and the output activations.

[0025] Additionally, each of the at least one neural core may further include an activation buffer that temporarily stores the input activations and the output activations between the processing unit and the local memory.

[0026] Additionally, each of the at least one neural core may further include an activation Load / Store Unit (LSU) that moves the input activations and the output activations between the activation buffer and the local memory.

[0027] Additionally, each of the at least one neural core may further include a weight buffer that temporarily stores the weights between the local memory and the processing unit.

[0028] Additionally, each of the at least one neural core may further include an LSU that moves data between the local memory and the local interconnection.

[0029] The LSU may also include a local memory store unit that performs saves to the local memory and a local memory load unit that performs loads to the local memory.

[0030] The LSU may also include a neural core store unit that performs external stores on the neural core, and a neural core load unit that performs external loads on the neural core.

[0031] The memory may further include an L1 sink path between the local memory and another element, through which a synchronization signal from an L1 sink target for synchronization is transmitted.

[0032] Additionally, the L1 sink target may be included in the instruction set structure.

[0033] The processing unit may also include an array of processing elements that perform two-dimensional operations and a vector unit that performs one-dimensional operations.

[0034] The processing unit may further include a row register that receives a first input and supplies it to each row of the processing element array, and a column register that receives a second input and supplies it to each column of the processing element array, and the first and second inputs may each include one of a weight and an input activation.

[0035] In order to solve the other problems, a neural processing device according to some embodiments of the present invention includes at least one neural processor, a shared memory, and a global interconnection connecting the at least one neural processor and the shared memory and used for L3 synchronization of the neural processor, the neural processor including at least one neural core, a local interconnection connecting the at least one neural core, and an L2 sink path used for L2 synchronization of the at least one neural core, and the neural core including a process unit that performs calculation work, a local memory that temporarily stores data, and an L1 sink path used for L1 synchronization between the local memory and the process unit.

[0036] The global interconnection may also include a data channel for transmitting data between the at least one neural processor and the shared memory, a control channel for transmitting control signals between the at least one neural processor, and an L3 sink channel used for the L3 synchronization.

[0037] Additionally, the neural processor may further include a local interconnection for transmitting data between the at least one neural core.

[0038] It may also include a data path used for data exchange between the local memory and elements including the process unit.

[0039] The processing unit may also include an array of processing elements that perform two-dimensional operations and a vector unit that performs one-dimensional operations.

[0040] The processing element array also includes a plurality of processing elements arranged in rows and columns, each of the plurality of processing elements being capable of performing a multiplication.

[0041] The at least one neural processor may further include first and second semaphore memories corresponding to the first and second neural processors, respectively, and configured to receive and store synchronization signals corresponding to an L3 sink target, whereby synchronization of the first and second neural processors is performed by the L3 sink target.

[0042] The first semaphore memory may further include a first FIFO buffer including first and second fields corresponding to the first and second neural processors, respectively, and transmitting values ​​of the first field to the first neural processor in sequence.

[0043] The first neural processor may also transmit an instruction set structure, which may include an operation code, an L3 sink target for the L3 synchronization, an L2 sink target for the L2 synchronization, and an L1 sink target for the L1 synchronization.

[0044] In order to solve the above and other problems, a synchronization method for a neural processing device according to some embodiments of the present invention is a synchronization method for a neural processing device including first and second neural processors, in which the first neural processor generates an L3 sink target for L3 synchronization, the L3 sink target is sorted in order of virtual IDs of the first and second neural processors, the physical ID of the second neural processor is identified using the L3 sink target and a VPID table, the VPID table being a conversion table between the virtual ID and the physical ID of the neural processor, the synchronization signal generated by the L3 sink target is stored in a first semaphore memory of the second neural processor via an L3 sink channel of a global interconnection, and the second neural processor performs the L3 synchronization based on the value of the first semaphore memory.

[0045] Additionally, the first semaphore memory may include first and second fields corresponding to the first and second neural processors, respectively.

[0046] Additionally, performing the L3 synchronization may include providing the value of the first field to the second neural processor in a FIFO manner and providing the value of the second field to the second neural processor in a FIFO manner.

[0047] The virtual IDs may also include first and second virtual IDs corresponding to the first and second neural processors, respectively.

[0048] The first neural processor may also include first and second neural cores, a local interconnection for transmitting data between the first and second neural cores, and an L2 sink path for transmitting a synchronization signal by an L2 sink target for a synchronization signal between the first and second neural cores.

[0049] The first neural core may include a first process unit that receives a first input activation and a first weight, performs a deep learning operation, and outputs a first output activation; a first local memory that temporarily stores the first input activation, the first weight, and the first output activation; and a first L1 sink path that transmits a synchronization signal from an L1 sink target for synchronization between the first local memory and the first process unit. The second neural core may include a second process unit that receives a second input activation and a second weight, performs a deep learning operation, and outputs a second output activation; a second local memory that temporarily stores the second input activation, the second weight, and the second output activation; and a second L1 sink path that transmits a synchronization signal from the L1 sink target for synchronization between the second local memory and the second process unit.

[0050] The method may further include storing data in the first local memory, transmitting a synchronization signal from the L1 sink target via the first L1 sink path within the first neural core, the first neural core transmitting a synchronization signal from the L2 sink target to the second neural core via the second L2 sink path, and the second neural core receiving data via the local interconnection.

[0051] The first neural core may further include a first LSU that moves data between the first local memory and the first local interconnection, and the second neural core may further include a second LSU that moves data between the second local memory and the second local interconnection.

[0052] The first LSU may also include a first local memory store unit that performs saving of the first local memory, a first local memory load unit that performs loading of the first local memory, a first neural core store unit that performs saving to the outside in the first neural core, and a first neural core load unit that performs loading from the outside in the first neural core.

[0053] Also, within the first neural core, transmitting a synchronization signal from the L1 sink target via the first L1 sink path may include the first local memory store unit transmitting the L1 sink target to the first neural core store unit.

[0054] The second LSU may also include a second local memory store unit that performs saving of the second local memory, a second local memory load unit that performs loading of the second local memory, a second neural core store unit that performs saving from the second neural core to the outside, and a second neural core load unit that performs loading from the outside to the second neural core.

[0055] Also, transmitting the synchronization signal by the L2 sink target may include the first neural core store unit transmitting the synchronization signal by the L2 sink target to the second neural core load unit.

[0056] Furthermore, the second neural core receiving data may include the second neural core load unit requesting the data from the first local memory via the local interconnection, and the second neural core load unit receiving the data from the first local memory via the local interconnection.

[0057] To achieve the above and other objects, some embodiments of the present invention provide a method for synchronizing a neural processing device including first and second neural cores, a local interconnection connecting the first and second neural cores, and an L2 sink path used for L2 synchronization of the first and second neural cores, wherein the first neural core includes a first process unit that performs calculation work, a first local memory that temporarily stores data input / output to the first process unit, and a first L1 sink path that is used for L1 synchronization of the first local memory and the first process unit, and the second neural core includes a second process unit that performs calculation work, a first local memory that temporarily stores data input / output to the first process unit, and a first L1 sink path that is used for L1 synchronization of the first local memory and the first process unit. a second local memory for temporarily storing data input / output to / from the second process unit; and a second L1 sink path for use in L1 synchronization of the second local memory and the second process unit, the method further comprising: storing data in the first local memory; transmitting a synchronization signal from the L1 sink target via the first L1 sink path within the first neural core; the first neural core transmitting a synchronization signal from the L2 sink target to the second neural core via the second L2 sink path; and the second neural core receiving data via the local interconnection.

[0058] The first neural core may further include a first LSU that moves data between the first local memory and the first local interconnection, the first LSU including a first local memory store unit that performs storage in the first local memory and a first neural core store unit that performs storage external to the first neural core, and within the first neural core, transmitting a synchronization signal from the L1 sink target via the first L1 sink path may include the first local memory store unit transmitting a synchronization signal from the L1 sink target to the first neural core store unit.

[0059] The second neural core may further include a second LSU that moves data between the second local memory and the second local interconnection, the second LSU including a second neural core load unit that performs external loads on the second neural core, and transmitting a synchronization signal by the L2 sink target may include the first neural core store unit transmitting a synchronization signal by the L2 sink target to the second neural core load unit.

[0060] The neural processing device may also include the first and second neural cores, the local interconnection, a first neural processor including the L2 sink path, a second neural processor different from the first neural processor, a global interconnection for transmitting data between the first and second neural processors, and first and second semaphore memories corresponding to the first and second neural processors, respectively, wherein the global interconnection includes a data channel, a control channel, and an L3 sink channel for transmitting data, control signals, and synchronization signals by an L3 sink target between the first and second neural processors, respectively, wherein the first neural processor generates the L3 sink target and stores the L3 sink target in the second semaphore memory, and the second neural processor performs synchronization using the synchronization signal by the L3 sink target. [Effects of the Invention]

[0061] The neural processing device and synchronization method of the present invention can minimize latency due to synchronization requests transmitted to the control processor because each processor, core, and memory element transmits synchronization requests to each other to perform synchronization, instead of a centralized control processor.

[0062] Furthermore, since the scheduling tasks that were previously performed by the control processor no longer need to be performed, the scheduling overhead of the neural processing device can be significantly reduced.

[0063] In addition to the above, the specific effects of the present invention will be described below while explaining specific matters for implementing the present invention. [Brief explanation of the drawings]

[0064] [Figure 1]FIG. 1 is a block diagram illustrating a neural processing system according to some embodiments of the present invention. [Figure 2] FIG. 2 is a block diagram for explaining the neural processing device in FIG. 1 in more detail. [Figure 3] FIG. 3 is a block diagram for explaining the neural core SoC in FIG. 2 in detail. [Figure 4] FIG. 4 is a structural diagram for explaining the global interconnection in FIG. 3 in detail. [Figure 5] FIG. 5 is a block diagram for explaining the neural processor in FIG. 3 in more detail. [Figure 6] FIG. 6 is a block diagram for explaining the neural core in FIG. 5 in more detail. [Figure 7] FIG. 7 is a block diagram for explaining the LSU in FIG. 6 in more detail. [Figure 8] FIG. 8 is a block diagram for explaining the process unit in FIG. 6 in detail. [Figure 9] FIG. 9 is a block diagram for explaining the local memory in FIG. 6 in detail. [Figure 10] FIG. 10 is a block diagram for explaining the local memory bank in FIG. 9 in more detail. [Figure 11] FIG. 11 is a block diagram illustrating memory reconfiguration of a neural processing system according to some embodiments of the present invention. [Figure 12] FIG. 12 is a block diagram illustrating an example of memory reconfiguration for a neural processing system according to some embodiments of the present invention. [Figure 13] FIG. 13 is an enlarged block diagram of part A in FIG. [Figure 14] FIG. 14 is a diagram for explaining the first bank in FIG. 13 in detail. [Figure 15]FIG. 15 is a conceptual diagram illustrating virtual ID assignment in a neural processing device according to some embodiments of the present invention. [Figure 16] FIG. 16 is a diagram illustrating virtual ID allocation and a VPID table for a neural processing device according to some embodiments of the present invention. [Figure 17] FIG. 17 is a diagram for explaining the process of identifying a physical ID using a sink target and a VPID table. [Figure 18] Figure 18 is a directed acyclic graph for explaining the procedure of deep learning work. [Figure 19] FIG. 19 is a conceptual diagram illustrating a synchronization signal transmission operation by a sink target for L3 synchronization of a neural processing device according to some embodiments of the present invention. [Figure 20] FIG. 20 is a conceptual diagram illustrating the operation of receiving a synchronization signal by a sink target for L3 synchronization of a neural processing device according to some embodiments of the present invention. [Figure 21] FIG. 21 is a block diagram illustrating L1 and L2 synchronization of a neural processing device according to some embodiments of the present invention. [Figure 22] FIG. 22 is a ladder diagram illustrating L1 and L2 synchronization in a neural processing device according to some embodiments of the present invention. [Figure 23] FIG. 23 is a diagram illustrating the command set structure of a neural processing device according to some embodiments of the present invention. [Figure 24] FIG. 24 is a block diagram illustrating the software hierarchy of a neural processing device according to some embodiments of the present invention. [Figure 25] FIG. 25 is a conceptual diagram illustrating deep learning operations performed by neural processing devices according to some embodiments of the present invention. [Figure 26]FIG. 26 is a conceptual diagram for explaining the learning and inference operations of a neural network of a neural processing device according to some embodiments of the present invention. [Figure 27] FIG. 27 is a flow chart illustrating a method for synchronizing neural processing devices according to some embodiments of the present invention. [Figure 28] FIG. 28 is a flowchart illustrating in detail the L3 sink target storage step and the FIFO providing step in FIG. [Figure 29] FIG. 29 is a flow chart illustrating a method for synchronizing the L1 and L2 levels of a neural processing device according to some embodiments of the present invention. [Figure 30] FIG. 30 is a flowchart for explaining in detail the data request step in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0065] The terms and words used in this specification and claims should not be interpreted as being limited to their general or dictionary meanings. They should be interpreted as meanings and concepts that are consistent with the technical idea of ​​the present invention, based on the principle that the inventor can define the concepts of terms and words to best describe his or her invention. Furthermore, the embodiments described in this specification and the configurations shown in the drawings are merely one example in which the present invention is realized, and do not represent the entire technical idea of ​​the present invention. It should be understood that there may be various equivalents, modifications, and applicable examples that can replace them at the time of this application.

[0066] Terms such as "first," "second," "A," and "B" used in this specification and claims may be used to describe various components, but the components should not be limited by these terms. These terms are used only to distinguish one component from another. For example, a first component can be designated as a second component, and similarly, the second component can be designated as the first component, without departing from the scope of the present invention. The term "and / or" includes a combination of multiple related listed items or any item among multiple related listed items.

[0067] The terms used in the present specification and claims are merely used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "have" should be understood not to preclude the presence or additional possibility of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification.

[0068] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0069] Terms such as those defined in commonly used dictionaries should be interpreted to have a meaning consistent with the meaning they have in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly defined in this application.

[0070] Furthermore, the configurations, processes, steps, or methods included in the embodiments of the present invention may be shared within the scope of not being technically inconsistent with each other.

[0071] Neural processing devices according to several embodiments of the present invention will now be described with reference to FIGS.

[0072] FIG. 1 is a block diagram illustrating a neural processing system according to some embodiments of the present invention.

[0073] Referring to FIG. 1, a neural processing system NPS according to some embodiments of the present invention may include a first neural processing device 1, a second neural processing device 2, and an external interface 3.

[0074] The first neural processing device 1 may be a device that performs calculations using an artificial neural network. For example, the first neural processing device 1 may be a device specialized for performing deep learning calculations, although the present embodiment is not limited thereto.

[0075] The second neural processing device 2 may be a device having the same or similar configuration as the first neural processing device 1. The first neural processing device 1 and the second neural processing device 2 may be connected to each other via an external interface 3 and may share data and control signals.

[0076] 1 shows two neural processing devices, the neural processing system NPS according to some embodiments of the present invention is not limited to this. That is, the neural processing system NPS according to some embodiments of the present invention may have three or more neural processing devices connected to each other via the external interface 3. Conversely, the neural processing system NPS according to some embodiments of the present invention may include only one neural processing device.

[0077] FIG. 2 is a block diagram for explaining the neural processing device in FIG. 1 in more detail.

[0078] Referring to FIG. 2, the first neural processing device 1 may include a neural core SoC 10, a CPU 20, an off-chip memory 30, a first non-volatile memory interface 40, a first volatile memory interface 50, a second non-volatile memory interface 60, and a second volatile memory interface 70.

[0079] The neural core SoC10 may be a system-on-chip device. The neural core SoC10 may be an accelerator serving as an artificial intelligence computing device. The neural core SoC10 may be, for example, any one of a graphics processing unit (GPU), a field programmable gate array (FPGA), and an application-specific integrated circuit (ASIC). However, the present embodiment is not limited thereto.

[0080] Neural core SoC10 can exchange data with other external computing devices via external interface 3. Neural core SoC10 can also be connected to nonvolatile memory 31 and volatile memory 32 via first nonvolatile memory interface 40 and first volatile memory interface 50, respectively.

[0081] The CPU 20 may be a control device that controls the system of the first neural processing device 1 and executes program operations. As a general-purpose computing device, the CPU 20 may be less efficient at executing the parallel simple operations often used in deep learning. Therefore, the neural core SoC 10 may be more efficient at executing operations for deep learning inference and learning tasks.

[0082] The CPU 20 can exchange data with other external computing devices via the external interface 3. The CPU 20 can also be connected to a nonvolatile memory 31 and a volatile memory 32 via a second nonvolatile memory interface 60 and a second volatile memory interface 70, respectively.

[0083] The off-chip memory 30 may be memory located outside the chip of the neural core SoC 10. The off-chip memory 30 may include a non-volatile memory 31 and a volatile memory 32.

[0084] The non-volatile memory 31 may be a memory that continues to maintain stored information even when power is not supplied. The non-volatile memory 31 may be, for example, a ROM (Read-Only Memory), a PROM (Programmable Read-Only Memory), an EAROM (Erasable Alterable ROM), an EPROM (Erasable Programmable Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory) (e.g., NAND Flash memory, NOR Flash memory), a UVEPROM (Ultra-Violet Erasable Programmable Read-Only Memory), a FeRAM (Ferroelectric Random Access Memory), a MRAM (Magnetoresistive Random Access Memory), a PRAM (Phase-change Random Access Memory), a SONOS (silicon-oxide-nitride-oxide-silicon), a RRAM (Resistive Random Access Memory), a NRAM (Nanotube Random Access Memory), a magnetic computer storage device (e.g., a hard disk, a diskette drive, a magnetic tape), an optical disk drive, and a 3D XPoint memory (3D XPoint However, the present embodiment is not limited to this.

[0085] The volatile memory 32 may be a memory that continuously requires power to maintain stored information, unlike the nonvolatile memory 31. The volatile memory 32 may include, for example, at least one of a dynamic random access memory (DRAM), a static random access memory (SRAM), a synchronous dynamic random access memory (SDRAM), and a double data rate SDRAM (DDR SDRAM). However, the present embodiment is not limited thereto.

[0086] The first nonvolatile memory interface 40 and the second nonvolatile memory interface 60 may each include at least one of, for example, PATA (Parallel Advanced Technology Attachment), SCSI (Small Computer System Interface), SAS (Serial Attached SCSI), SATA (Serial Advanced Technology Attachment), and PCIe (PCI Express). However, the present embodiment is not limited thereto.

[0087] The first volatile memory interface 50 and the second volatile memory interface 70 may each be, for example, at least one of SDR (Single Data Rate), DDR (Double Data Rate), QDR (Quad Data Rate), and XDR (eXtreme Data Rate, Octal Data Rate), although the present embodiment is not limited thereto.

[0088] FIG. 3 is a block diagram for explaining the neural core SoC of FIG. 2 in more detail.

[0089] Referring to Figures 2 and 3, the neural core SoC 10 may include at least one neural processor 1000, a shared memory 2000, a DMA (Direct Memory Access) 3000, a non-volatile memory controller 4000, a volatile memory controller 5000, and a global interconnection 6000.

[0090] The neural processor 1000 may be a computing device that directly performs computational tasks. In the case of multiple neural processors 1000, computational tasks may be assigned to each neural processor 1000. The neural processors 1000 may be connected to each other via a global interconnection 6000.

[0091] The shared memory 2000 may be a memory shared by multiple neural processors 1000. The shared memory 2000 may store data for each neural processor 1000. The shared memory 2000 may also receive data from the off-chip memory 30, temporarily store the data, and transmit it to each neural processor 1000. Conversely, the shared memory 2000 may also receive data from the neural processor 1000, temporarily store the data, and transmit it to the off-chip memory 30 in FIG. 2.

[0092] The shared memory 2000 may require a relatively fast memory. Therefore, the shared memory 2000 may include, for example, an SRAM. However, this embodiment is not limited to this. In other words, the shared memory 2000 may also include a DRAM.

[0093] The shared memory 2000 may be a memory at the SoC level, that is, at L3 (level 3). Therefore, the shared memory 2000 may also be defined as an L3 shared memory.

[0094] The DMA 3000 can directly control data movement without the need for the neural processor 1000 to control data input and output, allowing the DMA 3000 to control data movement between memories, minimizing the number of interrupts to the neural processor 1000.

[0095] The DMA 3000 may control data movement between the shared memory 2000 and the off-chip memory 30. Under the authority of the DMA 3000, the non-volatile memory controller 4000 and the volatile memory controller 5000 may perform data movement.

[0096] The nonvolatile memory controller 4000 may control read or write operations to the nonvolatile memory 31. The nonvolatile memory controller 4000 may control the nonvolatile memory 31 via the first nonvolatile memory interface 40.

[0097] The volatile memory controller 5000 may control read or write operations to the volatile memory 32. The volatile memory controller 5000 may also perform refresh operations for the volatile memory 32. The volatile memory controller 5000 may control the volatile memory 32 via the first volatile memory interface 50.

[0098] The global interconnection 6000 may interconnect at least one neural processor 1000, shared memory 2000, DMA 3000, non-volatile memory controller 4000, and volatile memory controller 5000. An external interface 3 may also be connected to the global interconnection 6000. The global interconnection 6000 may be a path along which data travels between the at least one neural processor 1000, shared memory 2000, DMA 3000, non-volatile memory controller 4000, volatile memory controller 5000, and external interface 3.

[0099] The global interconnection 6000 can transmit not only data but also control signals and signals for synchronization. That is, in some embodiments of the neural processing device according to the present invention, each neural processor 1000 can directly transmit and receive synchronization signals, rather than a separate control processor managing synchronization signals. This can eliminate the latency of synchronization signals generated by the control processor.

[0100] That is, when there are multiple neural processors 1000, there may be dependencies between individual tasks, such that the next neural processor 1000 can start a new task only after the task of one neural processor 1000 has finished. The end and start of such individual tasks can be confirmed by a synchronization signal, but in the prior art, the control processor receives such a synchronization signal and issues an instruction to start a new task.

[0101] However, as the number of neural processors 1000 increases and task dependencies become more complex, the number of requests and instructions required for such synchronization increases exponentially, and the latency associated with each request and instruction can significantly reduce task efficiency.

[0102] Therefore, in some embodiments of the neural processing device of the present invention, instead of using a control processor, each neural processor 1000 can directly transmit synchronization signals to other neural processors 1000 according to task dependencies. In this case, multiple neural processors 1000 can perform synchronization tasks in parallel, which can minimize latency due to synchronization compared to a method managed by a control processor.

[0103] Furthermore, the control processor must perform task scheduling for the neural processors 1000 based on task dependency, and the overhead of this scheduling can increase significantly as the number of neural processors 1000 increases. Therefore, in neural processing devices according to some embodiments of the present invention, the scheduling task is also performed by each individual neural processor 1000, eliminating the scheduling burden associated with it, and thus improving the performance of the device.

[0104] FIG. 4 is a structural diagram for explaining the global interconnection in FIG. 3 in detail. Referring to FIG. 4, the global interconnection 6000 may include a data channel 6100, a control channel 6200, and an L3 sink channel 6300.

[0105] The data channel 6100 may be a dedicated channel for transmitting data. Through the data channel 6100, at least one neural processor 1000, shared memory 2000, DMA 3000, non-volatile memory controller 4000, volatile memory controller 5000, and external interface 3 may exchange data with each other.

[0106] The control channel 6200 may be a dedicated channel for transmitting control signals. Through the control channel 6200, at least one neural processor 1000, shared memory 2000, DMA 3000, non-volatile memory controller 4000, volatile memory controller 5000, and external interface 3 may exchange control signals with each other.

[0107] The L3 sink channel 6300 may be a dedicated channel for transmitting synchronization signals. Through the L3 sink channel 6300, at least one neural processor 1000, shared memory 2000, DMA 3000, non-volatile memory controller 4000, volatile memory controller 5000, and external interface 3 may exchange synchronization signals with each other.

[0108] L3 sync channel 6300 is set as a dedicated channel within global interconnection 6000 and can quickly transmit synchronization signals without overlapping with other channels. This allows neural processing devices according to some embodiments of the present invention to smoothly perform synchronization operations using existing global interconnection 6000 without the need for new wiring work.

[0109] FIG. 5 is a block diagram for explaining the neural processor in FIG. 3 in more detail. 3 to 5, the neural processor 1000 may include at least one neural core 100, an L2 shared memory 400, a local interconnection 200, and an L2 sink path 300.

[0110] At least one neural core 100 can share and execute the work of the neural processor 1000. The number of neural cores 100 can be, for example, eight. However, this embodiment is not limited to this. Although FIGS. 4 and 5 show that the neural processor 1000 includes multiple neural cores 100, this embodiment is not limited to this. In other words, the neural processor 1000 can be configured with only one neural core 100.

[0111] The L2 shared memory 400 may be a memory shared by each neural core 100 in the neural processor 1000. The L2 shared memory 400 may store data for each neural core 100. The L2 shared memory 400 may also receive data from the shared memory 2000 in FIG. 3, temporarily store the data, and transmit it to each neural core 100. Conversely, the L2 shared memory 400 may also receive data from the neural core 100, temporarily store the data, and transmit it to the shared memory 2000 in FIG. 3.

[0112] The L2 shared memory 400 may be memory corresponding to the neural processor level, i.e., L2 (level 2). The L3 shared memory, i.e., shared memory 2000, may be shared by the neural processor 1000, and the L2 shared memory 400 may be shared by the neural core 100.

[0113] The local interconnection 200 may interconnect at least one neural core 100 and the L2 shared memory 400. The local interconnection 200 may be a path along which data travels between the at least one neural core 100 and the L2 shared memory 400. The local interconnection 200 may be connected to the global interconnection 6000 in FIG. 3 to transmit data.

[0114] The L2 sink path 300 may connect at least one neural core 100 and the L2 shared memory 400 to each other. The L2 sink path 300 may be a path along which synchronization signals for the at least one neural core 100 and the L2 shared memory 400 travel.

[0115] The L2 sink path 300 may be formed physically separately from the local interconnection 200. Unlike the global interconnection 6000, the local interconnection 200 may not have sufficient channels formed therein. In such a case, the L2 sink path 300 may be formed separately to transmit synchronization signals quickly and without delay. The L2 sink path 300 may be used for synchronization performed at a level one step lower than the L3 sink channel 6300 of the global interconnection 6000.

[0116] FIG. 6 is a block diagram for explaining the neural core in FIG. 5 in more detail. Referring to FIG. 6, the neural core 100 may include a load / store unit (LSU) 110, a local memory 120, a weight buffer 130, an activation LSU 140, an activation buffer 150, and a process unit 160.

[0117] LSU 110 may receive at least one of data, control signals, and synchronization signals from the outside via local interconnection 200 and L2 sink path 300. LSU 110 may transmit at least one of the received data, control signals, and synchronization signals to local memory 120. Similarly, LSU 110 may communicate at least one of the data, control signals, and synchronization signals to the outside via local interconnection 200 and L2 sink path 300.

[0118] FIG. 7 is a block diagram illustrating the LSU of FIG. 6 in more detail. Referring to FIG. 7, LSU 110 may include a local memory load unit 111a, a local memory store unit 111b, a neural core load unit 112a, a neural core store unit 112b, a load buffer LB, a store buffer SB, a load engine 113a, a store engine 113b, and a translation index buffer 114.

[0119] The local memory load unit 111a may fetch and issue load instructions for the local memory 120. When the local memory load unit 111a provides the issued load instructions to the load buffer LB, the load buffer LB may transmit memory access requests to the load engine 113a sequentially according to the input order.

[0120] Furthermore, the local memory store unit 111b may fetch and issue a store instruction for the local memory 120. When the local memory store unit 111b provides the issued store instruction to the store buffer SB, the store buffer SB may transmit the memory access request to the store engine 113b sequentially in the order in which it was input.

[0121] Neural core load unit 112a may fetch and issue load instructions for neural core 100. When neural core load unit 112a provides the issued load instructions to load buffer LB, the load buffer LB may transmit memory access requests to load engine 113a sequentially according to the order in which they were input.

[0122] Additionally, neural core store unit 112b may fetch and issue store instructions for neural core 100. When neural core store unit 112b provides the issued store instructions to store buffer SB, store buffer SB may transmit memory access requests to store engine 113b sequentially according to the order in which they were input.

[0123] The load engine 113a receives a memory access request and can retrieve data via the local interconnection 200. In this case, the load engine 113a can quickly look up the data using a recently used virtual address-to-physical address translation table in the translation index buffer 114. If the virtual address of the load engine 113a is not in the translation index buffer 114, it can look up address translation information from other memories.

[0124] Store engine 113b receives a memory access request and can retrieve data via local interconnection 200. In this case, store engine 113b can quickly look up the data using a translation table between recently used virtual addresses and physical addresses stored in translation index buffer 114. If the virtual address of store engine 113b is not stored in translation index buffer 114, address translation information can be looked up from other memories.

[0125] The load engine 113a and the store engine 113b may transmit a synchronization signal to the L2 sink path 300. In this case, the synchronization signal may indicate that the work has been completed.

[0126] 6 again, the local memory 120 is a memory located inside the neural core 100, and can receive and temporarily store all input data required for the neural core 100 to perform its operations from the outside. The local memory 120 can also temporarily store output data calculated by the neural core 100 for transmission to the outside. The local memory 120 can serve as a cache memory for the neural core 100.

[0127] Local memory 120 may transmit input activations Act_In to activation buffer 150 and receive output activations Act_Out via activation LSU 140. In addition to activation LSU 140, local memory 120 may also directly send and receive data to and from process unit 160. That is, local memory 120 may exchange data with each of PE array 163 and vector unit 164.

[0128] The local memory 120 may be a memory corresponding to the neural core level, i.e., L1 (level 1). Therefore, the local memory 120 may also be defined as an L1 memory. However, unlike the L2 shared memory 400 and the L3 shared memory, i.e., the shared memory 2000, the L1 memory is not shared and may be a private memory for the neural core.

[0129] The local memory 120 may transmit data such as activation and wait signals via a Data Path. The local memory 120 may exchange synchronization signals via a separate dedicated path, the L1 Sync Path. The local memory 120 may exchange synchronization signals with, for example, the LSU 110, the wait buffer 130, the activation LSU 140, and the process unit 160 via the L1 Sync Path.

[0130] The weight buffer 130 may receive the weights from the local memory 120. The weight buffer 130 may transmit the weights to the processing unit 160. The weight buffer 130 may temporarily store the weights before transmitting them.

[0131] The input activation Act_In and output activation Act_Out may refer to the input and output values ​​of a neural network layer. In this case, if the neural network has multiple layers, the output value of the previous layer becomes the input value of the next layer, so the output activation Act_Out of the previous layer can be used as the input activation Act_In of the next layer.

[0132] The weights may refer to parameters that are multiplied by the input activation Act_In input to each layer. The weights are adjusted and determined in the deep learning training stage, and can be used to derive the output activation Act_Out with a fixed value in the inference stage.

[0133] Activation LSU 140 may communicate input activations Act_In from local memory 120 to activation buffer 150 and communicate output activations Act_Out from activation buffer 150 to the on-chip buffer. That is, activation LSU 140 may perform both activation load and store operations.

[0134] The activation buffer 150 may provide input activations Act_In to the process unit 160 and receive output activations Act_Out from the process unit 160. The activation buffer 150 may temporarily store the input activations Act_In and the output activations Act_Out.

[0135] The activation buffer 150 may quickly provide activations to the computationally intensive processing units 160, particularly the PE array 163, and quickly receive activations to increase the computational speed of the neural core 100.

[0136] The processing unit 160 may be a module that performs operations. The processing unit 160 may perform not only one-dimensional operations but also two-dimensional matrix operations, i.e., convolution operations. The processing unit 160 may receive input activations Act_In, multiply them by weights, and then add them to generate output activations Act_Out.

[0137] FIG. 8 is a block diagram for explaining the process unit of FIG. 6 in detail. 6 and 8, the process unit 160 may include a PE array 163, a vector unit 164, a column register 161, and a row register 162.

[0138] The PE array 163 may receive the input activation Act_In and weights and perform multiplication. In this case, the input activation Act_In and weights may be calculated by convolution in the form of a matrix. As a result, the PE array 163 may generate the output activation Act_Out. However, this embodiment is not limited thereto. The PE array 163 may easily generate other types of output besides the output activation Act_Out.

[0139] The PE array 163 may include at least one processing element PE, which may be aligned with one another and each perform a multiplication of one input activation Act_In and one weight.

[0140] The PE array 163 may generate a partial sum by summing the values ​​of each multiplication. Such a partial sum may be utilized as an output activation Act_Out. Because the PE array 163 performs two-dimensional matrix multiplication, it may also be referred to as a 2D matrix compute unit.

[0141] The vector unit 164 may mainly perform one-dimensional operations. The vector unit 164 may perform deep learning operations together with the PE array 163. This allows the processing unit 160 to specialize in the required operations. That is, the neural core 100 may have operation modules that perform a large amount of two-dimensional matrix multiplication and one-dimensional operations, respectively, and may efficiently perform deep learning tasks.

[0142] The column register 161 may receive a first input I1, which may be divided and provided to each column of the processing elements PE.

[0143] The row register 162 may receive the second input 12. The row register 162 may receive the second input 12 and divide and provide it to each row of the processing elements PE.

[0144] The first input I1 can be the input activation Act_In or a weight. The second input I2 can be a value of the input activation Act_In or a weight that is not the first input I1. Alternatively, the first input I1 and the second input I2 can be values ​​other than the input activation Act_In and a weight.

[0145] FIG. 9 is a block diagram for explaining the local memory in FIG. 6 in detail. Referring to FIG. 9, the local memory 120 may include a scheduler 121 and at least one local memory bank 122.

[0146] When data is to be stored in the local memory 120, the scheduler 121 may receive the data from the load engine 113a. At this time, the data is assigned to the local memory banks 122 in a round robin manner. Thus, the data may be stored in any one of at least one local memory bank 122.

[0147] Conversely, when data is loaded from local memory 120, scheduler 121 may receive the data from local memory bank 122 and communicate it to store engine 113b, which may store the data externally via local interconnection 200.

[0148] FIG. 10 is a block diagram illustrating the local memory bank of FIG. 9 in more detail. Referring to FIG. 10, the local memory bank 122 may include a local memory bank controller 122_1 and a local memory bank cell array 122_2.

[0149] The local memory bank controller 122_1 can manage read and write operations according to the addresses of data stored in the local memory bank 122. That is, the local memory bank controller 122_1 can manage the overall input and output of data.

[0150] The local memory bank cell array 122_2 may have a structure in which cells in which data is directly stored are arranged in rows and columns, and may be controlled by the local memory bank controller 122_1.

[0151] FIG. 11 is a block diagram illustrating memory reconfiguration of a neural processing system according to some embodiments of the present invention.

[0152] 11, the neural core SoC 10 may include first to eighth neural cores 100a to 100h and an on-chip memory OCM. Although eight neural cores are shown in FIG. 11, this is merely an example, and the number of neural cores may vary without limitation.

[0153] The on-chip memory OCM may include first to eighth local memories 120a to 120h and a shared memory 2000.

[0154] The first to eighth local memories 120a to 120h can be used as dedicated memories for the first to eighth neural cores 100a to 100h, respectively. That is, the first to eighth neural cores 100a to 100h and the first to eighth local memories 120a to 120h can have a one-to-one correspondence with each other.

[0155] The shared memory 2000 may include first to eighth memory units 2100a to 2100h. The first to eighth memory units 2100a to 2100h may correspond to the first to eighth neural cores 100a to 100h and the first to eighth local memories 120a to 120h, respectively. In other words, the number of memory units may be eight, the same as the number of neural cores and local memories.

[0156] The shared memory 2000 can operate as either of two types of on-chip memory formats: either a local memory format or a global memory format, i.e., the shared memory 2000 can realize two logical memories in one piece of hardware.

[0157] When the shared memory 2000 is implemented in a local memory format, the shared memory 2000 may operate as a private memory for each of the first to eighth neural cores 100a to 100h, similar to the first to eighth local memories 120a to 120h. Local memories may operate at a relatively faster clock than global memories, and the shared memory 2000 may also use a relatively faster clock when operating in a local memory format.

[0158] When the shared memory 2000 is implemented in the form of a global memory, the shared memory 2000 can operate as a common memory that is used by the first neural core 100a and the second neural core 100b together. In this case, the shared memory 2000 can be shared not only by the first to eighth neural cores 100a to 100h, but also by the first to eighth local memories 120a to 120h.

[0159] Global memory generally uses a slower clock than local memory, but is not limited to this. When the shared memory 2000 operates in the global memory format, the first to eighth neural cores 100a to 100h can share the shared memory 2000. In this case, the shared memory 2000 is connected to the volatile memory 32 in FIG. 2 via the global interconnection 6000, and can also operate as a buffer for the volatile memory 32.

[0160] At least a portion of the shared memory 2000 may operate in a local memory format, and the remainder may operate in a global memory format. That is, the entire shared memory 2000 may operate in a local memory format, or the entire shared memory 2000 may operate in a global memory format. Alternatively, a portion of the shared memory 2000 may operate in a local memory format, and the remaining portion may operate in a global memory format.

[0161] FIG. 12 is a block diagram illustrating an example of memory reconfiguration for a neural processing system according to some embodiments of the present invention.

[0162] 11 and 12, the first, third, fifth, and seventh dedicated areas AE1, AE3, AE5, and AE7 of the first, third, fifth, and seventh neural cores 100a, 100c, 100e, and 100g, respectively, may include only the first, third, fifth, and seventh local memories 120a, 120c, 120e, and 120g, respectively. Also, the second, fourth, sixth, and eighth dedicated areas AE2, AE4, AE6, and AE8 of the second, fourth, sixth, and eighth neural cores 100b, 100d, 100f, and 100h, respectively, may include the second, fourth, sixth, and eighth local memories 120b, 120d, 120f, and 120h, respectively. Additionally, the second, fourth, sixth, and eighth dedicated areas AE2, AE4, AE6, and AE8 may include the second, fourth, sixth, and eighth memory units 2100b, 2100d, 2100f, and 2100h. The first, third, fifth, and seventh memory units 2100a, 2100c, 2100e, and 2100g of the shared memory 2000 may be utilized as a common area AC.

[0163] The shared area AC may be a memory shared by the first to eighth neural cores 100a to 100h. The second dedicated area AE2 may include the second local memory 120b and the second memory unit 2100b. The second dedicated area AE2 may be an area in which the second local memory 120b and the second memory unit 210b, which are separated in hardware, operate in the same manner and logically operate as a single local memory. The fourth, sixth, and eighth dedicated areas AE4, AE6, and AE8 may also operate in the same manner as the second dedicated area AE2.

[0164] The shared memory 2000 according to this embodiment can switch between an optimized ratio of logical local memory and logical global memory for the area corresponding to each neural core. The shared memory 2000 can adjust this ratio at run time.

[0165] In other words, each neural core may perform the same task, but may also perform different tasks. In this case, the local memory capacity and global memory capacity required for the task performed by each neural core will be different each time. As a result, if the configuration ratio of local memory to shared memory is fixed, as in existing on-chip memories, inefficiencies may occur due to the computational tasks assigned to each neural core.

[0166] Therefore, the shared memory 2000 of the neural processing device according to this embodiment can set the optimal ratio of local memory and global memory depending on the calculation work during runtime, thereby improving the efficiency and speed of calculations.

[0167] FIG. 13 is an enlarged block diagram of part A in FIG. 11 and 13, the shared memory 2000 may include a first local memory controller 122_1a, a second local memory controller 122_1b, a fifth local memory controller 122_1e, a sixth local memory controller 122_1f, first to eighth memory units 2100a to 2100h, and a global controller 2200. Other local memory controllers not shown may also be included in this embodiment, but will not be described for convenience.

[0168] The first local memory controller 122_1a may control the first local memory 120a. The first local memory controller 122_1a may also control the first memory unit 2100a. Specifically, when the first memory unit 2100a is implemented in a logical local memory format, control by the first local memory controller 122_1a may be performed on the first memory unit 2100a.

[0169] The second local memory controller 122_1b may control the second local memory 120b. The second local memory controller 122_1b may also control the second memory unit 2100b. That is, when the second memory unit 2100b is implemented in a logical local memory format, control by the first local memory controller 122_1a may be performed on the second memory unit 2100b.

[0170] The fifth local memory controller 122_1e may control the fifth local memory 120e. The fifth local memory controller 122_1e may also control the fifth memory unit 2100e. That is, when the fifth memory unit 2100e is implemented in a logical local memory format, control by the fifth local memory controller 122_1e may be performed on the fifth memory unit 2100e.

[0171] The sixth local memory controller 122_1f may control the sixth local memory 120f. Also, the sixth local memory controller 122_1f may control the sixth memory unit 2100f. That is, when the sixth memory unit 2100f is implemented in a logical local memory format, control by the sixth local memory controller 122_1f may be performed on the sixth memory unit 2100f.

[0172] The global controller 2200 can control all of the first to eighth memory units 2100a to 2100h. Specifically, the global controller 2200 can control the first to eighth memory units 2100a to 2100h when the first to eighth memory units 2100a to 2100h logically operate in a global memory format (i.e., when the first to eighth memory units 2100a to 2100h do not logically operate in a local memory format).

[0173] That is, the first to eighth memory units 2100a to 2100h can be controlled by the first to eighth local memory controllers 122_1a to 122_1h, respectively, or can be controlled by the global controller 2200, depending on the type of memory that is logically realized.

[0174] When the local memory controllers including the first, second, fifth, and sixth local memory controllers 122_1a, 122_1b, 122_1e, and 122_1f control the first to eighth memory units 2100a-2100h, respectively, the first to eighth local memory controllers 122_1a-122_1h control the first to eighth memory units 2100a-2100h in the same manner as the first to eighth local memories 120a-120h, and can therefore control them as dedicated memories for the first to eighth neural cores 100a-100h. As a result, the first to eighth memory units 2100a-2100h can operate at clock frequencies corresponding to the clock frequencies of the first to eighth neural cores 100a-100h.

[0175] The local memory controllers including the first local memory controller 122_1a, the second local memory controller 122_1b, the fifth local memory controller 122_1e, and the sixth local memory controller 122_1f may each include the LSU 110 in FIG.

[0176] When the global controller 2200 controls at least one of the first to eighth memory units 2100a to 2100h, the global controller 2200 can control the first to eighth memory units 2100a to 2100h with the global memories of the first to eighth neural cores 100a to 100h, respectively. This allows at least one of the first to eighth memory units 2100a to 2100h to operate at a clock frequency independent of the clock frequencies of the first to eighth neural cores 100a to 100h, respectively. However, this embodiment is not limited to this.

[0177] Global controller 2200 can connect first to eighth memory units 2100a to 2100h to global interconnection 6000 in Figure 3. Global controller 2200 can enable first to eighth memory units 2100a to 2100h to exchange data with off-chip memory 30 in Figure 1 and to exchange data with first to eighth local memories 120a to 120h, respectively.

[0178] Each of the first to eighth memory units 2100a to 2100h may include at least one memory bank. The first memory unit 2100a may include at least one first memory bank 2110a. The first memory bank 2110a may be an area obtained by dividing the first memory unit 2100a into a specific size. Each of the first memory banks 2110a may be memory elements of the same size. However, this embodiment is not limited to this. FIG. 13 shows four memory banks included in one memory unit.

[0179] Similarly, the second, fifth, and sixth memory units 2100b, 2100e, 2100f may include at least one second, fifth, and sixth memory bank 2110b, 2110e, 2110f, respectively.

[0180] The following description will be based on the first memory bank 2110a and the fifth memory bank 2110e, which may be the same for other memory banks including the second and sixth memory banks 2110b and 2110f.

[0181] The first memory banks 2110a may operate logically as local memory or as global memory, and may operate independently of other memory banks in the first memory unit 2100a. However, this embodiment is not limited to this.

[0182] When each memory bank operates independently, the first memory unit 2100a may include a first region that operates in the same manner as the first local memory 120a and a second region that operates in a different manner from the first local memory 120a. In this case, the first region and the second region do not necessarily coexist, and either region may occupy the entire first memory unit 2100a.

[0183] Similarly, the second memory unit 2100b may include a third area that operates in the same manner as the second local memory 120b and a fourth area that operates in a different manner from the second local memory 120b. In this case, the third area and the fourth area do not necessarily coexist, and either area may occupy the entire first memory unit 2100a.

[0184] In this case, the ratio of the first region to the second region may be different from the ratio of the third region to the fourth region. However, this embodiment is not limited thereto. Therefore, the ratio of the first region to the second region may be the same as the ratio of the third region to the fourth region. In other words, the memory configuration ratio in each memory unit may vary without limitation.

[0185] In general, in conventional systems on a chip, on-chip memory, excluding high-speed local memory, is often configured with high-density, low-power SRAM. This is because SRAM offers high efficiency in terms of chip area and power consumption relative to the required capacity. However, conventional on-chip memory inevitably results in significantly slower processing speeds when tasks require more data quickly than the predetermined local memory capacity. Furthermore, even when there is little need for global memory, there is no way to utilize the remaining global memory, resulting in inefficiencies.

[0186] In contrast, the shared memory 2000 according to some embodiments of the present invention can be selectively controlled by either one of the two controllers as needed. In this case, the shared memory 2000 is not entirely controlled by only one of the two controllers, but can be independently controlled on a memory unit basis or a memory bank basis.

[0187] As a result, the shared memory 2000 according to this embodiment can obtain an optimal memory configuration ratio according to a calculation operation during runtime, thereby enabling faster and more efficient calculation operations. In the case of a process unit specialized for artificial intelligence, the required sizes of local memory and global memory may differ for each specific application. Furthermore, even in the same application, when a deep learning network is used, the required sizes of local memory and global memory may differ for each layer. The shared memory 2000 according to this embodiment can change the memory configuration ratio during runtime in response to changes in the calculation stage of each layer, thereby enabling faster and more efficient deep learning operations.

[0188] Figure 14 is a diagram for explaining in detail the first bank in Figure 13. Figure 14 shows the first memory bank 2110a, but the other memory banks may have the same structure as the first memory bank 2110a.

[0189] Referring to FIG. 14, the first memory bank 2110a may include a cell array Ca, a bank controller Bc, a first pass unit P1, and a second pass unit P2.

[0190] The cell array Ca may include a plurality of memory elements (Cells) arranged in a lattice structure, and may be, for example, a static random access memory (SRAM) cell array.

[0191] The bank controller Bc may control the cell array Ca. The bank controller Bc may determine whether the cell array Ca operates in a local memory format or a global memory format, and control the cell array Ca accordingly.

[0192] Specifically, the bank controller Bc may determine during runtime whether to transmit or receive data in the direction of the first path unit P1 or the direction of the second path unit P2. The bank controller Bc may determine the data transmission / reception direction in response to the path control signal Spc.

[0193] The routing control signal Spc may be generated by a pre-designed device driver or compiler. The routing control signal Spc may be generated according to the characteristics of the computing task. Alternatively, the routing control signal Spc may be generated by receiving input from a user. That is, the user may directly input the routing control signal Spc to select the optimal memory configuration ratio.

[0194] The bank controller Bc may determine the transmission / reception path of data stored in the cell array Ca using the path control signal Spc. The bank controller Bc's determination of the data transmission / reception path may change the data exchange interface. That is, the bank controller Bc may use the first interface when exchanging data with the first path unit P1, and the second interface when exchanging data with the second path unit P2. In this case, the first interface and the second interface may be different from each other.

[0195] The address scheme in which data is stored may also vary, i.e., when a particular interface is selected, read and write operations may be performed in the corresponding address scheme.

[0196] The bank controller Bc can operate at a specific clock frequency. For example, if the cell array Ca is an SRAM cell array, the bank controller Bc can operate at a general SRAM operating clock frequency.

[0197] The first pass unit P1 may be connected to the bank controller Bc. The first pass unit P1 may directly exchange data from the cell array Ca with the first neural core 100a. Note that "directly" may mean that data is exchanged between them without going through the global interconnection 6000. In other words, the first neural core 100a may exchange data directly with the first local memory 120a, and the first neural core 100a may exchange data via the first pass unit P1 when the shared memory 2000 is logically realized in a local memory format. The first pass unit P1 may include a local memory controller including the first local memory controller 122_1a and the second local memory controller 122_1b in FIG. 13.

[0198] The first path unit P1 may configure a multi-cycle sync-path. That is, the operating clock frequency of the first path unit P1 may be the same as the operating clock frequency of the first neural core 100a. The first local memory 120a may rapidly exchange data at the same clock frequency as the operating clock frequency of the first neural core 100a in order to rapidly exchange data at the same speed as the operation of the first neural core 100a. Similarly, the first path unit P1 may operate at the same clock frequency as the operating clock frequency of the first neural core 100a.

[0199] In this case, the operating clock frequency of the first pass unit P1 may be a multiple of the operating clock frequency of the bank controller Bc. In this case, a separate clock domain crossing (CDC) operation for clock synchronization between the bank controller Bc and the first pass unit P1 is not required, and therefore, data transmission delays may not occur. Therefore, faster and more efficient data exchange may be possible.

[0200] 14, the operating clock frequency of the first pass unit P1 may be 1.5 GHz, which may be twice the frequency of the bank controller Bc (750 MHz). However, this embodiment is not limited to this, and the first pass unit P1 may operate at any frequency other than an integer multiple of the clock frequency of the bank controller Bc.

[0201] The second path unit P2 may be connected to the bank controller Bc. The second path unit P2 may exchange data of the cell array Ca not directly with the first neural core 100a but via the global interconnection 6000. That is, the first neural core 100a may exchange data with the cell array Ca via the global interconnection 6000 and the second path unit P2. In this case, the cell array Ca may exchange data not only with the first neural core 100a but also with other neural cores.

[0202] That is, when the first memory bank 2110a is logically implemented in a global memory format, the second path unit P2 can be a data exchange path between the cell array Ca and all the neural cores. The second path unit P2 can include the global controller 2200 in FIG. 13.

[0203] The second path unit P2 may constitute an async-path. The operating clock frequency of the second path unit P2 may be the same as the operating clock frequency of the global interconnection 6000. Similarly, the second path unit P2 may operate at the same clock frequency as the operating clock frequency of the global interconnection 6000.

[0204] In this case, the operating clock frequency of the second pass unit P2 may not be synchronized with the operating clock frequency of the bank controller Bc. In this case, a clock domain crossing (CDC) operation may be required to synchronize the clocks between the bank controller Bc and the second pass unit P2. If the operating clock frequency of the bank controller Bc and the operating clock frequency of the second pass unit P2 are not synchronized with each other, the degree of freedom in clock domain design may be increased. Therefore, the difficulty of hardware design may be reduced, and hardware operation may be more easily derived.

[0205] The bank controller Bc may use different address architectures when exchanging data via the first path unit P1 and when exchanging data via the second path unit P2. That is, the bank controller Bc may use a first address architecture through the first path unit P1 and a second address architecture through the second path unit P2. In this case, the first address architecture and the second address architecture may be different from each other.

[0206] A bank controller Bc does not necessarily have to exist for each memory bank. In other words, the bank controller Bc is not a part for scheduling but serves to transmit signals, so it is not an essential part for each memory bank having two ports. Therefore, one bank controller Bc can control multiple memory banks. Multiple memory banks can operate independently even when controlled by the bank controller Bc. However, this embodiment is not limited to this.

[0207] Of course, a bank controller Bc may exist for each memory bank, in which case the bank controller Bc may control each memory bank individually.

[0208] 13 and 14, the first memory unit 210a may use a first address realm when exchanging data through the first path unit P1, and may use a second address realm when exchanging data through the second path unit P2. Similarly, the second memory unit 210b may use a third address realm when exchanging data through the first path unit P1, and may use the second address realm when exchanging data through the second path unit P2. In this case, the first address realm and the third address realm may be the same. However, this embodiment is not limited thereto.

[0209] The first address scheme and the third address scheme may be used exclusively for the first neural core 100a and the second neural core 100b, respectively. The second address scheme may be commonly used for the first neural core 100a and the second neural core 100b.

[0210] 14, the second pass unit P2 may operate at an operating clock frequency of 1 GHz, which may be a frequency that is not synchronized with the operating clock frequency of the bank controller Bc, 750 MHz. In other words, the operating clock frequency of the second pass unit P2 is not dependent on the operating clock frequency of the bank controller Bc at all and may be freely set.

[0211] A typical global memory uses slow SRAM (e.g., 750 MHz) and an even faster global interconnection (e.g., 1 GHz), which inevitably causes delays due to CDC operations. In contrast, the shared memory 2000 according to some embodiments of the present invention has the ability to use the first path unit P1 in addition to the second path unit P2, making it possible to avoid delays due to CDC operations.

[0212] Furthermore, in a typical global memory, multiple neural cores use one global interconnection 6000, which can easily cause a decrease in overall processing speed when large amounts of data are transmitted simultaneously. In contrast, the shared memory 2000 according to some embodiments of the present invention can utilize the first path unit P1 in addition to the second path unit P2, which can also provide the effect of appropriately distributing the data processing load concentrated in the global controller 2200.

[0213] FIG. 15 is a conceptual diagram illustrating the assignment of virtual IDs for neural processing devices according to some embodiments of the present invention.

[0214] 15, neural core SoC10 may include multiple neural processors. Figure 15 shows a case where there are eight neural processors, for example. Neural core SoC10 may include first to eighth neural processors PP0 to PP7.

[0215] In this case, the first to fourth neural processors PP0 to PP3 can divide and execute one task using the same program. The fifth neural processor 1000 can execute one task by itself, and the sixth to eighth neural processors PP5 to PP7 can divide and execute another task.

[0216] That is, the eight neural processors can be divided into three sets. In this case, the first set Set1 can include the first to fourth neural processors PP0 to PP3. The second set Set2 can include the fifth neural processor 1000. The third set Set3 can include the sixth to eighth neural processors PP5 to PP7.

[0217] A new virtual ID may be assigned to each set. That is, the first to fourth neural processors PP0 to PP3 of the first set Set1 may be assigned first to fourth virtual IDs (VP0 to VP3), respectively. The fifth neural processor 1000 of the second set Set2 may be assigned the first virtual ID (VP0). The sixth to eighth neural processors PP5 to PP7 of the third set Set3 may be assigned first to third virtual IDs (VP0 to VP2).

[0218] Therefore, when different programs are executed, the same virtual ID may be assigned to different neural processors, but when the same program is executed together, the physical ID (i.e., the unique ID of each neural processor) and the virtual ID may correspond 1:1.

[0219] FIG. 16 is a diagram illustrating the allocation of virtual IDs and a VPID table for neural processing devices according to some embodiments of the present invention.

[0220] Referring to FIG. 16, a case will be described in which the first to fourth neural processors PP0 to PP3 of the first set Set1 are assigned first to fourth virtual IDs (VP0 to VP3). The procedures for physical IDs and virtual IDs do not have to be the same. That is, the first neural processor 1000 may be assigned the third virtual ID (VP2) rather than the first virtual ID (VP0). The second neural processor 1000 may be assigned the second virtual ID (VP1), and the third neural processor 1000 may be assigned the first virtual ID (VP0). The fourth neural processor 1000 may be assigned the fourth virtual ID (VP3).

[0221] As a result, the VPID table TB_VTP can record the physical IDs corresponding to the virtual IDs. For example, if the VPID table TB_VTP records the values ​​3, 0, 1, and 2 in sequence, it can be confirmed in reverse order which physical IDs correspond to the first to fourth virtual IDs (VP0 to VP3).

[0222] Specifically, the neural processor assigned the first virtual ID (VP0) is the third neural processor 1000 with the number 2, and the neural processor assigned the second virtual ID (VP2) is the second neural processor 1000 with the number 1. The neural processor assigned the third virtual ID (VP2) is the first neural processor 1000 with the number 0, and the neural processor assigned the fourth virtual ID (VP4) is the fourth neural processor 1000 with the number 3.

[0223] FIG. 17 is a diagram for explaining the process of identifying a physical ID using a sink target and a VPID table.

[0224] Referring to FIG. 17, the L3 sink target Sm_V may be a signal generated by each neural processor that sends a synchronization signal. That is, the L3 sink target Sm_V may include, for example, four fields. This may be due to the fact that there are four neural processors in the same set. Each field of the L3 sink target Sm_V may correspond to the first to fourth virtual IDs (VP0 to VP3). That is, if 1, 0, 1, and 1 are written in the L3 sink target Sm_V, the first to fourth virtual IDs (VP0 to VP3) may correspond to 1, 1, 0, and 1, respectively, in reverse order.

[0225] The meaning of "1" of the L3 sink target Sm_V may indicate the virtual ID of the neural processor 1000 to which the synchronization signal from the L3 sink target Sm_V should be transmitted. That is, the last 1 among 1, 0, 1, 1 may mean that the synchronization signal from the L3 sink target Sm_V should be transmitted to the neural processor of the first virtual ID (VP0). That is, 1, 0, 1, 1 may represent that the synchronization signal from the L3 sink target Sm_V should be transmitted to the remaining three neural processors excluding the neural processor of the third virtual ID (VP2).

[0226] After the L3 sink target Sm_V identifies the virtual IDs of the neural processors to which the synchronization signal from the L3 sink target Sm_V should be transmitted using the first, second, and fourth virtual IDs (VP0, VP1, VP3), the neural processor that should transmit the synchronization signal from the L3 sink target Sm_V can check the physical ID of the corresponding neural processor using the VPID table TB_VTP. The neural processor can check its actual address only after checking the physical ID.

[0227] Since the VPID table TB_VTP has values ​​of 3, 0, 1, and 2, it can be seen that the physical IDs of the first, second, and fourth virtual IDs (VP0, VP1, VP3) are 2, 1, and 3, respectively. In other words, the second to fourth neural processors PP1 to PP3 can be neural processors that receive a synchronization signal from the L3 sink target Sm_V.

[0228] FIG. 18 is a directed acyclic graph (DAG) for explaining the procedure of deep learning work.

[0229] 18, the computational tasks of a neural processing device according to some embodiments of the present invention can be represented by a directed acyclic graph, where if a current task is represented as Task N, the previous task can be Task (N-1), and the next task can be Task (N+1).

[0230] That is, for the current task, TaskN, to be executed, Task(N-1) must be completed, and similarly, for the next task, Task(N+1), to be executed, the current task, TaskN, must be completed.

[0231] Therefore, a synchronization signal indicating the completion of each task should be sent from the neural processor that performed the task, and the synchronization signal may be determined by the dependency chain for the next task to be performed. Thus, the L3 sink target Sm_V may be an instruction in which information for the neural processor that should perform the next task is written. When a value is written to the L3 sink target Sm_V, a synchronization signal may be transmitted accordingly.

[0232] FIG. 19 is a conceptual diagram illustrating a synchronization signal transmission operation by a sink target for L3 synchronization of a neural processing device according to some embodiments of the present invention.

[0233] 19, the first neural processor 1000 may transmit a synchronization signal according to the sink target Sm_V to the second to fourth neural processors PP1 to PP3, thereby performing synchronization at the SoC level, i.e., L3 (level 3).

[0234] A neural processing device according to some embodiments of the present invention may include first to third semaphore memories smp1 to smp3 corresponding to the second to fourth neural processors PP1 to PP3, respectively. The first to third semaphore memories smp1 to smp3 may be included in the second to fourth neural processors PP1 to PP3, respectively. The first to third semaphore memories smp1 to smp3 may have the same configuration as each other. Therefore, only the first semaphore memory smp1 will be described in detail below.

[0235] The first semaphore memory smp1 may correspond to the second neural processor 1000. The first semaphore memory smp1 may include four fields corresponding respectively to the four neural processors included in the first set Set1.

[0236] For example, the first semaphore memory smp1 may include first to fourth fields, which may correspond to the first to fourth neural processors PP0 to PP3, respectively. That is, the first to fourth fields may be arranged in the same order as the physical IDs of the first to fourth neural processors PP0 to PP3.

[0237] That is, the first field of the first semaphore memory smp1 is the part for the first neural processor 1000, and if a synchronization signal from the L3 sink target Sm_V is received from the first neural processor 1000, it is expressed as 1, otherwise it is expressed as 0. Of course, it may also be expressed the other way around.

[0238] Similarly, the first field values ​​of the second semaphore memory smp2 and the third semaphore memory smp3 may also be displayed as 1 upon receiving a synchronization signal from the L3 sink target Sm_V by the first neural processor 1000. Thus, the display of 1, 0, 1, 1 in the first semaphore memory smp1 may mean that a synchronization signal from the L3 sink target Sm_V is received by the first, third, and fourth neural processors PP0, PP2, PP3.

[0239] When the current task, Task N, is completed, the first neural processor 1000 may transmit a synchronization signal to the L3 sink target Sm_V via the L3 sink channel 6300 in Figure 4 to start the next task, Task (N+1). Such synchronization may also be performed by the other neural processors 1000.

[0240] Since the synchronization tasks of the neural processing device of this embodiment do not require a separate central control processor, they can be performed in parallel, minimizing latency. Also, since the overhead of scheduling tasks based on the dependency of synchronization tasks is not required, the overall efficiency of the device can be maximized.

[0241] FIG. 20 is a conceptual diagram illustrating the operation of receiving a synchronization signal by a sink target for L3 synchronization of a neural processing device according to some embodiments of the present invention.

[0242] 20, the first neural processor 1000 may receive synchronization signals from the first, third, and fourth neural processors PP0, PP2, and PP3, which causes the first to fourth fields of the first semaphore memory smp1 corresponding to the first neural processor 1000 to be filled with 1, 0, 1, and 1, respectively.

[0243] A neural processing device according to some embodiments of the present invention may include first to fourth FIFO buffers B1 to B4 corresponding to the first to fourth fields, respectively. The first to fourth FIFO buffers may provide the values ​​of the first to fourth fields of the first semaphore memory smp1 to the first neural processor 1000 in a FIFO (First In First Out) manner.

[0244] In general, the operations of a neural processing device are not simply represented by a straight line as in Figure 18. That is, an operation may have a dependency chain with multiple previous operations. This may require multiple semaphore memories for an operation with more than one dependency chain.

[0245] However, increasing the number of semaphore memories also increases the memory space required, which can result in excessive resources being required for a small space. Therefore, some embodiments of the neural processing device of the present invention can provide one semaphore memory per neural processor plus a FIFO buffer to efficiently utilize memory space.

[0246] In other words, when synchronization signals due to multiple dependencies are sequentially input to the FIFO buffer, even a single semaphore memory can process the synchronization signals sequentially without missing any of them. Therefore, this embodiment can easily perform operations for multiple dependency chains while also improving memory efficiency.

[0247] FIG. 21 is a block diagram illustrating L1 and L2 synchronization of a neural processing device according to some embodiments of the present invention, and FIG. 22 is a ladder diagram illustrating L1 and L2 synchronization of a neural processing device according to some embodiments of the present invention.

[0248] Referring to FIG. 21, the first neural core 100a may include a first neural core store unit 112b, a first neural core load unit 112a, a first local memory 120a, a first local memory store unit 111b, and a first local memory load unit 111a.

[0249] Similarly, the second neural core 100b may include a second neural core store unit 112d, a second neural core load unit 112c, a second local memory 120b, a second local memory store unit 111d, and a second local memory load unit 111c.

[0250] The second neural core store unit 112d of the second neural core 100b generates an L1 sink request signal (S10).

[0251] The L1 sync request signal can be maintained in a stall state until the L1 sync generate signal arrives, in other words, the L1 sync request signal can be generated in a state ready for synchronization.

[0252] The fourth neural core load unit 112f of the fourth neural core 100d may generate a receive L2 sink (S11).

[0253] When there are multiple neural cores, the timing of synchronization preparation for each may differ. Naturally, a receive L2 sink may be generated early, such as in the fourth neural core 100d.

[0254] Next, the second local memory store unit 111d stores the data in the second local memory 120b (S12, circle 1). Subsequently, the second local memory store unit 111d transmits an L1 sync generation signal to the second neural core store unit 112d (S13, circle 2). At this time, the L1 sync generation signal may be transmitted using an L1 sync path. This may synchronize the L1 sync request signal of the second neural core store unit 112d.

[0255] The second neural core store unit 112d may then broadcast the send L2 sink to the first neural core load unit 112a of the first neural core 100a, the third neural core load unit 112e of the third neural core 100c, and the fourth neural core load unit (112f) of the fourth neural core 100d (S14, S15, S16, circle 3). At this time, the send L2 sink may be transmitted via the L2 sink path 300.

[0256] At this time, the fourth neural core 100d, which has already generated a receive L2 sync, is immediately synchronized and loads (S17).

[0257] In contrast, the first neural core 100a and the third neural core 100c may perform loading operations (S19, circles 4 and 5) when a receive L2 sink is generated (S18).

[0258] The loading operation may involve the first neural core load unit 112a making a data request to the second local memory 120b via the local interconnection 200 (circle 4) and receiving a data reply to the request (circle 5).

[0259] Similarly, the third neural core 100c may also perform a load operation (S21) when a receive L2 sink is generated (S20).

[0260] In this embodiment, neither the L2 (level 2) synchronization nor the L1 (level 1) synchronization is managed by a control processor, but rather each element executes in parallel, which can provide significant advantages in terms of latency and efficiency.

[0261] FIG. 23 is a diagram illustrating the command set structure of a neural processing device according to some embodiments of the present invention.

[0262] 23, an instruction set architecture (ISA) of a neural processing device according to some embodiments of the present invention may include an operation code (opcode), a source register (Src0), an L1 sink target (Target for L1 SYNC), an L2 sink target (Target for L2 SYNC), an L3 sink target (Target for L3 SYNC), and a branch end (BE). In other words, the instruction set architecture may include all sink targets from level 1 to level 3.

[0263] FIG. 24 is a block diagram illustrating the software hierarchy of a neural processing device according to some embodiments of the present invention.

[0264] Referring to FIG. 24, the software hierarchy of a neural processing device according to some embodiments of the present invention may include a DL (Deep Learning) framework 10000, a compiler stack 20000, and a back-end module 30000.

[0265] The DL framework 10000 may refer to a framework for a deep learning model network used by a user. For example, a trained neural network may be generated using a program such as TensorFlow (registered trademark) or PyTorch (registered trademark).

[0266] The compiler stack 20000 may include an adaptation layer 21000, a compute library 22000, a front-end compiler 23000, a back-end compiler 24000, and a runtime driver 25000.

[0267] The adaptation layer 21000 may be a layer that interfaces with the DL framework 10000. The adaptation layer 21000 may quantize the user's neural network model generated by the DL framework 10000 and perform graph modification. The adaptation layer 21000 may also convert the model type to a required type.

[0268] The front-end compiler 23000 may convert various neural network models and graphs transmitted from the adaptation layer 21000 into a uniform intermediate representation (IR), which may then be a pre-defined representation that is easy for the back-end compiler 24000 to handle.

[0269] The IR of the front-end compiler 23000 can be subjected to preliminary optimization in the graph dimension, and the front-end compiler 23000 can finally generate the IR by converting it into a layout optimized for hardware.

[0270] The back-end compiler 24000 optimizes the IR converted by the front-end compiler 23000 and converts it into a binary file for use by the runtime driver. The back-end compiler 24000 can divide the job at a scale that matches the details of the hardware and generate optimized code.

[0271] The operation library 22000 may store, among other operations, template operations designed in a form suitable for hardware, and provide the hardware-required template operations to the backend compiler 24000 so that optimized code can be generated.

[0272] The runtime driver 25000 may continue to monitor during operation and may execute the operation of the neural network device according to some embodiments of the present invention. Specifically, it may be responsible for executing the interface of the neural network device.

[0273] The back-end module 30000 may include an ASIC (Application Specific Integrated Circuit) 31000, an FPGA (Field Programmable Gate Array) 32000, and a C-model 33000. The ASIC 31000 may refer to a hardware chip determined by a predetermined design method. The FPGA 32000 may be a programmable hardware chip. The C-model 33000 may refer to a model that is realized by imitating hardware in software.

[0274] The back-end module 30000 may utilize the binary code generated by the compiler stack 20000 to perform various operations and derive results.

[0275] FIG. 25 is a conceptual diagram illustrating deep learning operations performed by neural processing devices according to some embodiments of the present invention.

[0276] Referring to FIG. 25, the artificial neural network model 40000 is an example of a machine learning model, which is a statistical learning algorithm or a structure that executes the algorithm, realized based on the structure of a biological neural network in machine learning technology and cognitive science.

[0277] The artificial neural network model 40000 may represent a machine learning model with problem-solving capabilities, in which nodes, which are artificial neurons formed by synaptic connections in a network, learn to reduce the error between a correct output corresponding to a specific input and an inferred output by repeatedly adjusting synaptic weights, as in a biological neural network. For example, the artificial neural network model 40000 may include any probability model, neural network model, etc. used in artificial intelligence learning methods such as machine learning and deep learning.

[0278] Neural processing devices according to some embodiments of the present invention may perform operations implementing such a form of artificial neural network model 40000. For example, the artificial neural network model 40000 may receive an input video and output information about at least a portion of an object contained in the input video.

[0279] The artificial neural network model 40000 is implemented as a multilayer perceptron (MLP) consisting of multiple nodes and connections between them. The artificial neural network model 40000 according to this embodiment can be implemented using one of various artificial neural network model structures, including an MLP. As shown in FIG. 25, the artificial neural network model 40000 includes an input layer 41000 that receives an input signal or data 40100 from the outside, an output layer 44000 that outputs an output signal or data 40200 corresponding to the input data, and n hidden layers 42000-43000 (where n is a positive integer) that are positioned between the input layer 41000 and the output layer 44000 and receive signals from the input layer 41000, extract characteristics, and transmit them to the output layer 44000. The output layer 44000 receives signals from the hidden layers 42000-43000 and outputs them to the outside.

[0280] There are two learning methods for the artificial neural network model 40000: a supervised learning method, in which the model learns to optimize problem solving by inputting a teacher signal (correct answer), and an unsupervised learning method, in which no teacher signal is required.

[0281] The neural processing device can directly generate training data for training the artificial neural network model 40000 through simulation. In this way, multiple input variables and corresponding multiple output variables are matched to the input layer 41000 and output layer 44000 of the artificial neural network model 40000, respectively, and the synaptic values ​​between nodes included in the input layer 41000, hidden layers (42000-43000), and output layer 44000 are adjusted, so that the artificial neural network model 40000 can be trained to extract the correct output corresponding to a specific input. Through this training process, it is possible to understand the characteristics hidden in the input variables of the artificial neural network model 40000, and to adjust the synaptic values ​​(or weights) between nodes of the artificial neural network model 40000 so as to reduce the error between the output variables calculated based on the input variables and the target output.

[0282] FIG. 26 is a conceptual diagram for explaining the learning and inference operations of a neural network of a neural processing device according to some embodiments of the present invention.

[0283] 26, in the training phase, a large number of training materials TD are forwarded to the artificial neural network model NN and then backward. This adjusts the weights and biases of each node of the artificial neural network model NN, thereby enabling training to be performed to derive increasingly accurate results. In this way, the artificial neural network model NN can be transformed into a trained neural network model NN_T through the training process.

[0284] In the inference phase, new data ND may be input to the trained neural network model NN_T. The trained neural network model NN_T may derive result data RD using the new data ND as input and the weights and biases already trained. For such result data RD, it may be important to consider what learning materials TD were used in the training process and how many learning materials TD were used.

[0285] Hereinafter, a method for synchronizing a neural processing device according to some embodiments of the present invention will be described with reference to Figures 17, 19, 20, 27, and 28. Portions that overlap with the previous embodiments will be simplified or omitted.

[0286] FIG. 27 is a flowchart illustrating a method for synchronizing a neural processing device according to some embodiments of the present invention, and FIG. 28 is a flowchart illustrating in detail the L3 sink target storage step and the FIFO provision step in FIG. 27.

[0287] Referring to FIG. 27, the first neural processor generates an L3 sink target (S100).

[0288] Specifically, referring to FIG. 17, the L3 sink target Sm_V may be a signal generated by each neural processor that sends a synchronization signal. That is, the L3 sink target Sm_V may include, for example, four fields. This may be due to the fact that there are four neural processors in the same set. Each field of the L3 sink target Sm_V may correspond to the first to fourth virtual IDs (VP0 to VP3). That is, if 1, 0, 1, and 1 are written in the L3 sink target Sm_V, the first to fourth virtual IDs (VP0 to VP3) may correspond to 1, 1, 0, and 1, respectively, in reverse order.

[0289] Referring again to FIG. 27, the L3 sink target and the VPID table are used to identify the second neural processor that is the intended recipient (S200).

[0290] 17, the neural processor that is to send the synchronization signal from the L3 sink target Sm_V can check the physical ID of the corresponding neural processor from the VPID table (TB_VTP) after the L3 sink target Sm_V identifies the virtual ID of the neural processor to which the synchronization signal from the L3 sink target Sm_V should be transmitted using the first, second, and fourth virtual IDs (VP0, VP1, VP3).The neural processor can check the actual address only after checking the physical ID.

[0291] Since the VPID table TB_VTP has values ​​of 3, 0, 1, and 2, it can be seen that the physical IDs of the first, second, and fourth virtual IDs (VP0, VP1, VP3) are 2, 1, and 3, respectively. In other words, the second to fourth neural processors PP1 to PP3 can be neural processors that receive a synchronization signal from the L3 sink target Sm_V.

[0292] Referring again to FIG. 27, the synchronization signal from the L3 sink target is stored in the semaphore memory of the second neural processor via the L3 sink channel (S300).

[0293] 19, the first semaphore memory smp1 includes first to fourth fields, which may correspond to the first to fourth neural processors PP0 to PP3, respectively. That is, the first to fourth fields may be arranged in the same order as the physical IDs of the first to fourth neural processors PP0 to PP3.

[0294] That is, the first field of the first semaphore memory smp1 is the part for the first neural processor 1000, and if a synchronization signal from the L3 sink target Sm_V is received from the first neural processor 1000, it is expressed as 1, otherwise it is expressed as 0. Of course, it may also be expressed the other way around.

[0295] Referring again to FIG. 27, the value of the semaphore memory is provided to the second neural processor in a FIFO manner (S400).

[0296] 20, a neural processing device according to some embodiments of the present invention may include first to fourth FIFO buffers B1 to B4 corresponding to the first to fourth fields, respectively. The first to fourth FIFO buffers may provide the values ​​of the first to fourth fields of the first semaphore memory smp1 to the first neural processor 1000 in a FIFO manner.

[0297] Steps S300 and S400 will be described in detail with reference to FIG. A synchronization signal from the L1 sink target of the first neural processor is stored in a first field of a semaphore memory of the second neural processor (S310), and the first field value of the semaphore memory is provided to the second neural processor in a FIFO manner (S410).

[0298] Similarly, a synchronization signal from the L1 sink target of the second neural processor is stored in the second field of the semaphore memory of the second neural processor (S320), and the value of the second field of the semaphore memory is provided to the second neural processor in a FIFO manner (S420).

[0299] A synchronization signal from the L1 sink target of the third neural processor is stored in a third field of the semaphore memory of the second neural processor (S330), and the value of the third field of the semaphore memory is provided to the second neural processor in a FIFO manner (S430).

[0300] A synchronization signal from the L1 sink target of the fourth neural processor is stored in the fourth field of the semaphore memory of the second neural processor (S340), and the value of the fourth field of the semaphore memory is provided to the second neural processor in a FIFO manner (S440).

[0301] That is, each field corresponds to a respective neural processor, and synchronization can be performed in parallel in a FIFO format.

[0302] Referring again to FIG. 27, the second neural processor performs synchronization with the L3 sink target (S500).

[0303] Hereinafter, a method for synchronizing a neural processing device according to some embodiments of the present invention will be described with reference to Figures 21, 22, 29, and 30. Portions that overlap with the previous embodiments will be simplified or omitted.

[0304] FIG. 29 is a flowchart illustrating a method for synchronizing the L1 and L2 levels of a neural processing device according to some embodiments of the present invention, and FIG. 30 is a flowchart illustrating in detail the data request step in FIG. 29.

[0305] 29, data is stored in the local memory of the first neural core (S1100). Then, in the first neural core, the local memory store unit transmits a synchronization signal by the L1 sink target to the neural core store unit (S1200).

[0306] 21 and 22, the second local memory store unit 111d stores data in the second local memory 120b (S12, circle 1). Then, the second local memory store unit 111d transmits an L1 sync generation signal to the second neural core store unit 112d (S13, circle 2). At this time, the L1 sync generation signal may be transmitted via an L1 sync path. This may synchronize the L1 sync request signal of the second neural core store unit 112d.

[0307] Referring again to FIG. 29, the neural core store unit of the first neural core transmits a synchronization signal according to the L2 sink target to the neural core load units of the second to fourth neural cores (S1300).

[0308] 21 and 22, the second neural core store unit 112d may then broadcast the send L2 sink to the first neural core load unit 112a of the first neural core 100a, the third neural core load unit 112e of the third neural core 100c, and the fourth neural core load unit (112f) of the fourth neural core 100d (S14, S15, S16, circle 3). At this time, the send L2 sink may be transmitted via the L2 sink path 300.

[0309] Referring again to FIG. 29, the second to fourth neural core load units request data from the local memory of the first neural core via the local interconnection (S1400).

[0310] 30, the second neural core receives a synchronization signal from the L2 sink target (S1410) and determines whether a receive L2 sink signal has already been generated (S1420). If not, it waits for the receive L2 sink signal to be generated (S1430). If yes, the second neural core requests data from the local memory of the first neural core (S1440).

[0311] Referring again to FIG. 29, the second to fourth neural core load units receive data (S1500).

[0312] The above description merely exemplifies the technical concept of the present embodiment, and various modifications and variations are possible within the scope of the essential characteristics of the present embodiment, if one skilled in the art to which the present embodiment pertains. Therefore, the present embodiment is intended to be illustrative and not to limit the technical concept of the present embodiment, and the scope of the technical concept of the present embodiment is not limited by such examples. The scope of protection of the present embodiment should be interpreted by the following claims, and all technical concepts within the scope equivalent thereto should be interpreted as being included in the scope of the present embodiment.

Claims

1. first and second neural processors; a shared memory shared by the first and second neural processors; a first semaphore memory and a second semaphore memory corresponding to the first neural processor and receiving and storing an L3 sink target, the first semaphore memory and the second semaphore memory being synchronized by the L3 sink target; a global interconnection connecting the first and second neural processors and the shared memory, the global interconnection including an L3 sink channel through which a synchronization signal from the L3 sink target is transmitted; Neural processing device.

2. The global interconnection the L3 sync channel; a data channel for transmitting data between the shared memory and the first and second neural processors; a control channel for transmitting control signals to the first and second neural processors; 2. A neural processing device according to claim 1.

3. the first semaphore memory includes first and second fields corresponding to the first and second neural processors, respectively; 2. A neural processing device according to claim 1.

4. a first FIFO buffer for sequentially transmitting the values ​​of the first field to the first neural processor; 4. A neural processing device according to claim 3.

5. the L3 sink targets include first and second L3 sink targets, the first neural processor generates the first L3 sink target; the second neural processor generates the second L3 sink target; 2. A neural processing device according to claim 1.

6. the L3 sink target includes first and second sink target fields corresponding to the first and second neural processors, respectively; the first and second sink target fields contain information regarding whether the first and second neural processors receive synchronization signals from the L3 sink target; 2. A neural processing device according to claim 1.

7. the first and second sink target fields are sorted in order of the virtual IDs of the first and second neural processors, respectively; 7. A neural processing device according to claim 6.

8. the first neural processor uses the L3 sink target and a VPID table to identify a physical ID of a neural processor that receives a synchronization signal from the L3 sink target; The VPID table includes information for converting between the virtual ID and the physical ID.

8. A neural processing device according to claim 7.

9. The L3 sink target is included in an instruction set architecture (ISA).

2. A neural processing device according to claim 1.

10. The first and second neural processors at least one neural core; a local interconnection for transmitting data between the at least one neural core; 2. A neural processing device according to claim 1.

11. The first and second neural processors and an L2 sink path through which a synchronization signal from an L2 sink target is transmitted to perform synchronization between the at least one neural core.

11. A neural processing device according to claim 10.

12. Each of the at least one neural core comprises: a processing unit that receives input activations and weights, performs deep learning operations, and outputs output activations; a local memory for temporarily storing the input activations, the weights, and the output activations; 11. A neural processing device according to claim 10.

13. at least one neural processor; Shared memory and a global interconnection connecting the at least one neural processor and the shared memory, the global interconnection being used for L3 synchronization of the neural processors; The neural processor at least one neural core; a local interconnection connecting the at least one neural core; an L2 sink path used for L2 synchronization of the at least one neural core; The neural core comprises: a process unit for performing computational tasks; a local memory for temporarily storing data; an L1 sink path used for L1 synchronization between the local memory and the process unit; Neural processing device.

14. The global interconnection a data channel for transmitting data between said at least one neural processor and said shared memory; a control channel for transmitting control signals between said at least one neural processor; an L3 sync channel used for said L3 synchronization; 14. A neural processing device according to claim 13.

15. The local interconnection transmitting data between said at least one neural core; 14. A neural processing device according to claim 13.

16. a data path used for data exchange between the local memory and an element including the processing unit; 14. A neural processing device according to claim 13.

17. the at least one neural processor includes first and second neural processors; and further including first and second semaphore memories corresponding to the first and second neural processors, respectively, for receiving and storing a synchronization signal corresponding to an L3 sink target, wherein synchronization of the first and second neural processors is performed according to values ​​of the first and second semaphore memories.

14. A neural processing device according to claim 13.

18. the first semaphore memory includes first and second fields corresponding to the first and second neural processors, respectively; a first FIFO buffer for sequentially transmitting the values ​​of the first field to the first neural processor; 18. A neural processing device according to claim 17.

19. the first neural processor transmits an instruction set structure; the instruction set structure includes an operation code, an L3 sync target for the L3 synchronization, an L2 sync target for the L2 synchronization, and an L1 sync target for the L1 synchronization; 18. A neural processing device according to claim 17.

20. 1. A method of synchronizing a neural processing device including first and second neural processors, comprising: the first neural processor generates L3 sink targets for L3 synchronization, the L3 sink targets being sorted in order of virtual IDs of the first and second neural processors; The physical ID of the second neural processor is identified using the L3 sink target and a VPID table, the VPID table being a conversion table between the virtual ID and the physical ID of the neural processor; storing a synchronization signal from the L3 sink target in a first semaphore memory of the second neural processor via an L3 sink channel of a global interconnection; the second neural processor performing L3 synchronization according to the value of the first semaphore memory; A method for synchronizing neural processing devices.

21. the first semaphore memory includes first and second fields corresponding to the first and second neural processors, respectively; 21. A method for synchronizing neural processing devices according to claim 20.

22. performing the L3 synchronization providing the value of the first field to the second neural processor in a FIFO manner; providing the value of the second field to the second neural processor in a FIFO manner; 22. A method for synchronizing neural processing devices according to claim 21.

23. the virtual IDs include first and second virtual IDs corresponding to the first and second neural processors, respectively; 21. A method for synchronizing neural processing devices according to claim 20.

24. The first neural processor first and second neural cores; a local interconnection for transmitting data between the first and second neural cores; an L2 sink path for transmitting a synchronization signal by an L2 sink target between the first and second neural cores; 21. A method for synchronizing neural processing devices according to claim 20.

25. The first neural core comprises: a first processing unit that receives a first input activation and a first weight, performs a deep learning operation, and outputs a first output activation; a first local memory for temporarily storing the first input activations, the first weights, and the first output activations; a first L1 sink path for transmitting a synchronization signal by an L1 sink target between the first local memory and the first process unit; The second neural core comprises: a second processing unit that receives the second input activations and the second weights, performs a deep learning operation, and outputs a second output activation; a second local memory for temporarily storing the second input activations, the second weights, and the second output activations; a second L1 sink path for transmitting a synchronization signal by the L1 sink target between the second local memory and the second process unit; 25. A method for synchronizing neural processing devices according to claim 24.

26. storing data in the first local memory; transmitting a synchronization signal from the L1 sink target via the first L1 sink path within the first neural core; the first neural core transmits a synchronization signal by the L2 sink target to the second neural core via the L2 sink path; the second neural core further receiving data via the local interconnection.

26. A method for synchronizing neural processing devices according to claim 25.

27. 1. A neural processing device comprising: first and second neural cores; a local interconnection connecting said first and second neural cores; and an L2 sink path used for L2 synchronization of said first and second neural cores, the first neural core includes a first process unit that executes a calculation operation, a first local memory that temporarily stores data input to and output from the first process unit, and a first L1 sink path that is used for L1 synchronization between the first local memory and the first process unit; a second neural core including a second process unit that executes a calculation operation, a second local memory that temporarily stores data input to and output from the second process unit, and a second L1 sink path that is used for L1 synchronization of the second local memory and the second process unit, storing data in the first local memory; transmitting a synchronization signal by an L1 sink target through the first L1 sink path within the first neural core; the first neural core transmits a synchronization signal by an L2 sink target to the second neural core via the L2 sink path; the second neural core further receiving data via the local interconnection. A method for synchronizing neural processing devices.

28. the first neural core further includes a first LSU that moves data between the first local memory and the local interconnection; The first LSU includes a first local memory store unit that performs storage of the first local memory, and a first neural core store unit that performs storage from the first neural core to an external device; transmitting a synchronization signal from the L1 sink target through the first L1 sink path within the first neural core; the first local memory store unit transmitting a synchronization signal by the L1 sink target to the first neural core store unit; 28. A method for synchronizing neural processing devices according to claim 27.

29. the second neural core further includes a second LSU that moves data between the second local memory and the local interconnection; the second LSU includes a second neural core load unit that performs an external load on the second neural core; transmitting a synchronization signal by the L2 sink target, the first neural core store unit transmitting a synchronization signal by the L2 sink target to the second neural core load unit; 29. A method for synchronizing neural processing devices according to claim 28.

30. the neural processing device includes a first neural processor including the first and second neural cores, the local interconnection, and the L2 sink path, a second neural processor different from the first neural processor, a global interconnection for transmitting data between the first and second neural processors, and first and second semaphore memories corresponding to the first and second neural processors, respectively; the global interconnection includes a data channel, a control channel, and an L3 sink channel through which data, control signals, and synchronization signals by an L3 sink target are transmitted between the first and second neural processors, respectively; the first neural processor generates the L3 sink target; storing a synchronization signal from the L3 sink target in the second semaphore memory; the second neural processor performing synchronization according to the value of the second semaphore memory; 28. A method for synchronizing neural processing devices according to claim 27.

Images