Latency processing unit

Abstract

A latency processing unit is provided. A latency processing unit according to one embodiment includes: a plurality of MAC (Multipliers-Accumulators) trees that perform matrix multiplication operations for at least one partition among a plurality of partitions that realize an artificial intelligence model; simplified memory access that connects each of the plurality of MAC trees to a high-bandwidth memory storing the at least one partition through a plurality of channels; a vector execution engine that performs additional operations on the calculation results of the plurality of MAC trees; a local memory unit that stores the calculation results and activation values ​​of the vector execution engine; and an instruction word scheduling unit that schedules operations of the plurality of MAC trees and the vector execution engine.

Description

Technical Field

[0001] Embodiments of the present invention relate to a latency processing unit, and more particularly, to a latency processing unit capable of optimizing the amount of arithmetic processing and latency for the operation of an ultra-large-scale artificial intelligence model by maximizing the use of external memory bandwidth through Streamlined Memory Access and Streamlined eXecution Engine.

Background Art

[0002] With the development of machine learning technology in recent years, the application range of artificial intelligence applications has been expanding. Such artificial intelligence models require diverse and complex operations to obtain higher accuracy, and the scale of the models has been increasing continuously, resulting in the emergence of ultra-large-scale artificial intelligence models. Along with this, the importance of accelerators for processing such ultra-large-scale artificial intelligence models has been increasing.

[0003] In particular, natural language processing applications such as artificial intelligence translators and chatbots mainly use a transformer model-based generation model that uses an attention operation. The transformer model-based generation model has a temporal dependency, and the main attention operation used has a dependency between each operation, so it is difficult to parallelize and accelerate this model. Furthermore, for applications using such generation models, real-time services are extremely important, so not only the amount of arithmetic processing but also the latency is very important.

[0004] However, hardware accelerators that maximize computational processing power by utilizing data parallelism, like existing GPUs (Graphics Processing Units), are not suitable for accelerating applications using ultra-large-scale artificial intelligence models. Furthermore, because ultra-large-scale artificial intelligence models have extremely large data sizes, data bottlenecks can occur with the structure of existing hardware accelerators. Therefore, a new type of hardware accelerator is needed to accelerate ultra-large-scale artificial intelligence models. [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] This latency processing unit optimizes the computational load and latency for computations on ultra-large-scale artificial intelligence models by maximizing the use of external memory bandwidth through streamlined memory access and a streamlined execution engine.

[0006] The technical problems of the present invention should not be limited to those described above, and other technical problems not described will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0007] The present invention provides a latency processing unit that includes multiple MAC (Multipliers-Accumulators) trees that perform matrix multiplication operations for at least one of multiple partitions that realize an artificial intelligence model; simplified memory access that connects each of the multiple MAC trees to a high-bandwidth memory storing the at least one partition via multiple channels; a vector execution engine that performs additional operations on the calculation results of the multiple MAC trees; a local memory unit that stores the calculation results and activation values ​​of the vector execution engine; and an instruction scheduling unit that schedules operations between the multiple MAC trees and the vector execution engine.

[0008] According to one side, each of the multiple MAC trees may be characterized by being connected in a one-to-one correspondence with multiple channels of the high-bandwidth memory.

[0009] In other respects, the simplified memory access may be characterized by comprising only a read interface between the multiple MAC trees and the high-bandwidth memory, and only a write interface between the local memory unit and the high-bandwidth memory.

[0010] In other respects, the local memory unit may be characterized by reading activation values ​​and model parameters from the high-bandwidth memory and transmitting them to the multiple MAC trees and the vector execution engine.

[0011] Another aspect of the local memory unit may be characterized by copying and transmitting the same activation value to each of the multiple MAC trees.

[0012] Another aspect of the high-bandwidth memory may be that the weight matrix is ​​stored such that it is mapped to the number of channels corresponding to the number of MAC trees in the column direction.

[0013] In other respects, the instruction scheduling unit may include an address-based instruction dependency determination and scheduling controller and a multibank buffer address state table, wherein the address-based instruction dependency determination and scheduling controller determines the dependency between the multiple MAC trees and the instruction words of the vector execution engine by referring to the operator addresses and result addresses of the instruction words stored in the multibank buffer address state table.

[0014] In other respects, the latency processing unit may further include a PCIe (Peripheral Component Interconnect express) interface connecting the latency processing unit to the host computer in order to receive at least one input from the host computer, consisting of an instruction word, an input value, and a weight.

[0015] In other respects, the latency processing unit may further include a P2P (Peer to Peer) interface for connecting to a plurality of latency processing units, each corresponding to at least one different partition among the plurality of partitions.

[0016] Another aspect of this is that the latency processing unit may be implemented using a PIM (Processing-in-memory) chip capable of performing calculations within the high-bandwidth memory.

[0017] Another aspect of the latency processing unit may be that it is implemented using a buffer chip with a PNM (Processing-near-memory) structure capable of performing operations on multiple high-bandwidth memories, each of which stores the multiple partitions.

[0018] According to another aspect, the plurality of MAC trees are implemented on a PIM-structured chip that can perform operations inside the high-bandwidth memory, and the simplified memory access, the vector execution engine, the local memory unit, and the instruction word scheduling unit may be implemented on a buffer chip of a PNM structure that can perform operations on a plurality of high-bandwidth memories in which the plurality of partitions are respectively stored.

[0019] Specific matters of other embodiments are included in the detailed description and the drawings.

Advantages of the Invention

[0020] By maximizing the use of external memory bandwidth through Streamlined Memory Access and Streamlined eXecution Engine, it is possible to optimize the amount of arithmetic processing and the delay time for the operation of ultra-large-scale artificial intelligence models.

[0021] The effects of the present invention should not be limited to the above-described effects, and other effects not described will be clearly understood by those skilled in the art from the description of the claims.

Brief Description of the Drawings

[0022] [Figure 1] It is a diagram showing an example of the structure of a latency processing unit in an embodiment of the present invention. [Figure 2] It is a diagram showing an example of an implementation model of an LPU in an embodiment of the present invention. [Figure 3] It is a diagram showing an example of an implementation model of an LPU in an embodiment of the present invention. [Figure 4] It is a diagram showing an example of an implementation model of an LPU in an embodiment of the present invention. [Figure 5] It is a diagram showing an example of an implementation model of an LPU in an embodiment of the present invention. [Figure 6]A diagram for explaining the weight matrix data mapping of the high-bandwidth memory for matrix multiplication operations of the latency processing unit in one embodiment of the present invention. [Figure 7] A diagram for explaining the high-bandwidth memory interface included in the latency processing unit in an embodiment of the present invention. [Figure 8] A diagram for explaining the reconfigurable multi-functional arithmetic unit included in the latency processing unit in an embodiment of the present invention. [Figure 9] A diagram for explaining the configuration of the address-based unordered multi-unit scheduler included in the latency processing unit in an embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0023] The advantages, features, and methods for achieving them of the present invention will become apparent by referring to the embodiments described in detail below together with the accompanying drawings. However, the present invention should not be limited to the embodiments disclosed below and may be implemented in various different forms. These embodiments are provided only to make the disclosure of the present invention complete and to fully inform those with ordinary knowledge in the technical field to which the present invention pertains of the scope of the invention. The present invention must be defined only based on the scope of the claims. Throughout the specification, the same reference numerals refer to the same components.

[0024] When one component is referred to as "connected to" or "coupled to" another component, it includes both cases where it is directly connected or coupled to the other component and cases where another component is interposed therebetween. On the contrary, when one component is referred to as "directly connected to" or "directly coupled to" another component, it indicates that no other component is interposed therebetween. "And / or" includes each of the items to be described and all combinations of one or more.

[0025] The terms used herein are for illustrative purposes only and are not intended to limit the invention. In this specification, singular expressions include plural forms unless otherwise specified in the context. As used in this specification, “comprises” and / or “comprising” means that the components, steps, operations, and / or elements described do not exclude the presence or addition of one or more other components, steps, operations, and / or elements.

[0026] While terms such as "first," "second," etc., are used to describe various components, it goes without saying that these components should not be limited by these terms. Such terms are merely used to distinguish one component from another. Therefore, it goes without saying that the first component referred to below may also be the second component within the technical concept of the present invention.

[0027] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall be used in a way that can be commonly understood by a person of ordinary skill in the art to which this invention pertains. Furthermore, terms defined in commonly used dictionaries shall not be interpreted ideally or excessively unless otherwise clearly defined.

[0028] Figure 1 shows an example of the structure of a latency processing unit in an embodiment of the present invention.

[0029] Referring to Figure 1, the LPU (Latency Processing Unit) 100 according to the embodiment of the present invention may include an SMA (Streamlined Memory Access) 110, an OIU (Operand Issue Unit) 120, an SXE (Streamlined eXecution Engine) 130, a VXE (Vector eXecution Engine) 140, an LMU (Local Memory Unit) 150, an ISU (Instruction Scheduling Unit) 160, a PCIe interface (PCIe (Peripheral Component Interconnect express) Interface) 170, and a P2P interface (Peer to Peer) Interface) 180.

[0030] The SMA110 may be a special DMA (Direct Memory Access). For example, the SMA110 may connect all channels of the HBM200 (for example, 32) to an execution engine (for example, SEE130) to transmit FP16 (half precision floating point) data at maximum bandwidth. The SMA110 may be designed with a deep FIFO (First In First Out) to transmit contiguous memory requests based on preloaded memory (MEM) instruction words. Hardware-aware memory mapping reduces latency by eliminating matrix modification or transpose operations. Thus, the SMA110 can stream data received at maximum burst size to the execution engine with minimal latency. The SMA110 can also efficiently perform matrix transpose using strobe signals. Streaming data may include parameters for vector matrix operations (e.g., weights, bias) and other vector-related operations (e.g., gamma / beta, embedding).

[0031] The OIU120 may reconcile data streamed by the SMA110 (e.g., the first operator) with input in on-chip memory (e.g., the second operator) before issuing it to the execution engine. Based on the EXE instruction, the OIU120 may configure the execution engine and generate microcode to determine the target engine for the operand. The OIU120 may also have a reuse buffer to eliminate read wait times for static operands (e.g., input vectors) and an asymmetric buffer to maintain vectorized data used as scalars (e.g., biases). Thus, the appropriate operand is almost always prefetched and ready to be immediately issued to the execution engine.

[0032] As the primary computing hardware for the LPU100, the SXE130 may be designed to maximize the use of the input bandwidth for performing vector-matrix multiplication (V·M) operations such as attention, 1D convolution, and feedforward networks. The SXE130 may include a number of MAC (Multiply-Accumulate) trees 131 that can match the receiving bandwidth of the HBM200 with the processing bandwidth. For example, if 1024 elements are received from the HBM200 per cycle, the receiving bandwidth and processing bandwidth can be matched by using 16 MAC trees 131, each having 64 input vectors. Alternatively, a MAC tree 131 with 64 input vectors may consist of 64 multipliers and 63 adders.

[0033] Multiple MAC trees 131 may perform matrix multiplication operations and may be connected channel by channel via external memory, such as the high-bandwidth memory HBM200 and SMA110. Specifically, by connecting one of the multiple MAC trees 131 to the HBM200 via one channel, the transfer bandwidth between the LPU 100 and the HBM200 is maximized, allowing the matrix multiplication operations required for ultra-large-scale artificial intelligence models to be executed without bottlenecking. Therefore, the number of multiple MAC trees 131 and the number of memory channels in the HBM200 can be configured to be the same.

[0034] The results of matrix multiplication operations on multiple MAC trees 131 may be provided to VXE140. VXE140 may be implemented using a user-specified low-latency ALU (Arithmetic Logic Unit) and may perform vector operations such as token embedding, softmax, normalization, and residual calculations. Because these vector operations occur relatively infrequently, hardware resources can be reduced with negligible performance loss by adjusting the fan-in to this path in OIU120. VXE140 may receive the results of operations on multiple MAC trees 131 and receive activation values ​​from LMU150 to perform subsequent operations. VXE140 may be configured to include various combinations of arithmetic units by including multiple multifunction arithmetic data paths.

[0035] The LMU150 may transmit activation values ​​to multiple MAC trees 131 and VXE140. In this case, the LMU150 may copy and transfer the activation values ​​to transmit the same activation values ​​to multiple MAC trees 131. The LMU150 may also store the results of calculations performed by the multiple MAC trees 131 and VXE140. In other words, the LMU150 may function within the LPU100 as an internal buffer corresponding to the HBM200. In this case, the LPU100 may store activation values ​​or model parameters with high reuse rates in matrix multiplication operations in the LMU150, and weights with low reuse rates in the HBM200. The LMU150 may be implemented as a 4MB multibank register file with scalar vector separation for fast, high-bandwidth access to input, output, and intermediate data. The LMU150 may also be a multiport to support simultaneous read and write operations between the OIU120 and the execution engine during the write and storage phases.

[0036] The ISU160 may control the overall execution flow of the LPU100. The ISU160 may utilize the PIC (Parallel Instruction Chaining) method, which allows for the sequential execution of dependent instructions. PIC separates instructions requiring independent hardware into groups of dependent instructions (e.g., memory (MEM) instructions, execution (EXE) instructions, network (NET) instructions), so that all instructions are executed in parallel with the instruction chain of each group, achieving low control overhead and latency savings. The ISU160 may also update control registers (e.g., tokens and hierarchy numbers) for engine execution. An internal scheduler may support the non-sequential execution of SXE130 and VXE140 to maximize hardware utilization, and a robust scoreboard may be designed to handle data risk. For example, the ISU160 may schedule multiple MAC trees 131 and VXE140 to perform calculations simultaneously. Furthermore, the ISU160 can improve computational processing capacity and latency by minimizing the idle time of each arithmetic unit and memory access unit by pre-executing non-dependent instructions to maximize parallel processing.

[0037] The LPU100 may be connected to a host computer via a PCIe interface 170, and may receive instructions necessary for its operation, input values ​​and weights of a super-large-scale artificial intelligence model from the host computer, perform calculations, and then transmit the results back to the host computer.

[0038] The LPU100 may be scaled out as a cluster of multiple LPUs connected via the P2P interface 180. The extended cluster structure can further improve the acceleration of computations for ultra-large-scale artificial intelligence models.

[0039] Figures 2-5 show examples of LPU implementation models in embodiments of the present invention. In the embodiment shown in Figure 1, an example of an implementation model using external memory for HBM200 was described. Instead of HBM200, DDR (Double Data Rate) may be used as the external memory. In this case, since large-scale models are difficult to store on a single device, they may be separated into multiple partitions and stored in separate partitions in external memory for multiple devices (multiple LPUs). In this case, synchronization between multiple devices may be required for inference of large-scale models.

[0040] In the embodiment shown in Figure 2, similar to the description of the embodiment in Figure 1, multiple external memories 320 are shown to store multiple partitions 310 of a large-scale model, and multiple LPUs 330 are shown connected in parallel to the multiple external memories 320. One LPU may be implemented on one FPGA (Field Programmable Gate Array), and one partition may be connected in parallel to one FPGA. The transformer structure includes multi-head attention, layer normalization, feedforward, etc., within the decoder layer, but multi-head attention and feedforward may be model-parallelized. In this case, when multi-head attention is completed, one embedding vector may be output as a result. Since one device has only a portion of the embedding vector, multiple devices need to share each embedding vector in order to proceed to the next operation, and synchronization is required. In this case, considering scalability, one LPU may be implemented in a form that has multiple external memories (for example, two or four). As an example, the embodiment in Figure 1 shows an example in which two HBM200s, each storing one partition, are used.

[0041] In the embodiment shown in Figure 3, as an example of a Processing-in-Memory (PIM) model, one LPU is implemented on a PIM chip, and both the partition and the LPU arithmetic unit are integrated on a single chip. In the embodiment of Figure 3, multiple LPUs 410, multiple partitions 310, and multiple LPU arithmetic units 420 that can be implemented on a PIM chip are shown. In this case, each of the multiple LPUs 410 may include one partition and one LPU arithmetic unit.

[0042] The embodiment in Figure 4 shows an example of a PNM (Processing-near-Memory) model. It is difficult to include the configuration for processing all LPU operations within a single PIM chip. The embodiment in Figure 4 shows a model in which multiple memory chips 510 store multiple partitions 310, and a buffer chip 520, such as a PNM chip, includes an LPU arithmetic unit 521 for LPU operations.

[0043] The embodiment shown in Figure 5 illustrates an example of a model in which PIM and PNM are combined. For example, multiple memory chips 610 may store multiple partitions 310. Furthermore, each of the multiple memory chips 610 may implement a PIM-type LPU arithmetic unit 611 as a storage unit such as a MAC tree. In this case, a buffer chip 620 may implement an LPU arithmetic unit 621 for the remaining high-level operations of the LPU using a PNM method.

[0044] Figure 6 illustrates the weight matrix data mapping of a high-bandwidth memory for matrix multiplication operations in a latency processing unit in one embodiment of the present invention.

[0045] Referring to Figure 6, in this embodiment, the LPU 100 has the same number of MAC trees as the number of MAC trees 131, and the same number of memory channels in the SMA 110. Therefore, weight matrix data mapped to a high-bandwidth memory 610 such as HBM200 can be stored so that weight data can be retrieved without accessing other memory channels during matrix multiplication operations of each MAC tree.

[0046] Specifically, the weight matrix data may be stored in the high-bandwidth memory 610 such that the column direction D1 of the weight matrix is ​​mapped to each channel 620-n as many times as there are MAC trees 131. In the weight matrix, matrix multiplication operations can be performed in parallel in the column direction, so each of the MAC trees 131 may read the column direction data from its assigned memory channel 620-n and perform matrix multiplication operations.

[0047] Next, multiple MAC trees 131 may map weight matrix data so that it can be accumulated in the row direction D2 of the weight matrix to complete the final calculation result. The number of row data that can be mapped at one time may be determined according to the bandwidth of the high-bandwidth memory 610, and this may be determined by the size of the tile that multiple MAC trees 131 can process at one time.

[0048] Figure 7 illustrates a high-bandwidth memory interface included in a latency processing unit in one embodiment of the present invention.

[0049] Referring to Figure 7, the SMA110 may connect the LMU150, multiple MAC trees 131, and high-bandwidth memory 610. The SMA110 is not connected to other computing units of the LPU100, and therefore the high-bandwidth memory interface can be minimized in terms of hardware resources.

[0050] Multiple MAC trees 131 and memory channels 620-n may be connected in a one-to-one correspondence. That is, since multiple MAC trees 131 do not need to access channels other than the one directly assigned to them, matrix multiplication operations can be performed without using complex interfaces that consume a lot of resources and have high latency, such as a crossbar interface.

[0051] The SMA110 may consist only of a read interface for multiple MAC trees 131 to read weight matrix data stored in the high-bandwidth memory 610. In other words, since the results of the calculations described below are stored in the high-bandwidth memory 610 via the LMU150, a write interface for multiple MAC trees 131 to the high-bandwidth memory 610 is not configured, thus reducing the amount of hardware resources required.

[0052] Conversely, the SMA110 may only configure a write interface between the LMU150 and the high-bandwidth memory 610. The calculation results stored in the LMU150 as an internal buffer may be transmitted via the SMA110 to be recorded in the high-bandwidth memory 610, and the memory channel to be recorded may be selected using the demultiplexer 710.

[0053] Figure 8 illustrates a reconfigurable multifunction computing unit included in a latency processing unit in one embodiment of the present invention.

[0054] Referring to Figure 8, the VXE140 may include multiple multifunction arithmetic data paths 810 and 820, which may be connected to an operator / result value chain network 830 to form various combinations of arithmetic units.

[0055] As shown in Figure 8, the multi-function computation data paths 810 and 820 may include various computation units necessary for, for example, lookup table-based nonlinear activation functions and masking operations. The configuration of the computation units of the reconfigurable multi-function computation data paths 810 and 820 shown in Figure 8 is merely illustrative, and it goes without saying that any additional computation units necessary for large-scale model computations may be included in the multi-function computation data paths 810 and 820. The results computed by the VXE140 may be transmitted to the LMU150.

[0056] Figure 9 is a diagram illustrating the configuration of an address-based non-sequential multi-unit scheduler included in a latency processing unit in one embodiment of the present invention.

[0057] Referring to Figure 9, the ISU 160, which is an address-based non-sequential multi-unit scheduler included in the LPU 100 according to this embodiment, may include an address-based instruction dependency determination and scheduling controller 910, multiple instruction issue controllers 921, 922, 923, 924, a multi-bank buffer address state table 950, an instruction buffer 960, and result address state update logic 970, and a multi-unit instruction dispatcher 980.

[0058] The ISU160 may operate each arithmetic unit and data transfer unit simultaneously through the address-based instruction dependency determination and scheduling controller 910 and multiple instruction issue controllers 921, 922, 923, and 924. In this case, the ISU160 may change the state of the operator address and result address in the multibank buffer address state table 950 for instructions executed by each arithmetic unit to 1.

[0059] The multibank buffer address state table 950 may change the state of the result address of an instruction that has finished executing to 0 through the result address state update logic 970.

[0060] The address-based instruction dependency determination and scheduling controller 910 may refer to the address status through the multi-bank buffer address status table 970 to determine the dependency between the instruction to be executed and the instruction currently being executed, as well as the dependency between the instruction to be executed. This allows for pre-processing of instruction words without dependencies, thereby minimizing the idling time of each arithmetic unit and data transfer unit.

[0061] The address-based instruction dependency determination and scheduling controller 910 included in the ISU160 may load and process instructions from the instruction buffer 960. At this time, the address-based instruction dependency determination and scheduling controller 910 may execute loop instructions and decode other instructions to distinguish them, and then transmit them through the multi-unit instruction dispatcher 980 to the device-to-device instruction issue controller 921, the direct memory access instruction issue controller 922, the MAC tree instruction issue controller 923, and the reconfigurable multifunction arithmetic unit instruction issue controller 924.

[0062] The ISU160 may receive and store instruction words for the LPU100 from the host computer via the PCIe interface 170, and store the current state of the LPU100 in a state register. The host computer may check the state register via the PCIe interface 170.

[0063] As described above, embodiments of the present invention have been explained based on the drawings. However, a person with ordinary skill in the art to which the present invention belongs will understand that the present invention can be implemented in other specific forms without changing the technical idea or essential features. Therefore, the embodiments described above are illustrative in all respects and should not be understood as limiting.

Claims

1. Multiple MAC (Multipliers-Accumulators) trees that perform matrix multiplication operations for at least one of the multiple partitions that make up the artificial intelligence model, Simplified memory access that connects each of the multiple MAC trees to a high-bandwidth memory containing at least one partition via multiple channels, A vector execution engine that performs additional calculations on the calculation results of the aforementioned multiple MAC trees, A local memory unit that stores the calculation results and activation values ​​of the vector execution engine, The instruction word scheduling unit schedules the calculations of the plurality of MAC trees and the vector execution engine, A latency processing unit, including [a specific component].

2. The latency processing unit according to claim 1, characterized in that each of the plurality of MAC trees is connected in a one-to-one correspondence with a plurality of channels of the high-bandwidth memory.

3. The latency processing unit according to claim 1, characterized in that the simplified memory access constitutes only a read interface between the plurality of MAC trees and the high-bandwidth memory, and constitutes only a write interface between the local memory unit and the high-bandwidth memory.

4. The latency processing unit according to claim 1, characterized in that the local memory unit reads activation values ​​and model parameters from the high-bandwidth memory and transmits them to the plurality of MAC trees and the vector execution engine.

5. The latency processing unit according to claim 4, characterized in that the local memory unit copies and transmits the same activation value to each of the plurality of MAC trees.

6. The latency processing unit according to claim 1, characterized in that the high-bandwidth memory stores a weight matrix such that it is mapped to the plurality of channels in the column direction as many times as there are MAC trees.

7. The aforementioned instruction scheduling unit, Address-based instruction dependency determination and scheduling controller, Multibank buffer address state table, Includes, The address-based instruction dependency determination and scheduling controller determines the dependency between the multiple MAC trees and the instruction words of the vector execution engine by referring to the operator address and result address of the instruction words stored in the multibank buffer address state table. The latency processing unit according to claim 1, characterized by the above.

8. A PCIe (Peripheral Component Interconnect express) interface connects the host computer and the latency processing unit to receive at least one input from the host computer, consisting of an instruction word, an input value, and a weight. The latency processing unit according to claim 1, further comprising:

9. A P2P (Peer-to-Peer) interface for connecting to multiple latency processing units corresponding to at least one different partition among the aforementioned multiple partitions. The latency processing unit according to claim 1, further comprising:

10. The latency processing unit according to claim 1, characterized in that it is implemented in a chip with a PIM (Processing-in-memory) structure that can perform operations within the high-bandwidth memory.

11. The latency processing unit according to claim 1, characterized in that it is implemented by a buffer chip with a PNM (Processing-near-memory) structure capable of performing operations on a plurality of high-bandwidth memories, each of which stores the plurality of partitions.

12. The aforementioned multiple MAC trees are implemented on a chip with a PIM structure that can perform calculations within the high-bandwidth memory. The simplified memory access, the vector execution engine, the local memory unit, and the instruction scheduling unit are implemented in a buffer chip with a PNM structure capable of performing operations on multiple high-bandwidth memories, each of which stores multiple partitions. The latency processing unit according to claim 1, characterized by the above.

Images