Calculation device

Abstract

This invention provides a computing device with large storage capacity and high bandwidth, featuring a three-dimensional stacking of logic dies and memory dies. [Solution] The arithmetic unit 100 comprises a plurality of matrix arithmetic units (MAUs), one or more first memories (L1BMs) connected to the plurality of arithmetic units, and one or more second memories (MBs) connected to the plurality of arithmetic units, wherein the one or more second memories are stacked on the plurality of arithmetic units, and at least a portion of the data stored in the one or more second memories is used by the arithmetic units without going through the one or more first memories.

Description

[Technical Field]

[0001] This disclosure relates to a computing device. [Background technology]

[0002] The computing devices used in applications that perform processing such as deep learning require high computational performance because they execute a large number of matrix operations, thus necessitating memory with both large storage capacity and bandwidth. Typically, the internal memory installed in the computing device has high bandwidth but low storage capacity, while external memory connected to the computing device has high storage capacity but low bandwidth. For this reason, even with modern computing devices, it can be difficult to efficiently execute applications that have relatively high memory access. [Overview of the project] [Problems that the invention aims to solve]

[0003] This disclosure provides a computing device with large memory capacity and high bandwidth. [Means for solving the problem]

[0004] The arithmetic unit according to the embodiment of the present disclosure comprises a plurality of arithmetic units, one or more first memories connected to the plurality of arithmetic units, and one or more second memories connected to the plurality of arithmetic units, wherein the one or more second memories are stacked on the plurality of arithmetic units, and at least a portion of the data stored in the one or more second memories is used by the arithmetic units without going through the one or more first memories. [Brief explanation of the drawing]

[0005] [Figure 1] This is an exploded perspective view showing an example of the configuration of the computing device in the first embodiment of this disclosure. [Figure 2] This is an exploded perspective view showing another example of the configuration of the computing unit in the first embodiment of the present disclosure. [Figure 3]This is an exploded perspective view showing yet another example of the configuration of the computing unit in the first embodiment of the present disclosure. [Figure 4] This diagram shows variations in the connection methods between logic dies and memory dies. [Figure 5] Figure 1 is a plan view showing an example of the configuration of a logic die. [Figure 6] This block diagram shows an example of a system including a computer with multiple processing units and a host. [Figure 7] This figure shows an example of connecting a processing element and a memory block using a memory interface. [Figure 8] Figure 5 is a circuit block diagram showing an example of a processing element PE. [Figure 9] Figure 5 is a circuit block diagram showing another example of the processing element PE. [Figure 10] Figure 1 is a block diagram showing an example of the connection between the logic die and the memory die. [Figure 11] This figure shows an example of the logical configuration of each memory in the arithmetic unit and the connection relationships between each memory. [Figure 12] This figure shows an example of the arrangement of each memory in the arithmetic unit. [Figure 13] This figure shows another example of the logical configuration of each memory in the arithmetic unit and the connection relationships between each memory. [Figure 14] This figure shows an example of a logical configuration of a computing unit with a hierarchical cache and main memory. [Figure 15] Figure 14 shows an example of the physical configuration of the arithmetic unit and main memory. [Figure 16] This figure shows an example of redundantly arranging matrix operation blocks in the logic die of an arithmetic unit. [Figure 17] This figure shows an example of redundantly arranging memory blocks in the memory die of a computing unit. [Figure 18] This figure shows an example of redundant arrangement of matrix operation blocks and memory blocks in an arithmetic unit. [Figure 19] This block diagram shows an example of a machine learning model in a second embodiment of the present disclosure. [Figure 20] Figure 19 is a flowchart illustrating an example of the inference process of a machine learning model. [Figure 21] This block diagram shows an example of the hardware configuration of a computer equipped with the aforementioned arithmetic unit. [Modes for carrying out the invention]

[0006] Embodiments of this disclosure will be described in detail below with reference to the drawings. The computing unit may function as an accelerator that performs convolution operations, etc., in the training or inference of deep neural networks, large language models (LLMs), etc., and in this case it may be connected separately to a CPU (Central Processing Unit) or GPU (Graphics Processing Unit), and may function as a computer that performs scientific and technical calculations. In this specification (including the claims), terms such as "first," "second," etc., are used simply as a way to distinguish between two or more elements, and are not necessarily intended to impose technical meanings such as temporal aspects, spatial aspects, order, or quantity on the subject. Therefore, for example, references to a first element and a second element do not necessarily mean that only two elements can be adopted therein, that the first element must precede the second element, or that the first element must exist for the second element to exist.

[0007] Figures 1 to 3 are exploded perspective views showing an overview of the configuration of the arithmetic unit in the first embodiment of this disclosure. The arithmetic unit 100 shown in Figures 1 to 3 may have a three-dimensionally integrated logic die LD and a memory die MD. The logic die LD is an example of a first die, and the memory die MD is an example of a second die.

[0008] A logic die LD may have multiple processing elements PE that can operate in parallel. A processing element PE is an example of a processor. A logic die LD may be connected to one or more memory dies MD. As shown in Figures 1 to 3, the logic die LD and memory die MD may be stacked and interconnected via a data bus DB. Control signal lines (not shown) used for reading and writing data may be connected between the logic die LD and the memory die MD. Furthermore, "stacking" such as "die stacked" includes cases where the dies are arranged in the upward, downward, or both upward and downward directions. Therefore, for example, a memory block MB included in a memory die MD may be arranged in the upward, downward, or vertical direction of the matrix operation block MAB and processing element PE included in the logic die LD. In addition, elements other than dies may be inserted between the multiple dies to be stacked, as long as high-bandwidth data transfer can be maintained.

[0009] When logic dies LD and memory dies MD are stacked, the distance between the dies can be made less than or equal to the thickness of the dies. Therefore, compared to the case where the logic dies LD and memory dies MD are arranged side by side on the same substrate, for example, the load capacity of the data bus DB can be significantly reduced, and the propagation delay time of data signals transmitted to the data bus DB can be significantly shortened. As a result, the arithmetic processing performance of the arithmetic unit 100 can be improved.

[0010] As shown in Figures 1 and 2, a memory die MD may have multiple memory blocks MB, each positioned opposite a processing element PE. Each of the multiple memory blocks MB may be used exclusively for the opposing processing element PE. Figures 1 and 2 show an example where each memory die MD is connected to multiple processing element PEs, but as shown in Figure 3, each of the multiple memory die MDs may be positioned opposite multiple processing element PEs and connected to the opposing processing element PE in a 1:1 ratio. That is, the memory die MD and the processing element PE may be connected in a 1:m ratio (where m is an integer greater than or equal to 1). Each of the multiple memory blocks MB contained in one or more memory die MDs may be used as a scratchpad memory for the processing element PE connected to it.

[0011] As shown in Figures 1 and 2, each memory die MD may contain any number of memory blocks MB. The number of memory dies MD connected to the logic die LD may be determined according to the number of memory blocks MB contained in each memory die MD. As shown in Figure 2, the logic die LD may have one or more matrix arithmetic blocks MAB. Each matrix arithmetic block MAB may have one or more processing elements PE and matrix arithmetic units MAU.

[0012] For example, the memory die MD may be DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory), MRAM (Magnetoresistive Random Access Memory), PRAM (Phase-change RAM), or flash memory, etc. The memory die MD may also contain one or more DRAMs, one or more MRAMs, one or more PRAMs, or one or more flash memories, etc.

[0013] By integrating the logic die LD and memory die MD in three dimensions, the length of the data bus DB and control signal lines can be reduced to approximately the same as or less than the thickness of each die. This minimizes wiring capacitance and wiring delay, and suppresses increases in the charging and discharging currents of the data bus DB and control signal lines.

[0014] The memory block MB and memory die MD may be used as work memory for the processing element PE. By using work memory located outside the logic die LD, the storage capacity of the work memory can be significantly increased compared to using only the work memory inside the logic die LD. This makes it possible to realize an arithmetic unit 100 with large capacity and high bandwidth, enabling efficient execution of applications with high memory access frequency. As a result, the arithmetic unit 100 can perform tasks such as training, inference, or scientific and technical calculations that require processing large amounts of data at high speed and with low power consumption.

[0015] Figures 1 to 3 show examples where arithmetic units such as processing elements PE, matrix arithmetic units MAU or matrix arithmetic blocks MAB, and memory blocks MB are formed on different dies. However, various arithmetic units and memory blocks MB may be formed on different layers of a single die, and the arithmetic unit layer and the memory block MB layer may be interconnected via vias such as TSVs (Through Silicon Vias).

[0016] Figure 4 shows variations in the connection method between the logic die LD and the memory die MD. The logic die LD and the memory die MD may be connected to each other by a face-to-face connection method or a back-to-face connection method. In the face-to-face connection method, the logic die LD and the memory die MD may be connected to each other with semiconductor layers on which elements such as transistors and wiring are formed on a Si substrate or the like facing each other. In the back-to-face connection method, the logic die LD and the memory die MD may be connected to each other with the semiconductor layers positioned on the opposite side from the printed circuit board. The arithmetic unit 100 may have the form of a package including a printed circuit board, a logic die LD, and a memory die MD.

[0017] Connection Examples 1 to 3 demonstrate face-to-face connection methods, while Connection Examples 4 to 6 demonstrate back-to-face connection methods. Note that the stacking structure (stack order, number of stacks, etc.) of the logic die LD and memory die MD, and the stacking structure (stack order, number of stacks, etc.) of the processing element PE contained in the logic die LD and the memory block MB contained in the memory die MD, are not limited to Connection Examples 1 to 6. Furthermore, as shown in Figure 2, the memory die MD may have multiple memory blocks MB, and the logic die LD may have multiple matrix operation blocks MAB, each containing multiple PEs.

[0018] As shown in Connection Example 1, the logic die LD and memory die MD may be arranged on the printed circuit board in this order and connected face-to-face. As shown in Connection Example 2, the memory die MD and logic die LD may be arranged on the printed circuit board in this order and connected face-to-face. As shown in Connection Example 3, multiple memory die MDs may be arranged on the logic die LD, and each of the logic die LDs and multiple memory die MDs may be connected face-to-face.

[0019] In the face-to-face connection method, the logic die LD and the memory die MD may be interconnected by hybrid bonding or by microbumps. The data bus DB shown in Figures 1 and 2 may support either hybrid bonding or microbumps.

[0020] As shown in Connection Example 4, the logic die LD and the memory die MD may be arranged on the printed circuit board in this order with their semiconductor layers facing away from the printed circuit board, and connected in a back-to-face manner. In this case, the semiconductor layer of the memory die MD may be connected to the semiconductor layer of the logic die LD via through-vias provided within the memory die MD.

[0021] As shown in connection example 5, the memory die MD and the logic die LD may be arranged on the printed circuit board in this order with their semiconductor layers facing away from the printed circuit board, and connected back-to-face. In this case, the semiconductor layer of the logic die LD may be connected to the semiconductor layer of the memory die MD via through-through vias such as TSVs provided within the logic die LD. The data bus DB shown in Figures 1 and 2 may correspond to through-through vias.

[0022] As shown in connection example 6, multiple memory dies MD may be stacked on top of a logic die LD. In this case, the stacked memory dies MD may be connected by through vias. Furthermore, three or more memory dies MD may be stacked on top of the logic die.

[0023] Furthermore, the logic die LD and the memory die MD may be connected via a wiring board such as a silicon interposer. By using a silicon interposer, for example, the data terminals of the logic die LD and the data terminals of the memory die MD can be easily aligned. In addition, multiple logic dies LD and memory dies MD may be stacked and interconnected.

[0024] Figure 5 is a plan view showing an example of the configuration of the logic die LD in Figure 1. The logic die LD may have multiple blocks L2B, a data engine, an inter-die interconnect, a PCIe-IF (Peripheral Component Interconnect express interface), and a memory PDM.

[0025] Each block L2B may have multiple blocks L1B and memory L2BM. Each block L1B may have multiple matrix operation blocks MAB and memory L1BM. As shown in Figure 3, each matrix operation block MAB may have multiple processing elements PE and matrix operators MAU. The storage capacity of memory L2BM may be greater than the storage capacity of memory L1BM, and the storage capacity of memory block MB may be greater than the storage capacity of memory L1BM.

[0026] Multiple processing elements PE may be connected to multiple memory blocks MB, and at least some of the data stored in the memory blocks MB may be used by the matrix arithmetic unit MAU without going through memory L1BM. The matrix arithmetic unit MAU may be configured as a systolic array.

[0027] As shown in Figure 5, the logic die LD may have hierarchical blocks L1B and L2B that include a predetermined number of matrix operation blocks MAB, and the memories L1BM and L2BM may have a hierarchical structure. The matrix operation block MAB is an example of a first-level block, block L1B is an example of a second-level block, and block L2B is an example of a third-level block. Memory L1BM is an example of a first-level memory, memory block MB is an example of a second-level memory, and memory L2BM is an example of a third-level memory. Memory PDM is an example of a fourth-level memory. All matrix operation blocks MAB mounted on the logic die LD may operate by the same instruction.

[0028] The logic die LD may have four blocks L2B, and each block L2B may have eight blocks L1B and one memory L2BM. Each block L1B may have sixteen matrix operation blocks MAB and one memory L1BM, and each matrix operation block MAB may have four processing elements PE and one matrix arithmetic unit MAU. Note that the matrix operation block MAB may not have a matrix arithmetic unit MAU.

[0029] Note that the number of elements in the logic die LD is not limited to that shown in Figure 5. For example, the logic die LD may have two blocks L2B, and each block L2B may have four blocks L1B and one memory L2BM. Each block L1B may have sixteen matrix operation blocks MAB and one memory L1BM, and each matrix operation block MAB may have eight processing elements PE and one matrix arithmetic unit MAU.

[0030] A logic die LD has numerous matrix arithmetic units (MAUs) distributed across it. By connecting a memory block MB with large capacity and bandwidth to each processing element PE, data can be supplied to each of the distributed matrix arithmetic units (MAUs) without delay. In other words, it is possible to suppress the degradation of computation speed that occurs when matrix arithmetic units (MAUs) cannot perform calculations due to insufficient memory bandwidth. Note that one or both of the matrix arithmetic units (MAUs) and processing elements PE are examples of arithmetic units. Furthermore, an arithmetic unit may be an arithmetic circuit, processing circuit, or processing circuitry.

[0031] Furthermore, it is preferable that the number of blocks L2B mounted on the logic die LD, the number of blocks L1B mounted on each block L2B, the number of matrix operation blocks MAB mounted on each block L1B, and the number of processing elements PE mounted on each matrix operation block MAB are each 2 to the power of n (where n is an integer of 1 or more).

[0032] The data engine may control the transfer of data stored in memory PDM from outside the logic die LD to memory L2BM or memory L1B, and may also control the transfer of data between memory L2BM, memory L1B and matrix operation block MAB.

[0033] The die interconnect may be used to connect one arithmetic unit 100 to another arithmetic unit 100. The PCIe-IF may send and receive data or instructions to and from a host or device connected to the outside of the arithmetic unit 100. The memory PDM may function as a buffer memory that holds data sent and received via the PCIe-IF. For example, the PCIe-IF may be composed of an FPGA (Field Programmable Gate Array).

[0034] Each processing element PE may be connected to a memory block MB of a memory die MD, as shown in Figures 1 and 2. Each processing element PE does not need to be able to directly access any memory block MB other than the one it is directly connected to, and the program may be designed to operate without using the results of calculations performed by other processing elements PE. Examples of processing elements PE are shown in Figures 8 and 9. For example, the matrix arithmetic unit MAU may execute SIMD (Single Instruction Multiple Data) instructions. The arithmetic unit 100 may also have a SIMD model architecture in which all matrix arithmetic blocks MAB included in the arithmetic unit 100 are operated with a single instruction.

[0035] Figure 6 is a block diagram showing an example of a system including a computer 200 equipped with multiple arithmetic units 100 and a host 300. The computer 200 shown in Figure 6 has multiple interconnected arithmetic units 100. For example, the computer 200 may take the form of an electronic circuit board equipped with multiple arithmetic units 100. In Figure 6, the computer 200 has two arithmetic units 100, but the number of arithmetic units 100 may be one or three or more. For example, the computer 200 may perform inference such as large language models (LLM). The host 300 may have a CPU and host memory HOSTM.

[0036] Each arithmetic unit 100 is connected to the host 300 via PCIe-IF and may receive data and instruction information transferred from the host memory HOSTM by a control program executed by the host 300's CPU. The transfer of information between the host memory HOSTM and the arithmetic unit 100 may be performed by DMA (Direct Memory Access).

[0037] Each arithmetic unit 100 may perform calculations using the matrix arithmetic block MAB shown in Figure 5 based on arithmetic instructions received from the host 300, or it may perform data transfer processing based on data transfer instructions received from the host 300. For example, the host 300 may instruct each arithmetic unit 100 via software which calculation to perform on which MAB.

[0038] Data transfer to memory PDM, L2BM, L1BM, or memory die MD may be performed by executing a data transfer instruction. After the data to be processed has been transferred to the memory die MD, each arithmetic unit 100 may perform calculations using a processing element PE or matrix arithmetic unit MAU based on an arithmetic instruction received from the host 300.

[0039] For example, a program containing various instruction codes transferred from the host 300 to the arithmetic unit 100 may be generated by a compiler device (not shown) or stored in the host memory HOSTM. In Figure 6, the lines connecting the elements of the arithmetic unit 100 indicate the data transfer path, and arithmetic instructions and various control information may be transferred via a separate path (not shown) from the data transfer path.

[0040] In Figure 6, as shown in Figures 1 and 2, each processing element PE is connected to a memory block MB. However, multiple processing elements PE may be connected to a single memory block MB, or each processing element PE may be connected to multiple memory block MBs.

[0041] Figure 7 shows an example of connecting a processing element PE to a memory block MB using a memory interface MI / F. In connection example 1, two processing elements PE1 of the matrix operation block MAB may be connected to memory block MB1 via memory interface MI / F1, and two processing elements PE2 of the matrix operation block MAB may be connected to memory block MB2 via memory interface MI / F2. That is, memory interface MI / F1 may connect at least one processing element PE1 to one or more memory blocks MB1, and memory interface MI / F2 may connect at least one processing element PE2 to one or more memory blocks MB2. Furthermore, memory interface MI / F1 may connect at least one processing element PE1 to a predetermined memory block MB1 from among one or more memory blocks MB1, and memory interface MI / F2 may connect at least one processing element PE2 to a predetermined memory block MB2 from among one or more memory blocks MB2. Memory interface MI / F1 is an example of a first memory interface, and processing element PE1 connected to memory interface MI / F1 is an example of a first arithmetic unit. Memory interface MI / F2 is an example of a second memory interface, and processing element PE2 connected to memory interface MI / F2 is an example of a second arithmetic unit.

[0042] Alternatively, the two processing elements PE1 may be independently connected to the memory interface MI / F1, or the two processing elements PE2 may be independently connected to the memory interface MI / F1. In this case, the memory interface MI / F1 or MI / F2 has an arbitration circuit that arbitrates memory access instructions from the two processing elements PE1 or PE2.

[0043] In connection example 2, two processing elements PE1 may be connected to the memory interface MI / F1 in common, or two processing elements PE2 may be connected to the memory interface MI / F2 in common. In connection example 1 or connection example 2, the number of processing elements PE1 or PE2 connected to the memory interface MI / F1 or MI / F2 may be three or more.

[0044] In connection example 3, one of the two processing elements PE1 may have a memory interface MI / F1, and one of the two processing elements PE2 may have a memory interface MI / F2. In this case, the other processing element PE1 may be connected to the memory interface MI / F1 of one of the processing elements PE1 and connected to the memory block MB1 via the memory interface MI / F1. Similarly, the other processing element PE2 may be connected to the memory interface MI / F2 of one of the processing elements PE2 and connected to the memory block MB2 via the memory interface MI / F2.

[0045] In connection example 3, the matrix operation block MAB may have three or more processing elements PE1 and three or more processing elements PE2. In this case, one of the three or more processing elements PE1 may have a memory interface MI / F1, and one of the three or more processing elements PE2 may have a memory interface MI / F2. Furthermore, three or more memory blocks MB may be connected to a single matrix operation block MAB.

[0046] In connection example 4, a memory interface MI / F1 or MI / F2 may be provided for each matrix operation block MAB1 and MAB2. In this case, multiple processing elements PE1 (e.g., 4) of matrix operation block MAB1 may be connected to memory block MB1 via memory interface MI / F1, and multiple processing elements PE2 (e.g., 4) of matrix operation block MAB2 may be connected to memory block MB2 via memory interface MI / F2. Note that one memory block MB may be connected to multiple matrix operation blocks MAB.

[0047] As shown in the connection examples in Figure 7, the memory interfaces MI / F1 and MI / F2 may be used exclusively for accessing the respective memory blocks MB1 and MB2 provided on the memory die MD. Note that the memory interfaces MI / F1 and MI / F2 are not used for accessing memory L1BM and memory L2BM (not shown).

[0048] Furthermore, while Figure 7 shows an example where one memory interface MI / F1 or MI / F2 is provided for multiple processing elements PE1 or PE2, a memory interface MI / F may be provided for each processing element PE. In that case, the memory interface MI / F may be built into each processing element PE.

[0049] Figure 8 is a circuit block diagram showing an example of the processing element PE shown in Figure 5. The processing element PE includes an integer arithmetic unit (IALU), registers M-REG, GRF0, GRF1, T-REG, local memories LM0, LM1, and multiplexers MUX1, MUX2. For example, local memories LM0 and LM1 may be composed of SRAM. Local memories LM0 and LM1 are an example of a fifth memory.

[0050] Register M-REG may mask at least a portion of the calculation result data from the integer arithmetic unit IALU and prevent the writing of the masked calculation result data to local memory LM1. Registers GRF0 and GRF1 may be general-purpose registers that hold data used for calculations or calculation result data. Registers GRF0 and GRF1 may also be 2-port registers that can perform data writing and data reading in parallel. Register T-REG may be used to temporarily hold data.

[0051] Local memory LM0 may be a buffer memory that holds data to be read from and written to the memory block MB. In this case, local memory LM0 may be connected to the corresponding memory block MB via the memory interface MI / F1 or MI / F2 shown in Figure 7. Note that the memory block MB is connected only to local memory LM0 and not to multiplexers MUX1 and MUX2, so data input to and output to the memory block MB is transferred only via local memory LM0.

[0052] For example, the storage capacity of local memory LM0 may be 16 MB, and data may be read from or written to the memory block MB via local memory LM0 at a rate of 8 bytes per cycle. The data input / output ports of the memory block MB may also be 8 bytes. Furthermore, local memory LM0 may hold data that is not read from or written to the memory block MB.

[0053] Local memory LM1 may, for example, hold data used by the integer arithmetic unit IALU, and may also hold calculation result data from the integer arithmetic unit IALU. For example, the storage capacity of local memory LM1 may be 16k bytes, and data reading or writing to local memory LM1 may be performed at 8 bytes / cycle. Local memories LM0 and LM1 may be single-port memories capable of either reading or writing data, or dual-port memories capable of reading or writing data simultaneously or nearly simultaneously.

[0054] Multiplexer MUX1 may, according to the instruction, transfer the result data from the integer arithmetic unit IALU, the result data from the matrix arithmetic unit MAU, or data from memory L1BM to register GRF0, GRF1, T-REG, or local memory LM0, LM1. Multiplexer MUX2 may, according to the instruction, transfer the result data from the integer arithmetic unit IALU, register GRF0, GRF1, T-REG, local memory LM0, LM1, or data from the matrix arithmetic unit MAU to integer arithmetic unit IALU, matrix arithmetic unit MAU, or memory L1BM.

[0055] The processing element PE may supply data read from the memory block MB via local memory LM0 to the integer arithmetic unit IALU or the matrix arithmetic unit MAU to perform calculations. The processing element PE may also write the calculation result data output from the integer arithmetic unit IALU or the matrix arithmetic unit MAU back to the memory block MB via local memory LM0. The basic operation of the logic die LD is the calculation performed by the integer arithmetic unit IALU or the matrix arithmetic unit MAU on the data read from the memory block MB, and the writing back of the calculation result data to the memory block MB.

[0056] Figure 9 is a circuit block diagram showing another example of the processing element PE shown in Figure 5. The processing element PE shown in Figure 9 does not have a local memory LM0, and one of the outputs of multiplexer MUX1 and one of the inputs of multiplexer MUX2 may be connected to a memory block MB via a memory interface MI / F (not shown).

[0057] Figure 10 is a block diagram showing an example of the connection between the logic die LD and the memory die MD in Figure 1. Block L1B may have a memory controller MCNT that generates a control signal CNTL for controlling the reading and writing of data to the memory block MB, in addition to the multiple matrix operation blocks MAB and memory L1BM shown in Figure 2. The memory controller MCNT may include some of the functions of the memory interfaces MI / F1 and MI / F2 shown in Figure 7. For example, the control signal CNTL may include an address signal AD, a command signal CMD that indicates a read access request or a write access request, and a clock signal CLK, etc.

[0058] Furthermore, the memory controller MCNT may output a data synchronization signal DSYNC to each local memory LM0 within block L1B. The memory controller MCNT may also output a data synchronization signal DSYNC indicating the timing to write data read from memory block MB to data bus DB to local memory LM0 in response to the control signal CNTL during read access to memory block MB. The memory controller MCNT may also output a data synchronization signal DSYNC indicating the timing to output data to be written to memory block MB from local memory LM0 to data bus DB in response to the control signal CNTL during write access to memory block MB.

[0059] For example, the memory controller MCNT may generate a common control signal CNTL and a data synchronization signal DSYNC for all local memory LM0 mounted on block L1B. In this case, all memory blocks MB connected to block L1B will operate in sync with each other. Alternatively, the memory controller MCNT may generate a common control signal CNTL and a data synchronization signal DSYNC for each predetermined number of local memory LM0 mounted on block L1B.

[0060] Figure 11 shows an example of the logical configuration of each memory L2BM, L1BM, LM0, and LM1 of the arithmetic unit 100 and the connection relationships of each memory. Local memories LM0 and LM1 may be connected to their respective corresponding memory L1BM. Memory L1BM may be connected to its corresponding memory L2BM. Note that memories L1BM and L2BM may not be directly connected to the memory block MB, but may be used for data transfer between processing elements PE or to the outside of the logic die LD. Alternatively, memories L1BM and L2BM may be directly connected to the memory block MB or an external memory different from the memory block MB and used for data transfer. Furthermore, the processing element PE may be configured without local memory LM1.

[0061] Each memory block MB is positioned near and facing the processing element PE, and may be connected to the local memory LM0 by, for example, hybrid bonding the terminals of the logic die LD and the memory die MD. This reduces the data travel distance when reading and writing data to the memory block MB, and suppresses increases in wiring length and power consumption even when the bandwidth of the memory block MB is increased.

[0062] Furthermore, processing elements PE, which are connected to memory blocks MB on a 1:1 basis, are preferably connected to memory blocks MB located directly above or below them in order to shorten the wiring length. However, each processing element PE does not need to be positioned offset from directly above or below the memory block MB, as long as it is positioned closer to the memory block MB in a plan view than adjacent processing elements PE. In other words, each processing element PE only needs to be connected to the nearest memory block MB in a plan view. This helps to suppress the degradation of the memory block MB's bandwidth.

[0063] Figure 12 shows an example of the arrangement of the respective memories L2BM, L1BM, LM0, and LM1 of the arithmetic unit 100. As shown in Figures 1 and 2, the memory die MD containing the memory block MB is positioned above or below the logic die LD, so that the connection between the memory block MB and the local memory LM0 can be distributed within the chip without concentrating it on a portion of the memory die MD and logic die LD. As a result, a very large number of data buses DB can be used to connect the memory die MD and the logic die LD.

[0064] For example, the memory die MD and the logic die LD may be connected so that data can be read or written at a rate of 8 bytes per cycle for each processing element PE. This allows for the use of a memory block MB with a larger memory capacity without degrading bandwidth, compared to, for example, using a local memory LM0 with a smaller memory capacity to hold data during calculations.

[0065] Furthermore, memory blocks MB may be connected not only to each processing element PE, but also to each matrix operation block MAB (for every four processing element PEs). In addition, memory blocks MB may be connected to every two or more matrix operation blocks MAB. Also, as shown in Figure 7, two memory blocks MB1 and MB2 may be connected to one matrix operation block MAB, and each memory block MB1 and MB2 may be shared by two processing elements PE1 or two processing elements PE2. Furthermore, memory blocks MB may be connected to every four matrix operation blocks MAB (for every 16 processing element PEs). The number of processing element PEs connected to each memory block MB may be determined considering constraints such as the interface specifications of the memory block MB.

[0066] Figure 13 shows another example of the logical configuration and connection relationships of each memory L2BM, L1BM, LM0, and LM1 of the arithmetic unit 100. The memory block MB may be connected to every four processing elements PE. The configuration shown in Figure 13 is the same as in Figure 11, except that the number of processing elements PE connected to the memory block MB is different.

[0067] When four processing elements PE are assigned to one memory block MB, the port bandwidth of the memory block MB may be set to four times the port bandwidth of the memory block MB in Figure 11. This allows the bandwidth per processing element PE to be the same as in Figure 11.

[0068] In Figure 13, the four processing elements PE are constrained to access different portions of data held in the same address storage area of ​​a single memory block MB. If, in the configuration of Figure 11, each processing element PE reads or writes 16 bytes per cycle, then in Figure 13, each of the four processing elements PE may read or write 64 bytes per cycle. In this case, each processing element PE may read or write four equal parts of 16 bytes (64 bytes divided into four equal parts). Note that the number of processing elements PE allocated to a single memory block MB may differ from those shown in Figures 11 and 13.

[0069] Furthermore, when multiple processing elements PE are assigned to a single memory block MB, a memory die MD may be placed in place of each memory block MB. In addition, a memory die MD may be placed in place of each memory block MB as shown in Figure 5.

[0070] Figure 14 shows an example of a logical configuration of an arithmetic unit with a hierarchical cache and main memory. The arithmetic unit 101 shown in Figure 14 may have multiple processing elements PE, multiple L1 data caches L1D, multiple L2 data caches L2D and L3 data caches L3D.

[0071] Multiple L1 data caches L1D may be connected to multiple processing elements PE. Each of the multiple L2 data caches L2D may be connected to a predetermined number of L1 data caches L1D and shared by a predetermined number of processing elements PE. An L3 data cache L3D may be connected to multiple L2 data caches L2D and shared by all processing elements PE. The L3 data cache L3D may also be connected to a main memory 102 such as DRAM.

[0072] The configuration shown in Figure 14 allows for a substantial increase in the bandwidth of the main memory 102. For example, when a processing element PE reads data Da from the main memory 102, it can keep that data Da in the L3 data cache L3D for a while. When another processing element PE attempts to read data Da from the main memory 102, if the data Da remains in the L3 data cache L3D, it can obtain the data Da by accessing the L3 data cache L3D without accessing the main memory 102.

[0073] In this way, by tiering the cache, the bandwidth of main memory 102 can be saved. The saved bandwidth of main memory 102 can be used to access data that was not in the cache, so it can be considered that the bandwidth of main memory 102 has effectively increased.

[0074] Figure 15 shows an example of the physical configuration of the arithmetic unit 101 and main memory 102 shown in Figure 14. Cache tiering is not always optimal. For example, if multiple processing elements PE continuously read different data from each other, there is no point in sharing the cache between the processing elements PE. Rather, sharing the cache makes data movement inefficient.

[0075] As shown in Figure 15, the processing element PE and each memory are interconnected as follows: PE-L1D-L2D-L3D-main memory 102. Therefore, when the processing element PE reads data from the main memory 102, the arithmetic unit 101 sends the data read from the main memory 102 to the L3 data cache L3D, from the L3 data cache L3D to the L2 cache, from the L2 data cache L2D to the L1 cache, and from the L1 data cache L1D to the processing element PE. When writing data to the main memory 102, the data transfer is performed in the reverse order of reading.

[0076] In the configuration shown in Figure 15, the average distance data travels is approximately the size of the chip of the arithmetic unit 101, specifically in centimeters. To increase the bandwidth of the main memory 102, it is necessary to increase the bandwidth between processing element PE and L1 data cache L1D, the bandwidth between L1 data cache L1D and L2 data cache L2D, and the bandwidth between L2 data cache L2D and L3 data cache L3D. However, increasing the bandwidth of the main memory 102 is difficult due to the amount of wiring and power consumption, and furthermore, it is costly. In contrast, in the configurations shown in Figures 11 to 13, the logic die LD and memory die MD are stacked, so the distance between dies can be made less than or equal to the thickness of the die. Therefore, compared to the configuration shown in Figure 15, the load capacity of the data bus connecting the logic die LD and memory die MD can be significantly reduced, and the propagation delay time of data signals transmitted to the data bus can be significantly reduced. As a result, the arithmetic processing performance of the arithmetic unit 100 can be improved.

[0077] Figure 16 shows an example of redundant arrangement of matrix operation blocks MAB in the logic die LD of the arithmetic unit 100. In Figure 16, each memory block MB of the arithmetic unit 100 may be assigned to each matrix operation block MAB (each of the four processing elements PE). Here, when a large number of processing elements PE or a large number of matrix operation blocks MAB are assigned to a large number of memory blocks MB, it is preferable to consider redundancy. Generally, the yield during the manufacturing of semiconductor chips may not be high, so by adding spare circuits and switching from a failed circuit to the spare circuit, defects can be recovered and the yield can be improved.

[0078] The arithmetic unit 100 may have one matrix operation block MAB16 as a spare for each of the 16 matrix operation blocks MAB0-MAB15. That is, 16 memory blocks MB may be arranged corresponding to the 17 matrix operation blocks MAB0-MAB16. The spare matrix operation block MAB16 is an example of a redundant operation block.

[0079] If the matrix operation blocks MAB0-MAB15 are not faulty and the logic die LD is functioning normally, the 16 ports LP0-LP15 of the matrix operation blocks MAB0-MAB15 may be connected to the 16 ports MP0-MP15 of the memory blocks MB0-MB15 via a switching circuit SW provided on the logic die LD.

[0080] For example, if a failure in the matrix operation block MAB7, indicated by an "X" mark, is detected during the logic die LD testing process, the switching circuit SW may be switched, disconnecting the connection between the matrix operation block MAB7 and the memory block MB7. Then, the ports LP0-LP6 and LP8-LP16 of the matrix operation blocks MAB0-MAB6 and MAB8-MAB16 may be connected to the ports MP0-MP15 of the memory blocks MB0-MB15, respectively, via the switching circuit SW.

[0081] In other words, the switching circuit SW may connect the matrix arithmetic blocks MAB0-MAB16 to the memory blocks MB0-MB15 in a switchable manner. This allows the arithmetic unit 100 to operate normally even if one of the 17 matrix arithmetic blocks MAB0-MAB16 fails. This improves the yield of the arithmetic unit 100.

[0082] Figure 17 shows an example of redundant arrangement of memory blocks MB in the memory die MD of the arithmetic unit 100. In Figure 17, as in Figure 16, each memory block MB of the arithmetic unit 100 may be allocated to each matrix operation block MAB (each of the four processing elements PE). The arithmetic unit 100 may have one spare memory block MB16 for each of the 16 memory blocks MB0-MB15. That is, 17 memory blocks MB may be arranged corresponding to the 16 matrix operation blocks MAB0-MAB15. The spare memory block MB16 is an example of redundant memory and is an example of a redundant memory block containing redundant memory.

[0083] If memory blocks MB0-MB15 are not faulty and memory die MD is operating normally, the 16 ports MP0-MP15 of memory blocks MB0-MB15 may be connected to the 16 ports LP0-LP15 of matrix operation blocks MAB0-MAB15 via a switching circuit SW provided on memory die MD.

[0084] For example, if a failure of memory block MB4, indicated by an "X" mark, is detected during the testing process of the memory die MD, the switching circuit SW may be switched, disconnecting the connection between memory block MB4 and matrix operation block MAB4. Then, ports MP0-MP3 and MP5-MP16 of memory blocks MB0-MB3 and MB5-MB16 may be connected to ports LP0-LP15 of MAB0-MAB15, respectively, via the switching circuit SW.

[0085] In other words, the switching circuit SW may connect memory blocks MB0-MB16 to matrix operation blocks MAB0-MAB15 in a switchable manner. This allows the arithmetic unit 100 to operate normally even if one of the 17 memory blocks MB0-MB16 fails. This improves the yield of the arithmetic unit 100.

[0086] Figure 18 shows an example of redundant arrangement of matrix operation blocks MAB and memory blocks MB in the arithmetic unit 100. In Figure 18, as in Figures 16 and 17, each memory block MB of the arithmetic unit 100 may be allocated to each matrix operation block MAB (for each of the four processing elements PE). The arithmetic unit 100 may have 16 matrix operation blocks MAB0-MB15 and one spare matrix operation block MAB16, and 16 memory blocks MB0-MB15 and one spare memory block MB16.

[0087] If the matrix operation blocks MAB0-MAB15 and memory blocks MB0-MB15 are functioning correctly, and the logic die LD and memory die MD are operating normally, for example, the 16 ports LP0-LP15 of the matrix operation blocks MAB0-MAB15 may be connected to the 16 ports MP0-MP15 of the memory blocks MB0-MB15 via a switching circuit SW provided on the logic die LD. Note that the switching circuit SW may be provided on both the logic die LD and the memory die MD.

[0088] For example, if a failure of matrix operation block MAB7, indicated by an "x", is detected during the testing process of the logic die LD, and a failure of memory block MB4, also indicated by an "x", is detected during the testing process of the memory die MD, the connection of the switching circuit SW may be switched. Furthermore, in the arithmetic unit 100, the ports LP0-LP6 and LP8-LP16 of the matrix operation blocks MAB0-MAB6 and MAB8-MAB16 may be connected to the ports MP0-MP3 and MP5-MP16 of the memory blocks MB0-MB3 and MB5-MB16, respectively.

[0089] This allows the arithmetic unit 100 to operate normally even if one of the 17 matrix operation blocks MAB0-MAB16 and one of the 17 memory blocks MB0-MB16 fail. This improves the yield of the arithmetic unit 100.

[0090] In the first embodiment described above, by connecting multiple processing elements PE of the logic die LD to multiple memory dies MD or multiple memory blocks MB arranged opposite the logic die LD, and integrating them in three dimensions, a larger memory bandwidth can be achieved with lower power consumption compared to conventional methods. Because a large memory bandwidth can be used with low power and low cost, applications with relatively frequent memory access, such as training and inference of deep neural networks or scientific and technical computing, can be executed efficiently with low power consumption.

[0091] For example, the arithmetic unit 100 can achieve a bandwidth between the logic die LD and the memory die MD of, for instance, several tens of TG / s. This is significantly larger than the bandwidth of HBM2 (256 GB / s) and HBM3 (819 GB / s).

[0092] A second embodiment of this disclosure is an example of a calculation method for performing inference processing of a machine learning model using the arithmetic unit 100 of the first embodiment. Note that the calculation method according to this embodiment is not limited to inference processing of a machine learning model, but may also be used for training processing of a machine learning model.

[0093] The machine learning model in this embodiment may include a neural network. The machine learning model may include a neural network having an attention mechanism. The neural network having an attention mechanism may be, for example, a neural network also called a Transformer. The machine learning model may also include a state-space model (SSM).

[0094] The machine learning model may, for example, be a Large Language Model (LLM). The machine learning model may also be a generative model, foundational model, or neural network that generates various types of data such as audio, images, and videos. The machine learning model may also support multimodal processing.

[0095] The configuration of the machine learning model in this embodiment will be described with reference to Figure 19. Figure 19 is a block diagram showing an example of a machine learning model in the second embodiment of this disclosure.

[0096] As shown in Figure 19, the machine learning model M may include one or more transformers T (T-1, T-2, T-3, ...). The number of transformers T included in the machine learning model M can be arbitrary, but for example, it could be around 80. The machine learning model M may also include layers, blocks, or networks other than transformers.

[0097] From this point forward, when there are multiple Transformer T units, they will be distinguished using sub-numbers such as "Transformer T-1," "Transformer T-2," etc. Simply referring to "Transformer T" applies to all Transformer T units.

[0098] Transformer T may include a multi-layer perceptron (MLP) and an attention ATT. The multi-layer perceptron MLP may, for example, be a feedforward network. The number of layers in the feedforward network can be arbitrary. The attention ATT may, for example, be a multi-head attention system. The number of heads in the multi-head attention system can be arbitrary, but for example, it may be around 64.

[0099] In this embodiment, the number of parameters in the machine learning model M may be around 10B to 70B (10 billion to 70 billion). In this case, the amount of data when quantized in 8-bit floating-point format (FP8) will be around 10GB to 70GB. As an example, generating 1000 tokens using the machine learning model M will result in memory access of approximately 10TB to 70TB. Since the bandwidth of conventional GPUs is approximately 4TB per second, generating 1000 tokens on a conventional GPU would take approximately 2.5 seconds to 17.5 seconds.

[0100] Furthermore, in this embodiment, the arithmetic unit 100 may have two blocks L2B. Each block L2B may have four blocks L1B. Each block L1B may have sixteen matrix operation blocks MAB. Each matrix operation block MAB may have eight processing elements PE and one matrix arithmetic unit MAU.

[0101] The matrix operation block MAB may be configured as follows:

[0102] The local memory LM0 of the processing element PE is connected to the memory block MB. The processing element PE can read or write 16 bytes of data per cycle to or from the local memory LM0. The capacity of the local memory LM0 is 16 MB per processing element PE.

[0103] The local memory LM1 of the processing element PE is composed of SRAM. The processing element PE can read or write 16 bytes of data per cycle to or from the local memory LM1. The capacity of the local memory LM1 is 16KB per processing element PE.

[0104] The registers GRF0 and GRF1 of the processing element PE are composed of SRAM. Each of the registers GRF0 and GRF1 can read and write 16 bytes of data per cycle in parallel. The capacity of each of the registers GRF0 and GRF1 is 2KB per processing element PE.

[0105] The register T-REG of the processing element PE is composed of a flip-flop. Register T-REG can read and write 16 bytes of data per cycle in parallel. The capacity of register T-REG is 64B per processing element PE.

[0106] The integer arithmetic unit IALU is an arithmetic unit that performs integer addition and other operations. The input to the integer arithmetic unit IALU is one or two 16-byte data units read from local memory LM0 or similar per cycle. The output of the integer arithmetic unit IALU is 16 bytes.

[0107] The Matrix Operated Unit (MAU) is an arithmetic unit that performs floating-point arithmetic. The MAU performs matrix multiplication or element-wise multiplication or addition of vectors. For matrix multiplication, the MAU calculates the product of a 32x64 matrix and a 64x2 matrix per cycle.

[0108] The inference process of the machine learning model in this embodiment will be explained with reference to Figure 20. Figure 20 is a flowchart showing an example of the inference process of the machine learning model M shown in Figure 19. Although Figure 20 shows the process of one transformer T included in the machine learning model M, the processes of two or more transformers T included in the machine learning model M may be executed in parallel. When two or more transformers T execute processes in parallel, each transformer T may execute the process synchronously or asynchronously.

[0109] In step S1, the arithmetic unit 100 stores the parameters of the machine learning model M in the memory block MB. The arithmetic unit 100 may also store parameters of the machine learning model M input from an external source in the memory block MB. The process of storing information in the memory block MB may include the arithmetic unit 100 executing at least a part of the processing necessary for storage.

[0110] For example, the arithmetic unit 100 may store the parameters of the machine learning model M, which have been transferred from the host memory HOSTM by an instruction issued by the host CPU 300, in the memory block MB via memory L2BM, memory L1BM, and local memory LM0. The instruction issued by the host CPU 300 may be a SIMD instruction. A SIMD instruction is an instruction set that processes multiple data in parallel with a single instruction. Multiple matrix operation blocks MAB provided by the arithmetic unit 100 may operate in parallel according to a single SIMD instruction received from the host CPU 300.

[0111] The arithmetic unit 100 may store the parameters of the machine learning model M in a distributed manner across multiple memory blocks MB. The arithmetic unit 100 may assign the parameters of the machine learning model M to multiple matrix operation blocks MAB, and store a portion of the parameters of the machine learning model M in the memory blocks MB connected to each of the matrix operation blocks MAB. The assignment of parameters to the matrix operation blocks MAB may be performed by the arithmetic unit 100, or it may be performed by external specification (for example, an instruction issued by the CPU of the host 300).

[0112] In step S2, the arithmetic unit 100 acquires input information to be input to the machine learning model M. The arithmetic unit 100 may also acquire input information that is input from an external source. For example, the arithmetic unit 100 may receive input information transferred from the host memory HOSTM by an instruction issued by the CPU of the host 300.

[0113] The input information may include text data, image data, or audio data. Text data may, for example, be natural language text called a prompt. Image data may, for example, be a still image or a video. Text data may also be text data obtained by speech recognition of audio data or video. Text data may also be text obtained by character recognition of image data. Image data may, for example, include an image of the user. Audio data may, for example, include the voice of the user speaking. Audio data may also be speech data obtained by speech synthesis of text data.

[0114] The input information may be divided into tokens. A token is a processing unit used by a machine learning model to process electronic data, and the amount of data in one token may vary depending on the design of the machine learning model. For example, a token may be a single Japanese character or a single English word, but depending on the frequency of occurrence, one character may be represented by two tokens, or two or more characters may be represented by one token.

[0115] The arithmetic unit 100 may input the acquired input information into the matrix operation block MAB. The arithmetic unit 100 may input a predetermined number of tokens included in the input information into the matrix operation block MAB. The number of tokens to be input into the matrix operation block MAB may be predetermined by the machine learning model M or the design of the arithmetic unit 100. In this embodiment, the number of tokens may be 32, for example.

[0116] The input information entered into the matrix operation block MAB may be stored in the local memory LM1 of the matrix operation block MAB, or in the memory block MB connected to the matrix operation block MAB. The arithmetic unit 100 may distribute the input information to multiple matrix operation blocks MAB. The arithmetic unit 100 may assign the input information to the machine learning model M to multiple matrix operation blocks MAB, and input a portion of the input information to each of the multiple matrix operation blocks MAB. The assignment of input information to the matrix operation blocks MAB may be performed by the arithmetic unit 100, or it may be performed by external specification (for example, an instruction issued by the CPU of the host 300).

[0117] In step S3, the arithmetic unit 100 performs a prefill process. The prefill process generates intermediate data to be used in the inference process of the machine learning model M based on the input information obtained in step S2. During the prefill process, the arithmetic unit 100 repeatedly performs calculations for the multilayer perceptron MLP and the attention ATT. Before performing the calculations for the multilayer perceptron MLP or the attention ATT, the arithmetic unit 100 may read the parameters of the machine learning model M stored in step S1 from the memory block MB.

[0118] Intermediate data may vary depending on the type of machine learning model. For example, if the machine learning model includes a transformer, the intermediate data may include a key-value cache. A key-value cache is an array of cached key vectors and value vectors. A key-value cache is generated for each layer that makes up the transformer. As another example, if the machine learning model includes a state-space model, the intermediate data may include the states in the state-space model. The states in the state-space model may be represented by one or more vectors or tensors.

[0119] In step S3-1, the matrix operation block MAB of the arithmetic unit 100 reads the parameters of the machine learning model M from the memory block MB. The matrix operation block MAB may read at least some of the parameters of the machine learning model M stored in the memory block MB. The matrix operation block MAB may read only the parameters corresponding to the input information entered in step S2. The matrix operation block MAB may read only the parameters to be used in the subsequent steps S3-2 or S3-3.

[0120] The matrix operation block MAB may read at least some of the parameters of the machine learning model M from the memory block MB while step S3-2 or S3-3 is being executed. For example, while step S3-2 or S3-3 is being executed for some of the input information, the matrix operation block MAB may read the parameters to be used in step S3-2 or S3-3 for the subsequent parts of the input information.

[0121] In step S3-2, the matrix operation block MAB of the arithmetic unit 100 generates intermediate data. The matrix operation block MAB may generate intermediate data by calculating the attention ATT of the machine learning model M. The matrix operation block MAB may generate intermediate data corresponding to the input information input in step S2. The matrix operation block MAB may generate intermediate data using the parameters read in step S3-1.

[0122] The matrix operation block MAB may store the generated intermediate data in the memory block MB. The matrix operation block MAB may store at least a portion of the intermediate data in the memory block MB. The matrix operation block MAB may store the intermediate data in the memory block MB connected to the matrix operation block MAB.

[0123] In step S3-3, the matrix operation block MAB of the arithmetic unit 100 generates output information to be output from the machine learning model M. The matrix operation block MAB may generate output information by computing the multilayer perceptron MLP of the machine learning model M. The matrix operation block MAB may generate output information using the parameters read in step S3-1. The matrix operation block MAB may generate output information using the intermediate data generated in step S3-2.

[0124] The output information may include at least one of text data, audio data, or image data. The image data may include still images and videos, and videos may include audio data. The output information may also be multimodal data including at least two of text data, audio data, or image data. The output information may be divided into token units. The output information may also be a single token output for the last token included in the input information.

[0125] The matrix operation block MAB may store the generated output information. The matrix operation block MAB may store the output information in its local memory LM1, or it may store the output information in a memory block MB connected to the matrix operation block MAB. The matrix operation block MAB may store only the output information generated when the processing of the input information received in step S2 is completed in the local memory LM1 or memory block MB.

[0126] The matrix operation block MAB may repeatedly execute the processes from steps S3-1 to S3-3, or it may perform the processes from steps S3-1 to S3-3 only once and then execute step S4. The matrix operation block MAB may repeatedly execute the processes from steps S3-1 to S3-3 until the processing of the input information entered in step S2 is completed. The matrix operation block MAB may read all the parameters stored in the memory block MB in step S3-1 and then repeatedly execute only the processes in steps S3-2 and S3-3. As an example, if large input information is entered in step S2, the input information may be divided into multiple parts, and the processes from steps S3-1 to S3-3 may be repeatedly executed for each of the divided input pieces.

[0127] In step S4, the arithmetic unit 100 executes a decoding process. The decoding process is the process of generating output information from the machine learning model M using the intermediate data generated in the prefill process. During the decoding process, the arithmetic unit 100 repeatedly performs calculations for the multilayer perceptron MLP and the attention ATT. Before performing calculations for the multilayer perceptron MLP and the attention ATT, the arithmetic unit 100 may read the parameters of the machine learning model M stored in step S1 from the memory block MB.

[0128] In step S4-1, the matrix operation block MAB of the arithmetic unit 100 reads the parameters of the machine learning model M from the memory block MB. The matrix operation block MAB may read at least a portion of the parameters of the machine learning model M stored in the memory block MB. The matrix operation block MAB may read only the parameters corresponding to the input information entered in step S2. The matrix operation block MAB may read only the parameters to be used in the subsequent steps S4-3 or S4-4. The matrix operation block MAB may sequentially read at least a portion of the parameters of the machine learning model M stored in the memory block MB when needed for processing.

[0129] The matrix operation block MAB does not need to read parameters that have already been read during the prefill process. If all parameters have been read during the prefill process, the matrix operation block MAB does not need to perform step S4-1.

[0130] In step S4-2, the matrix operation block MAB of the arithmetic unit 100 reads intermediate data from the memory block MB. The matrix operation block MAB may read at least a portion of the intermediate data stored in the memory block MB. The matrix operation block MAB may read all of the intermediate data stored in the memory block MB. The matrix operation block MAB may read at least one of the intermediate data generated during the prefill process or the intermediate data generated during the decode process. The matrix operation block MAB may read at least a portion of the intermediate data generated during the prefill process. The matrix operation block MAB may read at least a portion of the intermediate data generated during the decode process. The matrix operation block MAB may read at least a portion of new intermediate data generated during past decode processes. The matrix operation block MAB may sequentially read at least a portion of the intermediate data stored in the memory block MB when it becomes necessary for processing.

[0131] In step S4-3, the matrix operation block MAB of the arithmetic unit 100 generates intermediate data. The matrix operation block MAB may generate intermediate data by calculating the attention ATT of the machine learning model M. The matrix operation block MAB may generate intermediate data corresponding to the output information generated in the prefill process. The matrix operation block MAB may generate intermediate data corresponding to the output information generated in the decoding process. The matrix operation block MAB may generate intermediate data using the parameters read in step S4-1.

[0132] The matrix operation block MAB may generate new intermediate data using the intermediate data read in step S4-2. The matrix operation block MAB may generate new intermediate data using at least a portion of the intermediate data generated in the prefill process. The matrix operation block MAB may generate new intermediate data using at least a portion of the intermediate data generated in the decode process. The matrix operation block MAB may generate new intermediate data using at least a portion of the intermediate data generated in the prefill process and at least a portion of the intermediate data generated in the decode process.

[0133] The matrix operation block MAB may store the newly generated intermediate data in the memory block MB. The matrix operation block MAB may store at least a portion of the new intermediate data in the memory block MB. The matrix operation block MAB may store the new intermediate data in the memory block MB connected to the matrix operation block MAB. At least a portion of the intermediate data stored in the memory block MB may be read out in the next decoding process.

[0134] In step S4-4, the matrix operation block MAB of the arithmetic unit 100 generates output information to be output from the machine learning model M. The matrix operation block MAB may generate output information by computing the multilayer perceptron MLP of the machine learning model M. The matrix operation block MAB may generate output information using the parameters read in step S4-1. The matrix operation block MAB may generate output information using the intermediate data read in step S4-2 and the intermediate data generated in step S4-3.

[0135] The matrix operation block MAB may store the generated output information. The matrix operation block MAB may store the output information in its local memory LM1, or it may store the output information in the memory block MB connected to the matrix operation block MAB.

[0136] The matrix operation block MAB may output the generated output information. The matrix operation block MAB may output output information containing a predetermined number of tokens. The matrix operation block MAB may output output information at predetermined time intervals. The output information output from the matrix operation block MAB may be output to the outside of the arithmetic unit 100. As an example, the output information may be transferred to the host memory HOSTM.

[0137] The matrix operation block MAB may repeatedly execute the processes from steps S4-1 to S4-4. The matrix operation block MAB may repeatedly execute the processes from steps S4-1 to S4-4 until a predetermined condition is met. For example, the matrix operation block MAB may repeatedly execute the processes from steps S4-1 to S4-4 until the end information of the output information is generated in step S4-4. The end information of the output information may be, for example, an EOS token (End of Sequence Token). As another example, the matrix operation block MAB may repeatedly execute the processes from steps S4-1 to S4-4 until the number of repetitions reaches a predetermined number. The matrix operation block MAB may read all the parameters stored in the memory block MB in step S4-1 and repeatedly execute only the processes from steps S4-2 to S4-4.

[0138] As another example, the arithmetic unit 100 may perform a decoding process of the state-space model. As another example, the arithmetic unit 100 may input the last token generated by the prefill process into the state-space model. The state-space model may read the state of the state-space model stored in the memory block MB and perform forward processing on the input token. The state-space model may generate a token following the input token and update the state of the state-space model. The arithmetic unit 100 may acquire the token output by the state-space model as output information. The arithmetic unit 100 may overwrite the state stored in the memory block MB with the updated state of the state-space model.

[0139] (Calculation using a multilayer perceptron) This section provides a more detailed explanation of multilayer perceptron (MLP) calculations. As an example, we will describe the calculation of a multilayer perceptron MLP during the decoding process.

[0140] In this embodiment, the multilayer perceptron MLP takes x as the input of a 32 × 8192 matrix, and W up Let W be the parameter of an 8192×32768 matrix. down Let be the parameter of a 32768×8192 matrix, and let f be the activation function, defined by equations (1) and (2).

[0141]

number

[0142] However, the activation function f may be either a ReLU function or a SiLU function. The ReLU function may be the function defined by equation (3). The SiLU function may be the function defined by equation (4).

[0143]

number

[0144] A key feature of multilayer perceptron (MLP) calculations is that one of the inputs in the matrix multiplication calculation is a long, narrow matrix, such as a 32x8192 matrix. The number 32 corresponds to the batch size. Small batch sizes offer advantages such as reduced token-to-token latency and lower memory usage.

[0145] Conventional processors cannot compute such matrix products quickly. The parameter W is related to the computational complexity. up ,W down Because the size is large, parameter W is taken from memory. up ,W down This is because the speed at which data is read is slower than the calculation speed.

[0146] The arithmetic unit 100 realizes high-speed reading of the parameter W up ,W down . In the arithmetic unit 100, a plurality of matrix operation blocks MAB share and execute matrix multiplication.

[0147] As an example, the product x·W of the input information x and the parameter W up may be divided into 96 matrix operation blocks MAB for processing as follows. The input information x is a 32×8192 matrix. The parameter W up is an 8192×32768 matrix. up is an 8192×32768 matrix.

[0148] Two blocks L2B per logic die LD have the input information x redundantly. Also, the two blocks L2B each have the parameter W up split in the column direction into two parameters W up (1,2) [[ID=Big5]] up (1,2) up is an 8192×16384 matrix.

[0149] Note that the calculation of x·W up can be realized by calculating the following processes (1) to (2) in parallel. Here, the 0th block L2B[0] has the parameter W up (1,2) 0, and the 1st block L2B[1] has the parameter W up (1,2) 1. / / 这里原文中这个标签下没有内容,翻译后也保留空行

[0150] (1) Block L2B[0] calculates the matrix product x·W up (1,2) 0 of the input information x and the parameter W up (1,2) The calculation result is a 32×16384 matrix.

[0151] (2) Block L2B[1] calculates the matrix product x·W up (1,2) 1 of the input information x and the parameter W up (1,2)Calculate 1. The result is a 32x16384 matrix.

[0152] Each block L2B has four L1B blocks that duplicate the input information x. Also, each of the four L1B blocks has a parameter W. up (1,2) The parameter W is obtained by dividing the column into four parts. up (1,8) Each has the following parameter W up (1,8) This is an 8192×4096 matrix.

[0153] Each block L1B has 16 matrix operation blocks MAB, which divide the input information x into four columns. (1,4) Each has its own input information x (1,4) This is a 32 × 2048 matrix. Also, the 16 matrix operation blocks MAB have parameter W up (1,8) The parameter W obtained by dividing the matrix into four parts in each direction. up (4,32) Each has the following parameter W up (4,32) This is a 2048 x 1024 matrix.

[0154] Each of the 16 matrix operation blocks MAB is based on input information x (1,4) and parameter W up (4,32) Matrix product x (1,4) ·W up (4,32) The calculation is performed. The result is a 32x1024 matrix.

[0155] Next, the arithmetic unit 100 divides the 16 matrix operation blocks MAB in block L1B into groups of four. The arithmetic unit 100 calculates the result x of each matrix operation block MAB for each group. (1,4) ·W up (4,32) Add the values. Since there are 4 groups, the calculation result is x·W. up (1,8) This results in a 32 × 4096 matrix. The arithmetic unit 100 calculates the result x·W. up(1,8) This may be stored in memory L1BM.

[0156] The arithmetic unit 100 calculates the result x·W for each of the eight blocks L1B. up (1,8) Arrange and combine them. Calculation result x·W up This results in a 32 × 32768 matrix. The arithmetic unit 100 calculates the result x·W so that it can be used in the subsequent matrix multiplication. up The matrix operations may be divided and stored in memory blocks MB connected to each matrix operation block MAB.

[0157] As another example, the output f(y) of the activation function f and the parameter W down The product of x·W down The process may be divided into 96 matrix operation blocks MAB as follows. The output f(y) of the activation function f is a 32 × 32768 matrix. Parameter W down This is a 32768×8192 matrix.

[0158] Two blocks L2B per logic die LD produce an output y obtained by splitting the output f(y) of the activation function f in the column direction. (1,2) Each has its own output y. (1,2) This is a 32 × 16384 matrix. Also, the two blocks L2B have parameter W down The parameter W is obtained by dividing the row into two. down (2,1) Each has the following parameter W down (2,1) This is a 16384×8192 matrix.

[0159] Block L1B has 4 blocks per block L2B, output y (1,2) The output y is obtained by dividing the output into four parts in the column direction. (1,8) Each has its own output y. (1,8) This is a 32x4096 matrix. Also, the four blocks L1B have parameter W down (2,1) The parameter W is obtained by dividing the row into four sections. down (8,1) Each has the following parameter W down (8,1)is a 4096×8192 matrix.

[0160] Each of the 16 matrix operation blocks MAB per block L1B has an output y (1,8) where the output y is divided into 4 parts in the column direction (1,32) and each has an output y. (1,32) The output y is a 32×1024 matrix. Also, the 16 matrix operation blocks MAB each have a parameter W down (8,1) where the parameter W is divided into 4 parts in the row direction down (32,4) and each has a parameter W. down (32,4) The parameter W is a 1024×2048 matrix.

[0161] Each of the 16 matrix operation blocks MAB calculates the matrix product y (1,32) and the parameter W down (32,4) y (1,32) ·W down (32,4) The calculation result is a 32×2048 matrix.

[0162] Subsequently, the arithmetic unit 100 divides the 16 matrix operation blocks MAB per block L1B into 4 groups of 4 each. For each group, the arithmetic unit 100 adds the calculation results y (1,32) ·W down (32,4) Since there are 4 groups, the calculation result y (1,8) ·W down (8,1) is a 32×8192 matrix. The arithmetic unit 100 may store the calculation result y (1,8) ·W down (8,1) in the memory L1BM.

[0163] Each of the 4 blocks L1B per block L2B adds the calculation results y (1,8) ·W down (8,1) The calculation result y (1,2) ·W down(2,1) This becomes a 32 × 8192 matrix. The arithmetic unit 100 calculates the result y (1,2) ·W down (2,1) This may be stored in memory L2BM.

[0164] Two blocks L2B per logic die LD calculate the result y of each of the four blocks L1B. (1,2) ·W down (2,1) Add the result y·W. down This results in a 32 × 8192 matrix. The arithmetic unit 100 calculates the results y·W of each of the two blocks L2B so that they can be used in the subsequent matrix multiplication. down One or both of these may be divided and stored in memory blocks MB connected to each matrix operation block MAB.

[0165] The above calculations may be performed in series or in parallel. For example, if the matrix multiplication calculation is partially completed, the summation may be performed in parallel. For example, if the matrix multiplication calculation is completed in multiple blocks L1B, the sum of the resulting matrices may be calculated.

[0166] The above is an outline of the process of calculating matrix products using the arithmetic unit 100. The parameters W of the machine learning model M. up ,W down Because the amount of data is large, in this embodiment, the parameter W is set in the multiple matrix operation blocks MAB provided by the arithmetic unit 100. up ,W down The parameters W are divided and allocated, and stored in the memory block MB connected to each matrix operation block MAB. In this embodiment, each matrix operation block MAB has parameter W up ,W down The data is divided in a way that avoids duplication. This allows the computing unit 100 to quickly read the parameters of the machine learning model M from the memory block MB.

[0167] matrix product x W up In the calculation of the matrix product x, the matrix operation block MAB calculates the matrix product x(1,4) ·W up (4,32) Let's explain this in more detail. Input information x (1,4) This is a 32 × 2048 matrix. Parameter W up (4,32) This is a 2048 x 1024 matrix.

[0168] The matrix operation block MAB includes a register called a matrix register. For example, the matrix register may be included in the matrix arithmetic unit MAU of the matrix operation block MAB. In this embodiment, the matrix register is capable of storing a 32x64 matrix. First, the matrix operation block MAB writes one of the 32x64 matrices M for which the product is to be calculated to the matrix register. Next, the matrix operation block MAB inputs the other matrix for which the product is to be calculated, a 64x2 matrix B and a 32x2 matrix C, into the matrix register. The 32x2 matrix C is also called the "accumulator". The matrix register calculates D = M·B + C and outputs the 32x2 matrix D. Note that matrices M and B may be quantized with FP8. Also, matrices C and D may be quantized with FP16.

[0169] Calculations using matrix registers are performed once per cycle by each matrix operation block (MAB). As an example, we will explain how to calculate the matrix product of two 64x64 matrices. Let the two matrices to be multiplied be A and B.

[0170] Here, we introduce a notation that uses subscripts to represent submatrices. A i:j,k:l Let this be a (ji)×(lk) matrix obtained by extracting rows i through j-1 and columns k through l-1 from matrix A. As an example, A 0:32,0:64 This is a submatrix obtained by extracting the upper half of matrix A. Also, A 0:32,0:32 This is a submatrix obtained by extracting the upper left half of matrix A.

[0171] First, the matrix operation block MAB is A 0:32,0:64 This is written to the matrix register. Next, the matrix operation block MAB calculates equation (5) for an integer i between 0 and 31 (inclusive). Note that 0 is the zero matrix.

[0172]

number

[0173] Next, the matrix operation block MAB is A 32:64,0:64 The result is written to the matrix register. Next, the matrix operation block MAB calculates equation (6) for an integer i between 0 and 31 (inclusive).

[0174]

number

[0175] This completes the multiplication of two 64x64 matrices. Note that if the matrix is ​​larger than a 64x64 matrix, you can substitute the zero matrix with the result of the previous calculation.

[0176] In a large-scale language model, which is an example of a machine learning model M, a matrix called an activation (such as x or y mentioned above) is written to a matrix register, and the parameter W up ,W down The process writes 64 x 2 of the data to the matrix register per cycle. The arithmetic unit 100 processes parameter W up ,W down This allows for high-speed reading from the memory block MB. Therefore, the inference process of the machine learning model M can be executed at high speed using the arithmetic unit 100.

[0177] When performing inference processing of a machine learning model M using the computing unit 100, data movement between matrix operation blocks MAB can be reduced by appropriately dividing the parameters of the machine learning model M. The computing unit 100 is configured so that each of the multiple processing elements PE can treat the memory block MB as local memory. Therefore, by storing the parameters used by each processing element PE in the memory block MB connected to that processing element PE, the inference processing of the machine learning model M can be executed at high speed.

[0178] Up to this point, we have explained the calculation of the multilayer perceptron MLP in the decoding process. In the calculation of the multilayer perceptron MLP in the decoding process, the input information x is a 32 × 8192 matrix, and the number of rows, 32, was the batch size. On the other hand, in the calculation of the multilayer perceptron MLP in the prefill process, calculation may be performed with a predetermined number of tokens as the processing unit for one input information (in other words, a batch size of 1). That is, in the calculation of the multilayer perceptron MLP in the decoding process, the number of rows of the input information x, 32, corresponded to the batch size, but in the calculation of the multilayer perceptron MLP in the prefill process, the number of rows of the input information x, 32, corresponds to the number of tokens input to the machine learning model M.

[0179] (Attention calculation) The calculation of Attention (ATT) will be explained in more detail. Here, as an example, the calculation of a multilayer perceptron (MLP) in the decoding process will be described.

[0180] The calculation of Attention (ATT) may include parts similar to those of the multilayer perceptron MLP calculation and parts that calculate the attention itself. The part that calculates the attention itself may include the process of calculating the key-value cache. The parts similar to those of the multilayer perceptron MLP calculation will be omitted from the explanation below.

[0181] In this embodiment, the attention ATT takes a query vector Q, a key vector K, and a value vector V as inputs. The query vector Q is a 32 × 8192 matrix. The key vector K is a 32 × L × 1024-dimensional vector. The value vector V is a 32 × L × 1024-dimensional vector, where L is an integer called the context length. The context length L increases by 1 each time the machine learning model M performs token inference. Therefore, the attention ATT calculation needs to be configured to accommodate various values ​​of L.

[0182] The arithmetic unit 100 divides the query vector Q, key vector K, and value vector V input to the attention ATT into 64 heads. For example, the arithmetic unit 100 divides the query vector Q into 64 parts, resulting in 64 32×128 matrices q. The arithmetic unit 100 assigns these 64 matrices q to 64 heads. The arithmetic unit 100 also divides the key vector K into 8 parts, resulting in 8 32×L×128 tensors k. The arithmetic unit 100 duplicates these 8 tensors k and assigns them to 64 heads. Furthermore, the arithmetic unit 100 divides the value vector V into 8 32×L×128 tensors v, duplicates these 8 tensors k, and assigns them to 64 heads.

[0183] Hereinafter, the matrix q, tensor k, and tensor v assigned to the head can be considered as 32 128-dimensional column vectors q', an L×128-dimensional tensor k', and an L×128-dimensional tensor v', respectively. The arithmetic unit 100 calculates equation (7) for the 32 q', k', and v'.

[0184]

number

[0185] however,· T The superscript T indicates transpose. The notation for the Softmax function will be omitted below.

[0186] The calculation of the attention itself involves, primitively speaking, the product of an 8x128 matrix and a 128xL matrix, and the product of an 8xL matrix and an Lx128 matrix, for each of the 8 heads. Since L is usually larger than 8, the speed of reading the 128xL matrix and Lx128 matrix from the memory block MB becomes a problem. By appropriately partitioning the query vector Q, key vector K, and value vector V, the calculation can be sped up in the same way as the calculation of a multilayer perceptron MLP. Here, the key vector K and value vector V are each 32xLx1024 tensors, and the query vector Q is a 32x8192 matrix, so the partitioning is made with an emphasis on the larger key vector K and value vector V.

[0187] Two blocks L2B per logic die LD are a tensor K obtained by dividing the key vector K into two parts. (1,1,2) Each has a tensor K. (1,1,2) This is a 32 × L × 5¹² tensor. Also, the two blocks L2B are tensors obtained by dividing the key vector V into two parts. (1,1,2) Each has a tensor V. (1,1,2) This is a 32 × L × 512 tensor. Furthermore, the two blocks L2B are matrices Q obtained by dividing the query vector Q into two parts. (1,2) Each has the following: Matrix Q (1,2) This is a 32x4096 matrix.

[0188] Block L1B has 4 blocks per block L2B, and tensor K (1,1,2) Tensor K divided into 4 parts (1,1,8) Each has a tensor K. (1,1,8) This is a 32 × L × 128 tensor. Also, the four blocks L1B are tensor V. (1,1,2) Tensor V divided into 4 parts (1,1,8) Each has a tensor V. (1,1,8) This is a 32 × L × 128 tensor. Furthermore, the four blocks L1B are matrix Q (1,2) Matrix Q obtained by dividing the matrix into four parts. (1,8) Each has the following: Matrix Q (1,8) This is a 32x1024 matrix.

[0189] Block L1B has 16 matrix operations, and block MAB has tensor K (1,1,8) Tensor K divided into 16 parts (1,16,8) Each has a tensor K. (1,16,8) This is a 32 × (L / 16) × 128 tensor. Also, the 16 matrix operation blocks MAB are tensor V. (1,1,8) Tensor V divided into 16 parts (1,16,8) Each has a tensor V. (1,16,8) This is a 32 × L / 16 × 128 matrix. Furthermore, the 16 matrix operation blocks MAB are matrix Q (1,8) It has duplicates.

[0190] Each of the four blocks L1B repeats the following process (1) to (6) 32 times.

[0191] (1) Block L1B is a 32 × 10²⁴ matrix Q (1,8) Take one row (i.e., a 1024-dimensional vector) from it. Block L1B treats the extracted 1024-dimensional vector as an 8x128 matrix q.

[0192] (2) Block L1B is a 32 × L × 128 tensor K (1,1,8) Extract the L×128 matrix k corresponding to matrix q.

[0193] (3) Block L1B is q@k T We calculate this and obtain an 8×L matrix t.

[0194] (4) Block L1B calculates the Softmax function for each row of matrix t and takes the result as s.

[0195] (5) Block L1B is a 32 × L × 128 tensor V (1,1,8) Extract the L×128 matrix v corresponding to matrix q from the matrix.

[0196] (6) Block L1B calculates s@v and obtains an 8x128 matrix. Block L1B considers the 8x128 matrix to be a 1024-dimensional vector.

[0197] Each of the four blocks L1B combines 32 1024-dimensional vectors obtained by repeating the process 32 times to obtain a 32 × 1024 matrix. Since the arithmetic unit 100 has eight blocks L1B within the logic die LD, a total of eight 32 × 1024 matrices are obtained. The arithmetic unit 100 combines these eight 32 × 1024 matrices to obtain a 32 × 8192 matrix. The 32 × 8192 matrix thus obtained is the calculation result of the attention unit.

[0198] In the calculation of block L1B described above, the matrix operation block MAB within block L1B repeats the following processes (1) to (8) 32 times.

[0199] (1) The 16 matrix operation blocks MAB each have a duplicate 8 × 128 matrix q.

[0200] (2) The matrix operation block MAB writes the lower half of matrix q, an 8x64 matrix, to the matrix register.

[0201] (3) Memory block MB stores a (L / 16) × 128 matrix k. Matrix operation block MAB reads 2 × 64 matrices from matrix k per cycle and calculates the product with the lower half of matrix q. The result of the calculation is an 8 × (L / 16) matrix. This calculation takes L / 32 cycles.

[0202] (4) The matrix operation block MAB writes the 8x64 matrix, which is the upper half of matrix q, to ​​the matrix register.

[0203] (5) The matrix operation block MAB calculates the product of matrix k and the upper half of matrix q in the same manner as in process (3). The result of the calculation is an 8 × (L / 16) matrix. The matrix operation block MAB calculates the sum of the two 8 × (L / 16) matrices. Note that the calculations in processes (2) to (5) may be performed by adding the matrix products after each calculation, or they may be performed all at once using the matrix product-sum operation.

[0204] (6) The matrix operation block MAB calculates the Softmax function for each row of the 8 × (L / 16) matrix, and the result is denoted as s. At this time, the matrix operation block MAB needs to communicate with other matrix operation blocks MAB in the same block L1B.

[0205] (7) The matrix operation block MAB calculates an 8 × (L / 16) matrix s and an (L / 16) × 128 matrix v. Specifically, the matrix operation block MAB repeats the following process ceil(L / 1024) times. First, it writes the 8 × 64 portion of matrix s to the matrix register. Next, it reads the 64 × 2 portion of matrix v from the memory block MB per cycle and inputs it to the matrix register. Accumulate is performed appropriately at this time. As a result, an 8 × 128 matrix is ​​obtained.

[0206] (8) The 16 matrix operation blocks MAB treat the 8 × 128 matrices as 1024-dimensional vectors and calculate their sum.

[0207] As a result of repeating processes (1) to (8) 32 times, block L1B yields a 32 × 10²⁴ matrix. The arithmetic unit 100 yields a total of 32 × 8192 matrices.

[0208] In the calculation of block L1B, it is important that matrices k and v are read in parallel from each matrix operation block MAB. If matrices k and v are appropriately partitioned and stored in memory block MB, the above calculation procedure can be realized. The method for partitioning matrices k and v is naturally derived from the above calculation procedure. That is, matrices k and v are channel-partitioned in two blocks L2B, channel-partitioned in four blocks L1B within block L2B, and the context length L is partitioned in sixteen matrix operation blocks MAB within block L1B.

[0209] Matrix k and matrix v are generated each time a new token is inferred, but they can be written to the appropriate memory block MB each time. For example, if a 1024-dimensional vector k is generated, the vector k may be divided into eight 128-dimensional vectors, each assigned to one of the eight blocks L1B. If matrix k corresponds to the nth token, the 128-dimensional vector may be assigned to the matrix operation block MAB corresponding to the remainder when n is divided by 16. The matrix operation block MAB may write the 128-dimensional vector assigned to it to the memory block MB connected to it.

[0210] Up to this point, we have explained the calculation of attention (ATT) in the decoding process. Below, as another example, we will explain the calculation of attention (ATT) in the prefill process. Here, we will focus on the differences between the calculation of attention (ATT) in the prefill process and the calculation of attention (ATT) in the decoding process. Unless otherwise specified, the calculation of attention (ATT) in the prefill process may be constructed in the same way as the calculation of attention (ATT) in the decoding process.

[0211] The calculation of attention (ATT) in the prefill process differs from the calculation of attention (ATT) in the decode process in the following respects:

[0212] (1) Consider the 32×8192 matrix q as a 32×8×8×128 tensor.

[0213] (2) Eight blocks L1B per logic die LD are associated with a 32×8×128 tensor obtained by dividing a 32×8×8×128 tensor into eight parts.

[0214] (3) The 16 matrix operation blocks MAB within block L1B have duplicate 32 × 8 × 128 tensors.

[0215] (4) The matrix operation block MAB performs the following operations <1> ~ <7> Repeat this 8 times.

[0216] <1>The matrix operation block MAB writes a 32×64 matrix, which is the lower half of matrix q, into the matrix register.

[0217] <2>The memory block MB stores an (L / 16)×128 matrix k. The matrix operation block MAB reads a 2×64 matrix from matrix k per cycle and calculates the product with the lower half of matrix q. The calculation result is a 32×(L / 16) matrix. This calculation takes L / 8 cycles.

[0218] <3>The matrix operation block MAB writes a 32×64 matrix, which is the upper half of matrix q, into the matrix register.

[0219] <4>The matrix operation block MAB calculates the product of matrix k and the upper half of matrix q in the same way as in process <2>. The calculation result is a 32×(L / 16) matrix. The matrix operation block MAB calculates the sum of two 32×(L / 16) matrices. Note that the calculations in processes <1> to <4> may be added after calculating the matrix products respectively, or may be calculated once by the matrix product-sum operation.

[0220] <5>The matrix operation block MAB calculates the Softmax function for each row of the 32×(L / 16) matrix and sets the calculation result as s. At this time, the matrix operation block MAB needs to communicate with other matrix operation blocks MAB within the same block L1B.

[0221] <6>The matrix operation block MAB calculates a 32×(L / 16) matrix s and an (L / 16)×128 matrix v. Specifically, the matrix operation block MAB repeats the following process ceil(L / 1024) times. First, it writes the 32×64 part of matrix s into the matrix register. Next, it reads 64×2 parts of matrix v from the memory block MB per cycle and inputs them into the matrix register, repeating this 64 times. At this time, accumulate is performed appropriately. As a result, a 32×128 matrix is obtained.

[0222] <7> The 16 matrix operation blocks MAB treat 32x128 matrices as 4096-dimensional vectors and calculate their sums.

[0223] process <1> ~ <7> As a result of repeating this process eight times, block L1B yields a 32 × 10²⁴ matrix. The arithmetic unit 100 yields a total of 32 × 8192 matrices.

[0224] The matrix operation block MAB may execute the prefill process in step S3 and the decode process in step S4 in parallel. In other words, the matrix operation block MAB may execute the prefill process in some batches while executing the prefill process in other batches.

[0225] As an example, suppose the matrix operation block MAB has a batch size of 17, with 16 prefill processes completed and 1 prefill process incomplete. In this case, the matrix operation block MAB may execute 16 decode processes and 1 prefill process in parallel. Specifically, the matrix operation block MAB may input a total of 32 tokens into the machine learning model M: 16 tokens from one prefill process and 1 token each from 16 decode processes. In calculating the attention body, the matrix operation block MAB only needs to perform the calculation of the decode process with a batch size of 16 and the calculation of the prefill process with 16 tokens.

[0226] The attention body of the Lth token can be calculated by constructing the Softmax function as follows.

[0227] Softmax((q of the Lth token)@(k of tokens 1 to L) T @(v of the 1st to Lth tokens))

[0228] By using the computing device 100 to perform inference processing of a machine learning model, for example, the throughput of the machine learning model's inference processing can be improved. Throughput may include, for example, throughput per user (e.g., the number of tokens that can be processed per second) or throughput per cost (e.g., the number of tokens that can be processed per unit of currency).

[0229] Throughput per user is throughput from the user's perspective. As an example, consider an information processing system that interacts with users using a machine learning model. Some such systems begin outputting responses only after receiving all user input, with the responses being displayed intermittently on the screen. Throughput per user can be described as the speed at which the responses are displayed on the screen.

[0230] Throughput per user is primarily determined by the following factors: • Processor memory bandwidth and capacity • Degree of parallelism in tensor parallelism • Network bandwidth used for tensor parallel processing Batch size

[0231] The batch size should be determined appropriately during the design of the information processing system. If the batch size is appropriately defined and the network bandwidth is sufficient, the throughput per user depends on the time required to retrieve machine learning model parameters from memory. For example, throughput per user is proportional to the speed at which machine learning model parameters are retrieved from memory.

[0232] Using the arithmetic unit 100, the reading of machine learning model parameters can be accelerated. Furthermore, by parallelizing the reading of machine learning model parameters using tensor parallel processing, the throughput per user can be significantly improved. For example, compared to a GPGPU (General Purpose Graphics Processing Unit) equipped with HBM (High Bandwidth Memory), the throughput per die can be improved by about 5 to 10 times. As a result, the cost per die can be reduced to about one-fifth.

[0233] Furthermore, this embodiment improves throughput per user, making it easier to handle large machine learning models. For example, because SRAM has limited storage capacity, attempting to perform inference processing on a large machine learning model requires allocating a large number of processors to a single machine learning model. Also, improved throughput per user reduces the latency or bandwidth requirements of the inter-chip network. As a result, the overall system cost can be reduced.

[0234] In the embodiments described above, some or all of the inference processing of the machine learning model may be implemented in hardware, or it may be implemented by information processing of software (programs) executed by a CPU, GPU (Graphics Processing Unit), etc. If it is implemented by information processing of software, the software that implements at least some of the functions of the machine learning model inference processing in the embodiments described above may be stored on a non-temporary storage medium (non-temporary computer-readable medium) such as a CD-ROM (Compact Disc-Read Only Memory) or USB (Universal Serial Bus) memory, and the information processing of the software may be executed by loading it into a computer. Alternatively, the software may be downloaded via a communication network. Furthermore, all or part of the processing of the software may be implemented in a circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA, so that the information processing by the software is executed in hardware.

[0235] The storage medium for the software may be a removable medium such as an optical disc, or a fixed storage medium such as a hard disk or memory. Furthermore, the storage medium may be located inside the computer (main memory, auxiliary storage, etc.) or outside the computer.

[0236] Figure 21 is a block diagram showing an example of the hardware configuration of a computer equipped with the above-described arithmetic unit 100. In the following description, we will assume that the computer is equipped with the arithmetic unit 100. In Figure 21, the computer may be implemented as a computer 500 comprising, for example, the arithmetic unit 100, main memory 30 (memory), auxiliary memory 40 (memory), network interface 50, and device interface 60, all connected via a bus 510.

[0237] The computer 500 in Figure 21 has one of each component, but it may have multiple identical components. Also, although Figure 21 shows one computer 500, the software may be installed on multiple computers, and each of these computers may execute the same or different parts of the software's processing. In this case, it may be a distributed computing configuration in which each computer communicates via a network interface 50 or the like to execute processing. In other words, a system may be configured in which one or more computers 500 execute instructions stored in one or more storage devices to realize functionality. Alternatively, information transmitted from a terminal may be processed by one or more computers 500 located on the cloud, and the processing results may be transmitted to the terminal.

[0238] Various calculations may be performed in parallel using one or more arithmetic units 100 installed in the computer 500, or using multiple computers 500 via a network. Alternatively, various calculations may be distributed to processing elements PE, which are examples of multiple arithmetic cores within the arithmetic unit 100, and performed in parallel. Furthermore, some or all of the processing and means of this disclosure may be implemented by at least one of a processor and a storage device located on a cloud that can communicate with the computer 500 via a network. Thus, each device in the embodiments described above may be in the form of parallel computing using one or more computers.

[0239] The arithmetic unit 100 may be at least an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.) that performs either computer control or calculations. Alternatively, the arithmetic unit 100 may be either a dedicated processing circuit designed to perform specific calculations or an arithmetic unit including both a general-purpose processor and a dedicated processing circuit. Furthermore, the arithmetic unit 100 may include optical circuits or quantum computing-based calculation functions.

[0240] The arithmetic unit 100 may perform arithmetic processing based on data and software input from each device and the like of the internal configuration of the computer 500, and may output the arithmetic result and control signal to each device and the like. The arithmetic unit 100 may control each component constituting the computer 500 by executing the OS (Operating System) of the computer 500, applications, and the like.

[0241] The inference processing of the machine learning model in the above-described embodiment may be realized by one or a plurality of arithmetic units 100. Here, the arithmetic unit 100 may refer to one or a plurality of electronic circuits arranged on one chip, or may refer to one or a plurality of electronic circuits arranged on two or more chips or two or more devices. When using a plurality of electronic circuits, each electronic circuit may communicate by wire or wirelessly.

[0242] The main storage device 30 may store instructions and various data executed by the arithmetic unit 100, and the information stored in the main storage device 30 may be read by the arithmetic unit 100. The auxiliary storage device 40 is a storage device other than the main storage device 30. These storage devices mean any electronic component capable of storing electronic information, and may be a semiconductor memory. The semiconductor memory may be either a volatile memory or a non-volatile memory. The storage device for storing various data and the like in the computer 500 may be realized by the main storage device 30 or the auxiliary storage device 40, or may be realized by the built-in memory built in the arithmetic unit 100.

[0243] If the computer 500 consists of at least one storage device (memory) and at least one arithmetic unit 100 connected to (coupled with) this at least one storage device, then at least one arithmetic unit 100 may be connected to one storage device. Also, at least one storage device may be connected to one arithmetic unit 100. Furthermore, the configuration may include at least one of the multiple arithmetic units 100 being connected to at least one of the multiple storage devices. This configuration may also be realized by the storage devices and arithmetic units 100 included in multiple computers 500. In addition, the configuration may include a storage device integrated with the arithmetic unit 100 (for example, a cache memory including an L1 cache and an L2 cache).

[0244] The network interface 50 is an interface for connecting to the communication network 600 wirelessly or via a wired connection. The network interface 50 can be any appropriate interface, such as one conforming to existing communication standards. Information may be exchanged between the computer 500 and an external device 710 connected via the communication network 600 through the network interface 50. The communication network 600 may be a WAN (Wide Area Network), LAN (Local Area Network), PAN (Personal Area Network), or a combination thereof, as long as it allows for information exchange between the computer 500 and the external device 710. Examples of WANs include the Internet, examples of LANs include IEEE 802.11 and Ethernet (registered trademark), and examples of PANs include Bluetooth (registered trademark) and NFC (Near Field Communication).

[0245] The device interface 60 is an interface such as USB that connects directly to the external device 720.

[0246] External device 710 is a device connected to computer 500 via a network. External device 720 is a device directly connected to computer 500.

[0247] External device 710 or external device 720 may, for example, be an input device. The input device may be a camera, microphone, motion capture device, various sensors, keyboard, mouse, touch panel, etc., and will provide the acquired information to the computer 500. Alternatively, it may be a personal computer, tablet terminal, smartphone, or other device equipped with an input unit, memory, and processor.

[0248] Furthermore, the external device 710 or external device 720 may, for example, be an output device. The output device may be a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) panel, or it may be a speaker that outputs sound, etc. It may also be a device equipped with an output unit, memory, and a processor, such as a personal computer, tablet terminal, or smartphone.

[0249] Furthermore, the external device 710 or external device 720 may be a storage device (memory). For example, the external device 710 may be network storage, and the external device 720 may be storage such as an HDD.

[0250] Furthermore, the external device 710 or external device 720 may be a device that has some of the functions of the computer 500. In other words, the computer 500 may transmit some or all of the processing results to the external device 710 or external device 720, or may receive some or all of the processing results from the external device 710 or external device 720.

[0251] In this specification (including the claims), when the expression "at least one of a, b, and c" or "at least one of a, b, or c" (including similar expressions) is used, it includes any of a, b, c, ab, ac, bc, or abc. Furthermore, any element may have multiple instances, such as aa, abb, aabbcc, etc. In addition, it is also possible to add other elements other than the enumerated elements (a, b, and c), such as abcd which has d.

[0252] In this specification (including the claims), when expressions such as "using data as input / based on data / according to / in accordance with data" (including similar expressions) are used, unless otherwise specified, this includes using the data itself or using data that has been processed in some way (e.g., data with added noise, normalized data, features extracted from the data, intermediate representations of the data, etc.). Furthermore, when it is stated that some result is obtained "using data as input / based on data / according to / in accordance with data" (including similar expressions), unless otherwise specified, this includes cases where the result is obtained based solely on the data in question or where the result is influenced by other data, factors, conditions, and / or states other than the data in question. Furthermore, when it is stated that "data is output" (including similar expressions), unless otherwise specified, this includes cases where the data itself is used as output or where data that has been processed in some way (e.g., data with added noise, normalized data, features extracted from the data, intermediate representations of various types of data, etc.) is used as output.

[0253] In this specification (including the claims), the terms “connected” and “coupled” are intended to be non-restrictive terms that include any direct connection / coupling, indirect connection / coupling, electrical connection / coupling, communicative connection / coupling, operational connection / coupling, physical connection / coupling, etc. The terms should be interpreted as appropriate in the context in which they are used, but any form of connection / coupling that is not intentionally or naturally excluded should be interpreted non-restrictively as being included in the terms.

[0254] In this specification (including the claims), when the expression "A configured to B" is used, it may include that the physical structure of element A has a configuration capable of performing operation B, and that the permanent or temporary setting / configuration of element A is configured to actually perform operation B. For example, if element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and that it is configured to actually perform operation B by the setting of a permanent or temporary program (instruction). Furthermore, if element A is a dedicated processor, dedicated arithmetic circuit, etc., it is sufficient that the circuit structure of the processor is implemented to actually perform operation B, regardless of whether control instructions and data are actually attached.

[0255] Wherever terms meaning "comprising" or "possessing" (e.g., "comprising / including," "having," etc.) are used herein, they are intended to be open-ended terms, including cases where the subject matter of such terms is not the object of the term. Where the object of such terms meaning "comprising" or "possessing" is an expression that does not specify a quantity or suggests a singular number (an expression with the article "a" or "an"), such expression should be interpreted as not being limited to a specific number.

[0256] In this specification (including the claims), even if expressions such as "one or more" or "at least one" are used in some places, and expressions that do not specify a quantity or suggest a singularity (expressions using the articles a or an) are used in other places, the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or suggest a singularity (expressions using the articles a or an) should not necessarily be interpreted as not being limited to a specific number.

[0257] In this specification, if a particular configuration of an embodiment is described as having a specific advantage or result, it should be understood, unless otherwise stated, that the same advantage or result can also be obtained from one or more other embodiments having that configuration. However, it should be understood that the presence or absence of such advantage or result generally depends on various factors, conditions, and / or states, and that the configuration does not necessarily guarantee that the advantage or result can be obtained. The advantage or result can only be obtained from the configuration described in the embodiment when various factors, conditions, and / or states are met, and the advantage or result cannot necessarily be obtained in the invention claimed to define that configuration or a similar configuration.

[0258] In this specification (including the claims), when terms such as "maximize" are used, they include finding the global maximum value, finding an approximation of the global maximum value, finding the local maximum value, and finding an approximation of the local maximum value, and should be interpreted appropriately depending on the context in which the term is used. They also include finding these approximations of maximum values ​​probabilistically or heuristically. Similarly, when terms such as "minimize" are used, they include finding the global minimum value, finding an approximation of the global minimum value, finding the local minimum value, and finding an approximation of the local minimum value, and should be interpreted appropriately depending on the context in which the term is used. They also include finding these approximations of minimum values ​​probabilistically or heuristically. Similarly, when terms such as "optimize" are used, they include finding the global optimal value, finding an approximation of the global optimal value, finding the local optimal value, and finding an approximation of the local optimal value, and should be interpreted appropriately depending on the context in which the term is used. This also includes finding approximate values ​​of these optimal values ​​probabilistically or heuristically.

[0259] In this specification (including the claims), when multiple hardware components perform a predetermined process, each component may cooperate to perform the predetermined process, or some components may perform all of the predetermined process. Alternatively, some components may perform part of the predetermined process, while other components perform the remainder. In this specification (including the claims), when expressions such as "one or more hardware components perform process A, and the one or more hardware components perform process B" (including similar expressions) are used, the hardware component performing process A and the hardware component performing process B may be the same or different. In other words, it is sufficient that the hardware component performing process A and the hardware component performing process B are included in the one or more hardware components. Hardware may include electronic circuits, devices containing electronic circuits, etc.

[0260] In this specification (including the claims), when multiple memory devices store data, each of the multiple memory devices may store only a portion of the data or the entire data. Furthermore, a configuration in which some of the multiple memory devices store data is also included.

[0261] While embodiments of this disclosure have been described in detail above, this disclosure is not limited to the individual embodiments described above. Various additions, modifications, substitutions, and partial deletions are possible, provided that they do not depart from the conceptual idea and spirit of the present invention derived from the claims and their equivalents. For example, where numerical values ​​or mathematical formulas are used in the description of the embodiments described above, these are provided for illustrative purposes only and do not limit the scope of this disclosure. Similarly, the sequence of operations shown in the embodiments is also illustrative and does not limit the scope of this disclosure.

[0262] Furthermore, the following forms are possible for disclosure technology.

[0263] (Note 1) Multiple arithmetic units, One or more first memories connected to the aforementioned plurality of arithmetic units, It comprises one or more second memories connected to the aforementioned plurality of arithmetic units, The one or more second memories are stacked on the plurality of arithmetic units, At least a portion of the data stored in the one or more second memories is used by the arithmetic unit without going through the one or more first memories. Computing device.

[0264] (Note 2) A first die comprising the plurality of arithmetic units and the one or more first memories, A second die comprising one or more second memories, The computing device described in Appendix 1.

[0265] (Note 3) The first die and the second die are stacked on a substrate. The first die and the second die are interconnected using a face-to-face connection method. The computing device described in Appendix 2.

[0266] (Note 4) The first die and the second die are interconnected by hybrid bonding. The computing device described in Appendix 3.

[0267] (Note 5) The first die and the second die are stacked on a substrate. The first die and the second die are interconnected using a back-to-face connection method. The computing device described in Appendix 2.

[0268] (Note 6) Of the first die and the second die, the die furthest from the substrate is connected to the die closer to the substrate via through vias provided in this die. The computing device described in Appendix 5.

[0269] (Note 7) The first die and a plurality of the second dies are stacked on a substrate. The computing device described in Appendix 2.

[0270] (Note 8) The aforementioned second die is One or more redundant memory units, The system includes a switching circuit that disconnects the connection between the faulty memory among the one or more second memories and the second die, and connects the one or more redundant memories to the first die. The arithmetic unit described in any one of the items in Appendix 2 to Appendix 7.

[0271] (Note 9) The aforementioned plurality of arithmetic units include one or more first arithmetic units and one or more second arithmetic units. A first memory interface connecting at least one preceding first arithmetic unit to one or more second memories, The system comprises a second memory interface connecting at least one of the second arithmetic units to one or more of the second memories, The arithmetic unit described in any one of the items from Appendix 1 to Appendix 8.

[0272] (Note 10) The first memory interface connects at least one of the first arithmetic units to a predetermined second memory among the one or more second memories, The second memory interface connects at least one of the second arithmetic units to another predetermined second memory among the one or more second memories. The computing device described in Appendix 9.

[0273] (Note 11) The aforementioned one or more second memories include one or more DRAMs, The arithmetic unit described in any one of the items in Appendix 1 to Appendix 10.

[0274] (Note 12) The one or more DRAMs are the scratchpad memories of the plurality of arithmetic units. The computing device described in Appendix 11.

[0275] (Note 13) Each of the following is a plurality of second-level blocks, each comprising the plurality of arithmetic units and the one or more first memories, It comprises one or more third memories connected to the plurality of second-level blocks, Each of the plurality of arithmetic units included in the plurality of second-level blocks is stacked in the one or more second memories. The arithmetic unit described in any one of the items in Appendix 1 to Appendix 12.

[0276] (Note 14) Each comprises multiple first-level blocks, each containing two or more arithmetic units included in the aforementioned multiple arithmetic units, Each of the aforementioned plurality of second-level blocks comprises the aforementioned plurality of first-level blocks. The computing device described in Appendix 13.

[0277] (Note 15) The first hierarchical block comprises one or more third arithmetic units that are commonly provided to the two or more arithmetic units. The computing device described in Appendix 14.

[0278] (Note 16) Each of the multiple third-level blocks comprises the multiple second-level blocks and the one or more third-level memories, A fourth memory with one or more elements, The arithmetic unit described in any one of the items in Appendix 13 to Appendix 15.

[0279] (Note 17) Each of the aforementioned plurality of arithmetic units is provided with one or more fifth memories having a smaller capacity than the one or more first memories. The arithmetic unit described in any one of the items in Appendix 1 to Appendix 16.

[0280] (Note 18) A third die stacked on the second die and having one or more third memories, The arithmetic unit described in any one of the items in Appendix 2 to Appendix 8.

[0281] (Note 19) The aforementioned computing device comprises a SIMD architecture, The arithmetic unit described in any one of the items in Appendix 1 to Appendix 18.

[0282] (Note 20) The aforementioned one or more second memories store the parameters of the machine learning model and the intermediate data calculated using the machine learning model. The plurality of arithmetic units retrieve the parameters and intermediate data from the one or more second memories, and use the retrieved parameters and intermediate data to execute the first process. The arithmetic unit described in any one of the items in Appendix 1 to Appendix 19.

[0283] (Note 21) The one or more second memories mentioned above store the intermediate data newly generated by the execution of the first process. The plurality of arithmetic units retrieve the parameters, the intermediate data, and the newly generated intermediate data from the one or more second memories, and use the retrieved parameters, the intermediate data, and the newly generated intermediate data to execute a second process. The computing device described in Appendix 20.

[0284] (Note 22) Each of the first and second processes includes a process for generating one or more output pieces of information using the machine learning model. The computing device described in Appendix 21.

[0285] (Note 23) The plurality of arithmetic units obtain the parameters and the intermediate data from the one or more second memories, The plurality of arithmetic units perform processing using the acquired parameters and the intermediate data, The one or more second memories store the intermediate data newly generated by the process as part of the intermediate data in the one or more second memories. This is repeated until the specified conditions are met. The arithmetic unit described in any one of the items in Appendix 20 to Appendix 22.

[0286] (Note 24) The machine learning model includes at least one of a transformer or a state-space model. The arithmetic unit described in any one of the items in Appendix 20 to Appendix 23.

[0287] (Note 25) The machine learning model includes the transformer, The aforementioned intermediate data includes at least a portion of the key vectors and value vectors used in the attention mechanism of the transformer. The computing device described in Appendix 24.

[0288] (Note 26) The machine learning model includes the state-space model, The aforementioned intermediate data includes at least a portion of the states in the state-space model, The computing device described in Appendix 24.

[0289] (Note 27) The system comprises a plurality of arithmetic units, one or more first memories connected to the plurality of arithmetic units, and one or more second memories connected to the plurality of arithmetic units, wherein the one or more second memories are stacked on the plurality of arithmetic units, and at least a portion of the data stored in the one or more second memories is an arithmetic method executed by an arithmetic device used by the arithmetic units without going through the one or more first memories. The aforementioned one or more second memories store the parameters of the machine learning model and the intermediate data calculated using the machine learning model. The plurality of arithmetic units retrieve the parameters and intermediate data from the one or more second memories, and use the retrieved parameters and intermediate data to execute the first process. Calculation method.

[0290] Furthermore, the processing performed by the arithmetic unit described in any one of the appendices 20 to 26 may be implemented by the arithmetic unit described in any one of the appendices 1 to 19 executing instructions generated by the compiler unit. [Explanation of Symbols]

[0291] 30 Main memory 40 Auxiliary storage 50 Network Interfaces 60 device interfaces 100, 101 Arithmetic device 102 Main storage 200 calculator 300 hosts 500 Computers 510 Bus 600 Communication Networks 710, 720 External device AD Address Signal CLK clock signal CMD command signals CNTL control signal DB Databus DSYNC Data Synchronization Signal GRF0, GRF1 registers HOSTM Host Memory IALU integer arithmetic unit L1B, L2B blocks L1BM, L2BM memory L1D L1 Data Cache L2D L2 Data Cache L3D L3 Data Cache LD Logic Die LM0, LM1 Local Memory LP0-LP16 ports MAB Matrix Operation Block MAU matrix operator MB memory blocks MCNT memory controller MD Memory Dial MI / F Memory Interface MP0-MP16 Ports M-REG Register MUX1, MUX2 Multiplexer PDM Memory PE Processing Element SW switching circuit T-REG Register

Claims

[Claim 1] Multiple arithmetic units, One or more first memories connected to the aforementioned plurality of arithmetic units, It comprises one or more second memories connected to the aforementioned plurality of arithmetic units, The one or more second memories are stacked on the plurality of arithmetic units, At least a portion of the data stored in the one or more second memories is used by the arithmetic unit without going through the one or more first memories. Computing device.

Images