Calculation device and calculation method

Abstract

The arithmetic device includes a plurality of arithmetic units, one or more first memories connected to the plurality of arithmetic units, and one or more second memories connected to the plurality of arithmetic units, the one or more second memories being stacked on the plurality of arithmetic units, and at least a portion of data stored in the one or more second memories being used by the arithmetic units without going through the one or more first memories. This makes it possible to provide a arithmetic device with a large memory capacity and a wide bandwidth in which a logic die and a memory die are three-dimensionally stacked.

Description

[Technical Field]

[0001] The present disclosure relates to a computing device and a computing method. [Background technology]

[0002] Computing devices used in applications that perform processing such as deep learning require high computing performance to perform a large number of matrix operations, and as a result, memories with both large storage capacity and bandwidth are required. Typically, internal memories installed in computing devices have large bandwidth but small storage capacity, while external memories connected to computing devices have large storage capacity but small bandwidth. For this reason, even with recent computing devices, it can be difficult to efficiently execute applications that require relatively large amounts of memory access. Summary of the Invention [Problem to be solved by the invention]

[0003] The present disclosure provides a computing device with a large storage capacity and a large bandwidth. [Means for solving the problem]

[0004] The arithmetic device according to an embodiment of the present disclosure includes a plurality of arithmetic units, one or more first memories connected to the plurality of arithmetic units, and one or more second memories connected to the plurality of arithmetic units. a plurality of second-level blocks each including the plurality of arithmetic units and the one or more first memories; and one or more third memories connected to the plurality of second-level blocks; Equipped with The plurality of computing units included in each of the plurality of second hierarchical blocks are The second memo of one or more of the above Li The one or more second memories are stacked, and at least a portion of the data stored in the one or more second memories is used by the computing unit without going through the one or more first memories. [Brief explanation of the drawings]

[0005] [Figure 1] 1 is an exploded perspective view showing an example of an outline of the configuration of a computing device according to a first embodiment of the present disclosure. [Figure 2] FIG. 2 is an exploded perspective view showing another example of the outline of the configuration of the arithmetic device according to the first embodiment of the present disclosure. [Figure 3] FIG. 10 is an exploded perspective view showing yet another example of the outline of the configuration of the arithmetic device according to the first embodiment of the present disclosure. [Figure 4] 10A to 10C are diagrams showing variations in the method of connecting the logic die and the memory die. [Figure 5] 2 is a plan view showing an example of the configuration of the logic die of FIG. 1. [Figure 6] FIG. 1 is a block diagram showing an example of a system including a computer on which a plurality of arithmetic units are installed and a host. [Figure 7] FIG. 1 is a diagram illustrating an example of connection between processing elements and memory blocks via a memory interface. [Figure 8] 6 is a circuit block diagram showing an example of a processing element PE shown in FIG. 5. [Figure 9] 6 is a circuit block diagram showing another example of the processing element PE shown in FIG. 5. FIG. [Figure 10] FIG. 2 is a block diagram showing an example of connections between the logic die and the memory die of FIG. 1. [Figure 11] FIG. 2 is a diagram illustrating an example of a logical configuration of each memory in the arithmetic device and a connection relationship between each memory. [Figure 12] FIG. 2 is a diagram illustrating an example of the arrangement of memories in a computing device. [Figure 13] FIG. 10 is a diagram illustrating another example of the logical configuration of each memory of the arithmetic device and the connection relationship of each memory. [Figure 14] FIG. 1 is a diagram illustrating an example of a logical configuration of a processor having a hierarchical cache and a main memory device. [Figure 15] FIG. 15 is a diagram illustrating an example of a physical configuration of the arithmetic unit and the main storage device illustrated in FIG. [Figure 16] FIG. 10 is a diagram showing an example of redundantly arranging matrix operation blocks on a logic die of an arithmetic device. [Figure 17] FIG. 1 is a diagram showing an example of redundantly arranging memory blocks in a memory die of a computing device. [Figure 18]FIG. 10 is a diagram showing an example in which matrix operation blocks and memory blocks are redundantly arranged in an arithmetic device. [Figure 19] FIG. 10 is a block diagram illustrating an example of a machine learning model according to a second embodiment of the present disclosure. [Figure 20] 20 is a flowchart showing an example of an inference process of the machine learning model shown in FIG. 19. [Figure 21] FIG. 2 is a block diagram showing an example of a hardware configuration of a computer on which the above-described arithmetic device is installed. DETAILED DESCRIPTION OF THE INVENTION

[0006] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Although not limited to this, the computing device may function as an accelerator that performs convolution operations and the like in the training or inference of deep neural networks, large-scale language models (LLMs), and the like. In this case, the computing device may be connected to a separate central processing unit (CPU) or graphics processing unit (GPU), and may function as a computer that performs scientific and technical calculations. Note that, in this specification (including the claims), terms such as "first," "second," and the like are used simply as a way to distinguish between two or more elements and are not necessarily intended to impose technical meanings such as temporal aspect, spatial aspect, order, or quantity on the subject. Thus, for example, a reference to a first element and a second element does not necessarily mean that only two elements can be employed therein, does not necessarily mean that the first element must precede the second element, or necessarily mean that the first element must be present for the second element to be present.

[0007] 1 to 3 are exploded perspective views showing an outline of the configuration of a computing device according to a first embodiment of the present disclosure. The computing device 100 shown in FIGS. 1 to 3 may have a logic die LD and a memory die MD that are three-dimensionally integrated. The logic die LD is an example of a first die, and the memory die MD is an example of a second die.

[0008] The logic die LD may have multiple processing elements PE that can operate in parallel. The processing element PE is an example of a processor. The logic die LD may be connected to one or more memory dies MD. As shown in FIGS. 1 to 3, the logic die LD and the memory die MD may be stacked and connected to each other via a data bus DB. A control signal line (not shown) used for reading and writing data may be connected between the logic die LD and the memory die MD. Furthermore, the term "stacked" in "dies are stacked" includes cases where the dies are arranged upward, downward, or both upward and downward. Therefore, for example, the memory block MB included in the memory die MD may be arranged upward, downward, or vertically above the matrix operation block MAB and the processing element PE included in the logic die LD. Furthermore, elements other than dies may be inserted between the dies to be stacked as long as high-bandwidth data transfer is maintained.

[0009] When the logic die LD and the memory die MD are stacked, the distance between the dies can be made equal to or less than the thickness of the dies. Therefore, compared to when the logic die LD and the memory die MD are arranged side by side on the same substrate, the load capacitance of the data bus DB can be significantly reduced, and the propagation delay time of data signals transmitted to the data bus DB can be significantly shortened. As a result, the arithmetic processing performance of the arithmetic device 100 can be improved.

[0010] As shown in FIGS. 1 and 2, a memory die MD may have multiple memory blocks MB arranged opposite a processing element PE. Each of the multiple memory blocks MB may be dedicated to the corresponding processing element PE. While FIGS. 1 and 2 show an example in which each memory die MD is connected to multiple processing elements PE, as shown in FIG. 3, each of the multiple memory dies MD may be arranged opposite multiple processing elements PE and connected to the corresponding processing element PE in a 1:1 ratio. That is, the memory die MD and the processing elements PE may be connected in a 1:m ratio (m is an integer greater than or equal to 1). Each of the multiple memory blocks MB included in one or more memory dies MD may be used as a scratchpad memory for the processing element PE connected to it.

[0011] 1 and 2, each memory die MD may include any number of memory blocks MB. The number of memory dies MD connected to a logic die LD may be determined according to the number of memory blocks MB included in each memory die MD. As shown in FIG. 2, the logic die LD may include one or more matrix arithmetic blocks MAB. Each matrix arithmetic block MAB may include one or more processing elements PE and a matrix arithmetic unit MAU.

[0012] For example, the memory die MD may be a dynamic random access memory (DRAM), a static random access memory (SRAM), a magnetoresistive random access memory (MRAM), a phase-change RAM (PRAM), a flash memory, etc. The memory die MD may include one or more DRAMs, one or more MRAMs, one or more PRAMs, one or more flash memories, etc.

[0013] By 3D integrating the logic die LD and memory die MD, the length of the data bus DB and control signal lines can be reduced to approximately the same as the thickness of each die, which minimizes wiring capacitance and wiring delay and suppresses increases in the charge / discharge current of the data bus DB and control signal lines.

[0014] The memory block MB and the memory die MD may be used as work memory for the processing element PE. Using a work memory external to the logic die LD can significantly increase the storage capacity of the work memory compared to using only the work memory internal to the logic die LD. This allows the arithmetic device 100 to have a large capacity and a wide bandwidth, enabling efficient execution of applications requiring frequent memory access. As a result, the arithmetic device 100 can execute deep neural network training, inference, scientific and technical calculations, and other tasks requiring large amounts of data processing at high speed and with low power consumption.

[0015] 1 to 3 show an example in which the processing elements PE, the matrix calculator MAU, the matrix calculation block MAB, and other calculators and the memory blocks MB are formed on different dies. However, the various calculators and the memory blocks MB may be formed on different layers of a single die, and the layer of the calculators and the layer of the memory blocks MB may be connected to each other via vias such as TSVs (Through Silicon Vias).

[0016] FIG. 4 illustrates variations in the method of connecting the logic die LD and the memory die MD. The logic die LD and the memory die MD may be connected to each other by a face-to-face connection method or a back-to-face connection method. In the face-to-face connection method, the logic die LD and the memory die MD may be connected to each other by placing semiconductor layers, on which elements such as transistors and wiring are formed on a Si substrate or the like, facing each other. In the back-to-face connection method, the logic die LD and the memory die MD may be connected to each other by placing the semiconductor layers on the opposite side of the printed circuit board. The computing device 100 may have the form of a package including the printed circuit board, the logic die LD, and the memory die MD.

[0017] Connection Examples 1 to 3 illustrate a face-to-face connection method, and Connection Examples 4 to 6 illustrate a back-to-face connection method. Note that the stacking structure (stacking order, number of stacked layers, etc.) between the logic die LD and the memory die MD and the stacking structure (stacking order, number of stacked layers, etc.) between the processing elements PE included in the logic die LD and the memory blocks MB included in the memory die MD are not limited to Connection Examples 1 to 6. Also, as shown in FIG. 2, the memory die MD may have multiple memory blocks MB, and the logic die LD may have multiple matrix operation blocks MAB each including multiple PEs.

[0018] As shown in Connection Example 1, the logic die LD and the memory die MD may be arranged in this order on the printed circuit board and connected face-to-face. As shown in Connection Example 2, the memory die MD and the logic die LD may be arranged in this order on the printed circuit board and connected face-to-face. As shown in Connection Example 3, multiple memory dies MD may be arranged on the logic die LD and the logic die LD may be connected face-to-face to each of the multiple memory dies MD.

[0019] In the face-to-face connection technique, the logic die LD and the memory die MD may be connected to each other by hybrid bonding or by microbumps. The data bus DB shown in FIGS. 1 and 2 may correspond to hybrid bonding or microbumps.

[0020] As shown in Connection Example 4, the logic die LD and the memory die MD may be arranged in this order on the printed circuit board with their semiconductor layers facing away from the printed circuit board, and may be connected back-to-face. In this case, the semiconductor layer of the memory die MD may be connected to the semiconductor layer of the logic die LD through a through via provided in the memory die MD.

[0021] As shown in Connection Example 5, the memory die MD and the logic die LD may be arranged in this order on a printed circuit board with their semiconductor layers facing away from the printed circuit board, and connected back-to-face. In this case, the semiconductor layer of the logic die LD may be connected to the semiconductor layer of the memory die MD via a through via such as a TSV provided in the logic die LD. The data bus DB shown in FIGS. 1 and 2 may correspond to the through via.

[0022] As shown in Connection Example 6, multiple memory dies MD may be stacked on a logic die LD. In this case, the stacked memory dies MD may be connected by through vias. Note that three or more memory dies MD may be stacked on a logic die.

[0023] The logic die LD and the memory die MD may be connected via a wiring substrate such as a silicon interposer. By sandwiching the silicon interposer, for example, it is possible to easily align the data terminals of the logic die LD and the data terminals of the memory die MD. Furthermore, multiple logic dies LD and multiple memory dies MD may be stacked and connected to each other.

[0024] Fig. 5 is a plan view showing an example of the configuration of the logic die LD in Fig. 1. The logic die LD may include a plurality of blocks L2B, a data engine, an inter-die interconnect, a PCIe-IF (Peripheral Component Interconnect express interface), and a memory PDM.

[0025] Each block L2B may have a plurality of blocks L1B and a memory L2BM. Each block L1B may have a plurality of matrix calculation blocks MAB and a memory L1BM. As shown in FIG. 3, each matrix calculation block MAB may have a plurality of processing elements PE and a matrix calculator MAU. The storage capacity of the memory L2BM may be larger than that of the memory L1BM, and the storage capacity of the memory block MB may be larger than that of the memory L1BM.

[0026] The processing elements PE may be connected to the memory blocks MB, respectively, and at least some of the data stored in the memory blocks MB may be used in the matrix calculator MAU without going through the memory L1BM. Note that the matrix calculator MAU may be configured as a systolic array.

[0027] As shown in FIG. 5, the logic die LD may have hierarchical blocks L1B and L2B, each including a predetermined number of matrix calculation blocks MAB, and the memories L1BM and L2BM may have a hierarchical structure. The matrix calculation block MAB is an example of a first-level block, the block L1B is an example of a second-level block, and the block L2B is an example of a third-level block. The memory L1BM is an example of a first memory, the memory block MB is an example of a second memory, and the memory L2BM is an example of a third memory. The memory PDM is an example of a fourth memory. All of the matrix calculation blocks MAB mounted on the logic die LD may operate according to the same instructions.

[0028] The logic die LD may have four blocks L2B, and each block L2B may have eight blocks L1B and one memory L2BM. Each block L1B may have 16 matrix calculation blocks MAB and one memory L1BM, and each matrix calculation block MAB may have four processing elements PE and one matrix calculator MAU. Note that the matrix calculation block MAB may not have a matrix calculator MAU.

[0029] The number of elements included in the logic die LD is not limited to that shown in Fig. 5. For example, the logic die LD may have two blocks L2B, and each block L2B may have four blocks L1B and one memory L2BM. Each block L1B may have 16 matrix operation blocks MAB and one memory L1BM, and each matrix operation block MAB may have eight processing elements PE and one matrix operation unit MAU.

[0030] A large number of matrix calculators MAU are distributed on the logic die LD. By connecting a memory block MB with a large capacity and a wide bandwidth to each processing element PE, data can be supplied to each of the distributed matrix calculators MAU without waiting for calculations. In other words, it is possible to prevent a time when the matrix calculator MAU cannot perform calculations due to insufficient memory bandwidth, resulting in a decrease in calculation speed. Note that one or both of the matrix calculator MAU and the processing element PE are examples of calculators. Furthermore, a calculator may be any of an arithmetic circuit, a processing circuit, and processing circuitry.

[0031] It is preferable that the number of blocks L2B mounted on the logic die LD, the number of blocks L1B mounted on each block L2B, the number of matrix calculation blocks MAB mounted on each block L1B, and the number of processing elements PE mounted on each matrix calculation block MAB are each 2 n (n is an integer greater than or equal to 1).

[0032] The data engine may control the transfer of data stored in the memory PDM from outside the logic die LD to the memory L2BM or the memory L1B, and may also control the transfer of data between the memory L2BM, the memory L1B, and the matrix calculation block MAB.

[0033] The inter-die interconnect may be used to connect the computing device 100 to another computing device 100. The PCIe-IF may transmit and receive data, instructions, etc. to and from a host or device connected externally to the computing device 100. The memory PDM may function as a buffer memory that holds data, etc. transmitted and received via the PCIe-IF. For example, the PCIe-IF may be configured with an FPGA (Field Programmable Gate Array).

[0034] As shown in FIGS. 1 and 2, each processing element PE may be connected to a memory block MB of a memory die MD. Each processing element PE may not directly access memory blocks MB other than the one directly connected to it, and may be programmed to operate without using the results of operations by other processing elements PE. Examples of processing elements PE are shown in FIGS. 8 and 9. For example, the matrix operation unit MAU may execute SIMD (Single Instruction Multiple Data) instructions. Note that the arithmetic device 100 may have a SIMD model architecture in which all matrix operation blocks MAB included in the arithmetic device 100 are operated by a single instruction.

[0035] FIG. 6 is a block diagram showing an example of a system including a computer 200 on which multiple arithmetic units 100 are installed, and a host 300. The computer 200 shown in FIG. 6 has multiple arithmetic units 100 connected to each other. For example, the computer 200 may have the form of an electronic circuit board on which multiple arithmetic units 100 are installed. In FIG. 6, the computer 200 has two arithmetic units 100, but the number of arithmetic units 100 may be one or three or more. For example, the computer 200 may perform inference such as large language models (LLM). The host 300 may have a CPU and a host memory HOSTM.

[0036] Each computing device 100 may be connected to the host 300 via a PCIe-IF and may receive information such as data and instructions transferred from the host memory HOSTM by a control program executed by the CPU of the host 300. Transfer of information between the host memory HOSTM and the computing device 100 may be performed by DMA (Direct Memory Access).

[0037] 5 based on an operation instruction received from the host 300, or may execute a data transfer process based on a data transfer instruction received from the host 300. For example, which operation to execute in which MAB may be instructed to each operation device 100 by the host 300 via software.

[0038] Execution of the data transfer instruction may result in data transfer to the memory PDM, L2BM, L1BM, or memory die MD. After the data to be operated on is transferred to the memory die MD, each arithmetic device 100 may execute an operation using the processing element PE or the matrix calculator MAU based on the operation instruction received from the host 300.

[0039] For example, a program including various instruction codes transferred from the host 300 to the arithmetic device 100 may be generated by a compiler device (not shown) and stored in the host memory HOSTM. Note that in Fig. 6, the lines connecting the elements of the arithmetic device 100 indicate data transfer paths, and the arithmetic instructions and various control information may be transferred via paths (not shown) separate from the data transfer paths.

[0040] In Figure 6, as shown in Figures 1 and 2, each processing element PE is connected to a memory block MB, but multiple processing elements PE may be connected to one memory block MB, or each processing element PE may be connected to multiple memory blocks MB.

[0041] 7 is a diagram illustrating an example of a connection between processing elements PE and memory blocks MB via a memory interface MI / F. In connection example 1, two processing elements PE1 of a matrix operation block MAB may be connected to memory block MB1 via a memory interface MI / F1, and two processing elements PE2 of the matrix operation block MAB may be connected to memory block MB2 via a memory interface MI / F2. That is, the memory interface MI / F1 may connect at least one processing element PE1 to one or more memory blocks MB1, and the memory interface MI / F2 may connect at least one processing element PE2 to one or more memory blocks MB2. Furthermore, the memory interface MI / F1 may connect at least one processing element PE1 to a predetermined memory block MB1 of one or more memory blocks MB1, and the memory interface MI / F2 may connect at least one processing element PE2 to a predetermined memory block MB2 of one or more memory blocks MB2. The memory interface MI / F1 is an example of a first memory interface, the processing element PE1 connected to the memory interface MI / F1 is an example of a first computing unit, the memory interface MI / F2 is an example of a second memory interface, and the processing element PE2 connected to the memory interface MI / F2 is an example of a second computing unit.

[0042] Alternatively, two processing elements PE1 may be independently connected to the memory interface MI / F1, and two processing elements PE2 may be independently connected to the memory interface MI / F1. In this case, the memory interface MI / F1 or MI / F2 has an arbitration circuit that arbitrates memory access commands from the two processing elements PE1 or PE2.

[0043] In connection example 2, two processing elements PE1 may be commonly connected to memory interface MI / F1, and two processing elements PE2 may be commonly connected to memory interface MI / F2. Note that in connection example 1 or connection example 2, the number of processing elements PE1 or PE2 connected to memory interface MI / F1 or MI / F2 may be three or more.

[0044] In connection example 3, one of the two processing elements PE1 may have a memory interface MI / F1, and one of the two processing elements PE2 may have a memory interface MI / F2. In this case, the other processing element PE1 may be connected to one memory interface MI / F1 of the processing element PE1 and connected to memory block MB1 via the memory interface MI / F1. Similarly, the other processing element PE2 may be connected to one memory interface MI / F2 of the processing element PE2 and connected to memory block MB2 via the memory interface MI / F2.

[0045] In connection example 3, the matrix operation block MAB may have three or more processing elements PE1 and three or more processing elements PE2. In this case, one of the three or more processing elements PE1 may have a memory interface MI / F1, and one of the three or more processing elements PE2 may have a memory interface MI / F2. Three or more memory blocks MB may be connected to one matrix operation block MAB.

[0046] In connection example 4, a memory interface MI / F1 or MI / F2 may be provided for each of the matrix operation blocks MAB1 and MAB2. In this case, a plurality of processing elements PE1 (for example, four) of the matrix operation block MAB1 may be connected to a memory block MB1 via the memory interface MI / F1, and a plurality of processing elements PE2 (for example, four) of the matrix operation block MAB2 may be connected to a memory block MB2 via the memory interface MI / F2. Note that one memory block MB may be connected to a plurality of matrix operation blocks MAB.

[0047] 7, the memory interfaces MI / F1 and MI / F2 may be used exclusively for accessing the memory blocks MB1 and MB2 provided on the memory die MD, respectively. Note that the memory interfaces MI / F1 and MI / F2 are not used for accessing the memory L1BM and the memory L2BM (not shown).

[0048] 7 shows an example in which one memory interface MI / F1 or MI / F2 is provided corresponding to a plurality of processing elements PE1 or PE2, but a memory interface MI / F may be provided for each processing element PE. In this case, the memory interface MI / F may be built into each processing element PE.

[0049] 8 is a circuit block diagram showing an example of the processing element PE shown in FIG. 5. The processing element PE has an integer arithmetic logic unit (IALU), registers M-REG, GRF0, GRF1, T-REG, local memories LM0, LM1, and multiplexers MUX1, MUX2. For example, the local memories LM0, LM1 may be configured with SRAM. The local memories LM0, LM1 are an example of a fifth memory.

[0050] The register M-REG may mask at least a portion of the calculation result data by the integer arithmetic unit IALU and prevent the masked calculation result data from being written to the local memory LM1. The registers GRF0 and GRF1 may be general-purpose registers that hold data used in calculations or calculation result data. The registers GRF0 and GRF1 may also be two-port registers that can write data and read data in parallel. The register T-REG may be used to temporarily hold data.

[0051] The local memory LM0 may be a buffer memory that holds data to be read from or written to the memory block MB. In this case, the local memory LM0 may be connected to the corresponding memory block MB via the memory interface MI / F1 or MI / F2 shown in Figure 7. Note that the memory block MB is connected only to the local memory LM0 and not to the multiplexers MUX1 and MUX2, so data input to and output from the memory block MB is transferred only via the local memory LM0.

[0052] For example, the storage capacity of the local memory LM0 may be 16 MB, and the local memory LM0 may read or write data from or to the memory block MB at 8 bytes per cycle. The data input / output port of the memory block MB may be 8 bytes. The local memory LM0 may hold data that is not read from or written to the memory block MB.

[0053] The local memory LM1 may, for example, hold data used by the integer arithmetic unit IALU and may hold data resulting from calculations by the integer arithmetic unit IALU. For example, the storage capacity of the local memory LM1 may be 16 kbytes, and data may be read from or written to the local memory LM1 at 8 bytes per cycle. Note that the local memories LM0 and LM1 may be single-port memories capable of either reading or writing data, or may be dual-port memories capable of reading and writing data simultaneously or almost simultaneously.

[0054] The multiplexer MUX1 may transfer any of the operation result data by the integer arithmetic unit IALU, the operation result data by the matrix arithmetic unit MAU, or the data from the memory L1BM to any of the registers GRF0, GRF1, T-REG, or the local memories LM0, LM1 in accordance with an instruction. The multiplexer MUX2 may transfer any of the operation result data by the integer arithmetic unit IALU, the operation result data by the matrix arithmetic unit MAU, the registers GRF0, GRF1, T-REG, the local memories LM0, LM1, or the matrix arithmetic unit MAU to any of the integer arithmetic unit IALU, the matrix arithmetic unit MAU, or the memory L1BM in accordance with an instruction.

[0055] The processing element PE may supply data read from the memory block MB via the local memory LM0 to the integer arithmetic unit IALU or matrix arithmetic unit MAU to execute the operation. The processing element PE may write the operation result data output from the integer arithmetic unit IALU or matrix arithmetic unit MAU back to the memory block MB via the local memory LM0. The basic operation of the logic die LD is to perform an operation on the data read from the memory block MB by the integer arithmetic unit IALU or matrix arithmetic unit MAU, and to write the operation result data back to the memory block MB.

[0056] Fig. 9 is a circuit block diagram showing another example of the processing element PE shown in Fig. 5. The processing element PE shown in Fig. 9 may not have a local memory LM0, and one of the outputs of the multiplexer MUX1 and one of the inputs of the multiplexer MUX2 may be connected to the memory block MB via a memory interface MI / F (not shown).

[0057] 10 is a block diagram showing an example of the connection between the logic die LD and the memory die MD of FIG. 1. In addition to the plurality of matrix operation blocks MAB and memory L1BM shown in FIG. 2, the block L1B may have a memory controller MCNT that generates a control signal CNTL that controls reading and writing of data from and to the memory block MB. The memory controller MCNT may include some of the functions of the memory interfaces MI / F1 and MI / F2 shown in FIG. 7. For example, the control signal CNTL may include an address signal AD, a command signal CMD that indicates a read access request or a write access request, a clock signal CLK, etc.

[0058] Furthermore, the memory controller MCNT may output a data synchronization signal DSYNC to each local memory LM0 in the block L1B. During a read access to the memory block MB, the memory controller MCNT may output a data synchronization signal DSYNC in response to the control signal CNTL, indicating the timing at which data read from the memory block MB to the data bus DB is written to the local memory LM0. During a write access to the memory block MB, the memory controller MCNT may output a data synchronization signal DSYNC in response to the control signal CNTL, indicating the timing at which data to be written to the memory block MB is output from the local memory LM0 to the data bus DB.

[0059] For example, the memory controller MCNT may generate a control signal CNTL and a data synchronization signal DSYNC common to all local memories LM0 mounted in the block L1B. In this case, all memory blocks MB connected to the block L1B operate in synchronization with one another. Alternatively, the memory controller MCNT may generate a common control signal CNTL and a common data synchronization signal DSYNC for each of a predetermined number of local memories LM0 mounted in the block L1B.

[0060] 11 is a diagram showing an example of the logical configuration of the memories L2BM, L1BM, LM0, and LM1 of the arithmetic device 100 and the connection relationships between the memories. The local memories LM0 and LM1 may be connected to the corresponding memory L1BM. The memory L1BM may be connected to the corresponding memory L2BM. The memory L1BM and the memory L2BM may not be directly connected to the memory block MB, but may be used for data transfer between the processing elements PE or data transfer with the outside of the logic die LD. Alternatively, the memory L1BM and the memory L2BM may be directly connected to the memory block MB or an external memory different from the memory block MB and used for data transfer. The processing element PE may also be configured without having the local memory LM1.

[0061] Each memory block MB is disposed adjacent to and opposite the processing element PE, and may be connected to the local memory LM0 by, for example, hybrid bonding the terminals of the logic die LD and the memory die MD. This reduces the distance that data must travel when reading or writing data from or to the memory block MB, and can suppress increases in wiring length and power consumption even when the bandwidth of the memory block MB increases.

[0062] In order to shorten the wiring length, a processing element PE connected to a memory block MB in a one-to-one relationship is preferably connected to a memory block MB located directly above or below the processing element PE. However, as long as each processing element PE is located closer to the memory block MB in a planar view than an adjacent processing element PE, it does not have to be located in a position shifted from directly above or below the memory block MB. In other words, each processing element PE only needs to be connected to the memory block MB closest to it in a planar view. This can suppress degradation of the bandwidth of the memory block MB.

[0063] 12 is a diagram showing an example of the arrangement of the memories L2BM, L1BM, LM0, and LM1 in the arithmetic device 100. As shown in FIGS. 1 and 2, the memory die MD including the memory block MB is arranged above or below the logic die LD, so that the connections between the memory block MB and the local memory LM0 can be distributed within the chip without being concentrated in a portion of the memory die MD and the logic die LD. As a result, the memory die MD and the logic die LD can be connected using a large number of data buses DB.

[0064] For example, the memory die MD and the logic die LD may be connected so that data can be read or written at 8 bytes per cycle for each processing element PE. This allows the memory block MB, which has a larger memory capacity, to be used without degrading the bandwidth, compared to when, for example, the local memory LM0, which has a smaller memory capacity, is used to store data during calculation.

[0065] Note that the memory block MB may be connected not only to each processing element PE but also to each matrix operation block MAB (every four processing elements PE). Furthermore, the memory block MB may be connected to two or more matrix operation blocks MAB. As shown in FIG. 7, two memory blocks MB1 and MB2 may be connected to one matrix operation block MAB, and each memory block MB1 and MB2 may be shared by two processing elements PE1 or two processing elements PE2. Furthermore, the memory block MB may be connected to four matrix operation blocks MAB (every 16 processing elements PE). The number of processing elements PE connected to each memory block MB may be determined taking into consideration constraints such as the interface specifications of the memory block MB.

[0066] 13 is a diagram showing another example of the logical configuration of the memories L2BM, L1BM, LM0, and LM1 of the arithmetic device 100 and the connection relationships between the memories. A memory block MB may be connected to every four processing elements PE. The configuration shown in FIG. 13 is the same as FIG. 11 except that the number of processing elements PE connected to a memory block MB is different.

[0067] When four processing elements PE are assigned to one memory block MB, the bandwidth of the port of the memory block MB may be set to four times the bandwidth of the port of the memory block MB in Figure 11. This allows the bandwidth per processing element PE to be the same as in Figure 11.

[0068] In Figure 13, the four processing elements PE are constrained to access different portions of data held in a storage area at the same address in one memory block MB. If each processing element PE reads or writes 16 bytes per cycle in the configuration of Figure 11, then in Figure 13, each of the four processing elements PE may read or write 64 bytes per cycle. In this case, each processing element PE may read or write four 16-byte blocks, which is equal division of 64 bytes. Note that the number of processing elements PE assigned to one memory block MB may be other than those shown in Figures 11 and 13.

[0069] In addition, when multiple processing elements PE are assigned to one memory block MB, a memory die MD may be arranged in place of each memory block MB. Furthermore, a memory die MD may be arranged in place of each memory block MB shown in FIG.

[0070] 14 is a diagram illustrating an example of a logical configuration of a processor having a hierarchical cache and a main memory device. The processor 101 shown in FIG. 14 may have multiple processing elements PE, multiple L1 data caches L1D, multiple L2 data caches L2D, and multiple L3 data caches L3D.

[0071] The multiple L1 data caches L1D may be connected to multiple processing elements PE, respectively. Each of the multiple L2 data caches L2D may be connected to a predetermined number of L1 data caches L1D and shared by a predetermined number of processing elements PE. The L3 data cache L3D may be connected to multiple L2 data caches L2D and shared by all processing elements PE. The L3 data cache L3D may be connected to a main memory device 102 such as a DRAM.

[0072] 14 makes it possible to substantially increase the bandwidth of the main memory device 102. For example, when a processing element PE reads data Da from the main memory device 102, the data Da can remain in the L3 data cache L3D for a while. When another processing element PE attempts to read data Da from the main memory device 102, if the data Da remains in the L3 data cache L3D, the other processing element PE can obtain the data Da by accessing the L3 data cache L3D without accessing the main memory device 102.

[0073] In this way, by hierarchizing the cache, it is possible to save bandwidth in the main memory device 102. The saved bandwidth in the main memory device 102 can be used to access data that is not in the cache, so it can be considered that the bandwidth of the main memory device 102 has effectively increased.

[0074] FIG. 15 is a diagram showing an example of the physical configuration of the arithmetic unit 101 and the main memory unit 102 shown in FIG. 14. Cache hierarchization is not necessarily optimal. For example, if multiple processing elements PE continue to read different data from each other, there is no point in sharing a cache between the processing elements PE. In fact, sharing a cache makes data movement inefficient.

[0075] 15, the processing elements PE and the memories are interconnected as follows: PE-L1D-L2D-L3D-main memory device 102. Therefore, when the processing element PE reads data from the main memory device 102, the arithmetic unit 101 sends the data read from the main memory device 102 to the L3 data cache L3D, from the L3 data cache L3D to the L2 cache, from the L2 data cache L2D to the L1 cache, and from the L1 data cache L1D to the processing element PE. When data is written to the main memory device 102, the data is transferred in the reverse order of the read order.

[0076] In the configuration shown in FIG. 15, the average distance data travels is approximately the chip size of the arithmetic device 101, specifically, in centimeters. Increasing the bandwidth of the main memory device 102 requires increasing the bandwidth between the processing element PE and the L1 data cache L1D, the bandwidth between the L1 data cache L1D and the L2 data cache L2D, and the bandwidth between the L2 data cache L2D and the L3 data cache L3D. However, increasing the bandwidth of the main memory device 102 is difficult due to the amount of wiring and power consumption, and also results in high costs. In contrast, in the configurations shown in FIGS. 11 to 13, the logic die LD and the memory die MD are stacked, allowing the distance between the dies to be equal to or less than the die thickness. Therefore, compared to the configuration shown in FIG. 15, the load capacitance of the data bus connecting the logic die LD and the memory die MD can be significantly reduced, significantly shortening the propagation delay time of data signals transmitted on the data bus. As a result, the arithmetic processing performance of the arithmetic device 100 can be improved.

[0077] 16 is a diagram showing an example of redundantly arranging matrix operation blocks MAB in the logic die LD of the arithmetic device 100. In FIG. 16, each memory block MB of the arithmetic device 100 may be assigned to each matrix operation block MAB (each of four processing elements PE). Here, when a large number of processing elements PE or a large number of matrix operation blocks MAB are assigned to a large number of memory blocks MB, it is preferable to consider redundancy. Generally, since the yield during the manufacture of semiconductor chips is not high, adding a spare circuit and switching from a faulty circuit to the spare circuit can repair defects and improve the yield.

[0078] The calculation device 100 may have one matrix calculation block MAB16 as a spare for the 16 matrix calculation blocks MAB0-MAB15. That is, 16 memory blocks MB may be arranged corresponding to the 17 matrix calculation blocks MAB0-MAB16. The spare matrix calculation block MAB16 is an example of a redundant calculation block.

[0079] When the matrix calculation blocks MAB0-MAB15 are not faulty and the logic die LD is operating normally, the 16 ports LP0-LP15 of the matrix calculation blocks MAB0-MAB15 may be connected to the 16 ports MP0-MP15 of the memory blocks MB0-MB15, respectively, via a switching circuit SW provided in the logic die LD.

[0080] For example, if a failure in the matrix calculation block MAB7 indicated by an x ​​is detected during a test process of the logic die LD, the connection of the switching circuit SW may be switched to disconnect the matrix calculation block MAB7 from the memory block MB7. Then, the ports LP0-LP6 and LP8-LP16 of the matrix calculation blocks MAB0-MAB6 and MAB8-MAB16 may be connected to the ports MP0-MP15 of the memory blocks MB0-MB15, respectively, via the switching circuit SW.

[0081] That is, the switching circuit SW may switchably connect the matrix operation blocks MAB0-MAB16 to the memory blocks MB0-MB15. This allows the arithmetic device 100 to operate normally even if one of the 17 matrix operation blocks MAB0-MAB16 fails. This allows the yield of the arithmetic device 100 to be improved.

[0082] FIG. 17 is a diagram showing an example of redundantly arranging memory blocks MB on a memory die MD of the arithmetic device 100. In FIG. 17, as in FIG. 16, each memory block MB of the arithmetic device 100 may be assigned to each matrix operation block MAB (each of four processing elements PE). The arithmetic device 100 may have one memory block MB16 as a spare for each of 16 memory blocks MB0-MB15. That is, 17 memory blocks MB may be arranged corresponding to the 16 matrix operation blocks MAB0-MAB15. The spare memory block MB16 is an example of a redundant memory, and an example of a redundant memory block including a redundant memory.

[0083] When the memory blocks MB0-MB15 are not faulty and the memory die MD operates normally, the 16 ports MP0-MP15 of the memory blocks MB0-MB15 may be connected to the 16 ports LP0-LP15 of the matrix operation blocks MAB0-MAB15, respectively, via a switching circuit SW provided on the memory die MD.

[0084] For example, if a failure in memory block MB4, indicated by an X, is detected during a test process of the memory die MD, the connection of the switching circuit SW may be switched to disconnect memory block MB4 from matrix operation block MAB4. Then, ports MP0-MP3 and MP5-MP16 of memory blocks MB0-MB3 and MB5-MB16 may be connected to ports LP0-LP15 of MAB0-MAB15, respectively, via the switching circuit SW.

[0085] That is, the switching circuit SW may switchably connect the memory blocks MB0-MB16 to the matrix operation blocks MAB0-MAB15. This allows the arithmetic device 100 to operate normally even if one of the 17 memory blocks MB0-MB16 fails. This allows the yield of the arithmetic device 100 to be improved.

[0086] 18 is a diagram showing an example of redundantly arranging matrix operation blocks MAB and memory blocks MB in the arithmetic device 100. In FIG. 18, similarly to FIGS. 16 and 17, each memory block MB of the arithmetic device 100 may be allocated to each matrix operation block MAB (each of four processing elements PE). The arithmetic device 100 may have 16 matrix operation blocks MAB0-MB15 and one spare matrix operation block MAB16, as well as 16 memory blocks MB0-MB15 and one spare memory block MB16.

[0087] When the matrix operation blocks MAB0-MAB15 and the memory blocks MB0-MB15 are not faulty and the logic die LD and the memory die MD are operating normally, for example, the 16 ports LP0-LP15 of the matrix operation blocks MAB0-MAB15 may be connected to the 16 ports MP0-MP15 of the memory blocks MB0-MB15, respectively, via a switching circuit SW provided in the logic die LD. Note that the switching circuit SW may be provided in each of the logic die LD and the memory die MD.

[0088] For example, if a failure in matrix operation block MAB7 indicated by an X is detected in a test process of the logic die LD, and a failure in memory block MB4 indicated by an X is detected in a test process of the memory die MD, the connection of the switching circuit SW may be switched. Then, in the arithmetic device 100, ports LP0-LP6 and LP8-LP16 of matrix operation blocks MAB0-MAB6 and MAB8-MAB16 may be connected to ports MP0-MP3 and MP5-MP16 of memory blocks MB0-MB3 and MB5-MB16, respectively.

[0089] As a result, even if one of the 17 matrix operation blocks MAB0-MAB16 and one of the 17 memory blocks MB0-MB16 fail, the arithmetic device 100 can operate normally, thereby improving the yield of the arithmetic device 100.

[0090] As described above, in the first embodiment, multiple processing elements PE of a logic die LD are connected to multiple memory dies MD or multiple memory blocks MB arranged opposite the logic die LD, respectively, for three-dimensional integration, thereby achieving a memory bandwidth larger than conventional technologies with low power consumption. Because high-bandwidth memory can be used with low power consumption and low cost, applications with relatively high memory access, such as deep neural network training, inference, or scientific and technological calculations, can be executed efficiently with low power consumption.

[0091] For example, the computing device 100 can achieve a bandwidth between the logic die LD and the memory die MD of approximately several tens of TG / s, which is significantly larger than the bandwidth of HBM2 (256 GB / s) and HBM3 (819 GB / s).

[0092] The second embodiment of the present disclosure is an example of a calculation method for executing an inference process of a machine learning model using the calculation device 100 according to the first embodiment. Note that the calculation method according to this embodiment is not limited to the inference process of a machine learning model, and may also be used for training a machine learning model.

[0093] The machine learning model in this embodiment may include a neural network. The machine learning model may include a neural network with attention (attention mechanism). The neural network with attention may be, for example, a neural network also called a Transformer. The machine learning model may include a space state model (SSM).

[0094] The machine learning model may be, for example, a large language model (LLM). The machine learning model may be, for example, a generative model, a foundation model, or a neural network that generates various types of data such as audio, images, and videos. The machine learning model may be multimodal.

[0095] The configuration of the machine learning model in this embodiment will be described with reference to Fig. 19. Fig. 19 is a block diagram showing an example of the machine learning model in the second embodiment of the present disclosure.

[0096] As shown in Figure 19, the machine learning model M may include one or more Transformers T (T-1, T-2, T-3, ...). The number of Transformers T included in the machine learning model M may be any number, and may be approximately 80, for example. Note that the machine learning model M may also include layers, blocks, networks, etc. other than Transformers.

[0097] Hereafter, when there are multiple Transformers T, to distinguish between them, we will use sub-numbers such as "Transformer T-1", "Transformer T-2", etc. When simply saying "Transformer T", it applies to all Transformers T.

[0098] The Transformer T may include a multi-layer perceptron (MLP) and an attention ATT. The multi-layer perceptron MLP may be a feedforward network, for example. The feedforward network may have any number of layers. The attention ATT may be a multi-head attention network, for example. The number of heads in the multi-head attention network may be any number, but may be around 64, for example.

[0099] In this embodiment, the number of parameters of the machine learning model M may be approximately 10B to 70B (10 billion to 70 billion). In this case, the amount of data when quantized in 8-bit floating-point format (FP8) is approximately 10GB to 70GB. As an example, generating 1000 tokens using the machine learning model M will require approximately 10TB to 70TB of memory access. Since the bandwidth of a conventional GPU is approximately 4TB per second, generating 1000 tokens using a conventional GPU would take approximately 2.5 seconds to 17.5 seconds.

[0100] Furthermore, in this embodiment, the arithmetic device 100 may have two blocks L2B. Each of the blocks L2B may have four blocks L1B. Each of the blocks L1B may have 16 matrix operation blocks MAB. Each of the matrix operation blocks MAB may have eight processing elements PE and one matrix operator MAU.

[0101] The matrix operation block MAB may be configured as follows.

[0102] The local memory LM0 of the processing element PE is connected to the memory block MB. The processing element PE can read or write 16 bytes of data from or to the local memory LM0 per cycle. The capacity of the local memory LM0 is 16 MB per processing element PE.

[0103] The local memory LM1 of the processing element PE is configured with SRAM. The processing element PE can read or write 16 bytes of data from or to the local memory LM1 per cycle. The capacity of the local memory LM1 is 16 KB per processing element PE.

[0104] The registers GRF0 and GRF1 of the processing element PE are configured with SRAM. The registers GRF0 and GRF1 can each read and write 16 bytes of data in parallel per cycle. The capacity of the registers GRF0 and GRF1 is 2 KB each per processing element PE.

[0105] The register T-REG of the processing element PE is composed of flip-flops. The register T-REG can read and write 16 bytes of data in parallel per cycle. The capacity of the register T-REG is 64B per processing element PE.

[0106] The integer arithmetic unit IALU is an arithmetic unit that performs integer addition, etc. The input to the integer arithmetic unit IALU is one or two 16-byte data read from the local memory LMO, etc. per cycle. The output of the integer arithmetic unit IALU is 16 bytes.

[0107] The matrix calculator MAU is a calculator that executes floating-point operations. The matrix calculator MAU performs matrix multiplication or element-by-element multiplication or summation of vectors. For matrix multiplication, the matrix calculator calculates the product of a 32x64 matrix and a 64x2 matrix per cycle.

[0108] The inference processing of the machine learning model in this embodiment will be described with reference to Fig. 20. Fig. 20 is a flowchart showing an example of the inference processing of the machine learning model M shown in Fig. 19. Note that Fig. 20 shows the processing of one Transformer T included in the machine learning model M, but the processing of two or more Transformers T included in the machine learning model M may be executed in parallel. When two or more Transformers T execute processing in parallel, the respective Transformers T may execute processing synchronously or asynchronously.

[0109] In step S1, the arithmetic device 100 stores parameters of the machine learning model M in a memory block MB. The arithmetic device 100 may store parameters of the machine learning model M input from outside in the memory block MB. Note that the process of storing information in the memory block MB may include the arithmetic device 100 executing at least a part of the process required for storage.

[0110] As an example, the arithmetic device 100 may store parameters of the machine learning model M transferred from the host memory HOSTM in response to an instruction issued by the CPU of the host 300 in the memory block MB via the memory L2BM, the memory L1BM, and the local memory LM0. Note that the instruction issued by the CPU of the host 300 may be a SIMD instruction. The SIMD instruction is an instruction set that processes multiple pieces of data in parallel with a single instruction. The multiple matrix operation blocks MAB included in the arithmetic device 100 may operate in parallel according to a single SIMD instruction received from the CPU of the host 300.

[0111] The arithmetic device 100 may distribute and store parameters of the machine learning model M across multiple memory blocks MB. The arithmetic device 100 may assign the parameters of the machine learning model M to multiple matrix operation blocks MAB, and store some of the parameters of the machine learning model M in memory blocks MB connected to each of the multiple matrix operation blocks MAB. The assignment of parameters to the matrix operation blocks MAB may be performed by the arithmetic device 100, or may be performed by an external designation (for example, an instruction issued by the CPU of the host 300).

[0112] In step S2, the arithmetic device 100 acquires input information to be input to the machine learning model M. The arithmetic device 100 may acquire input information input from outside. For example, the arithmetic device 100 may receive input information transferred from the host memory HOSTM in response to an instruction issued by the CPU of the host 300.

[0113] The input information may include text data, image data, or audio data. The text data may be, for example, a natural language sentence called a prompt. The image data may be, for example, a still image or a video. The text data may be text data obtained by speech recognition of audio data or video. The text data may be text obtained by character recognition of image data. The image data may include, for example, an image of the user. The audio data may include, for example, the voice spoken by the user. The audio data may be audio data obtained by speech synthesis of text data.

[0114] The input information may be divided into tokens. A token is a processing unit when a machine learning model processes electronic data, and the amount of data per token may vary depending on the design of the machine learning model. A token is, for example, one Japanese character or one English word, but depending on the frequency of occurrence, one character may be represented by two tokens, or two or more characters may be represented by one token.

[0115] The arithmetic device 100 may input the acquired input information to the matrix operation block MAB. The arithmetic device 100 may input a predetermined number of tokens included in the input information to the matrix operation block MAB. The number of tokens input to the matrix operation block MAB may be determined in advance by the machine learning model M or the design of the arithmetic device 100. In this embodiment, the number of tokens may be 32, for example.

[0116] The input information input to the matrix calculation block MAB may be stored in the local memory LM1 of the matrix calculation block MAB, or may be stored in a memory block MB connected to the matrix calculation block MAB. The calculation device 100 may distribute and input the input information to multiple matrix calculation blocks MAB. The calculation device 100 may assign input information for the machine learning model M to multiple matrix calculation blocks MAB, and input a portion of the input information to each of the multiple matrix calculation blocks MAB. The assignment of input information to the matrix calculation blocks MAB may be performed by the calculation device 100, or may be performed by an external designation (for example, an instruction issued by the CPU of the host 300, etc.).

[0117] In step S3, the arithmetic device 100 executes a pre-filling process. The pre-filling process is a process of generating intermediate data to be used in the inference process of the machine learning model M based on the input information acquired in step S2. During the pre-filling process, the arithmetic device 100 repeatedly executes calculations of the multilayer perceptron MLP and the attention ATT. Before executing the calculations of the multilayer perceptron MLP or the attention ATT, the arithmetic device 100 may read the parameters of the machine learning model M stored in step S1 from the memory block MB.

[0118] The intermediate data may vary depending on the type of machine learning model. As an example, if the machine learning model includes a Transformer, the intermediate data may include a key-value cache. The key-value cache is an array of cached key vectors and value vectors. A key-value cache is generated for each layer that constitutes a Transformer. As another example, if the machine learning model includes a state-space model, the intermediate data may include states in the state-space model. A state in the state-space model may be represented by one or more vectors or tensors.

[0119] In step S3-1, the matrix operation block MAB of the arithmetic device 100 reads parameters of the machine learning model M from the memory block MB. The matrix operation block MAB may read at least some of the parameters of the machine learning model M stored in the memory block MB. The matrix operation block MAB may read only parameters corresponding to the input information input in step S2. The matrix operation block MAB may read only parameters to be used in the subsequent step S3-2 or S3-3.

[0120] During execution of step S3-2 or S3-3, the matrix operation block MAB may read from the memory block MB at least a portion of the parameters of the machine learning model M. As an example, during execution of step S3-2 or S3-3 for a portion of the input information, the matrix operation block MAB may read parameters to be used in step S3-2 or S3-3 for another subsequent portion of the input information.

[0121] In step S3-2, the matrix operation block MAB of the arithmetic device 100 generates intermediate data. The matrix operation block MAB may generate the intermediate data by calculating the attention ATT of the machine learning model M. The matrix operation block MAB may generate the intermediate data corresponding to the input information input in step S2. The matrix operation block MAB may generate the intermediate data using the parameters read in step S3-1.

[0122] The matrix operation block MAB may store the generated intermediate data in the memory block MB. The matrix operation block MAB may store at least a part of the intermediate data in the memory block MB. The matrix operation block MAB may store the intermediate data in the memory block MB connected to the matrix operation block MAB.

[0123] In step S3-3, the matrix operation block MAB of the arithmetic device 100 generates output information to be output from the machine learning model M. The matrix operation block MAB may generate the output information by calculating a multilayer perceptron MLP of the machine learning model M. The matrix operation block MAB may generate the output information using the parameters read in step S3-1. The matrix operation block MAB may generate the output information using the intermediate data generated in step S3-2.

[0124] The output information may include at least one of text data, audio data, and image data. The image data may include still images and video, and the video may include audio data. The output information may be multimodal data including at least two of text data, audio data, and image data. The output information may be divided into tokens. The output information may be one token output for the last token included in the input information.

[0125] The matrix calculation block MAB may store the generated output information. The matrix calculation block MAB may store the output information in a local memory LM1 of the matrix calculation block MAB, or may store the output information in a memory block MB connected to the matrix calculation block MAB. The matrix calculation block MAB may store in the local memory LM1 or the memory block MB only the output information generated when processing of the input information input in step S2 is completed.

[0126] The matrix calculation block MAB may repeatedly execute the processes from steps S3-1 to S3-3, or may execute the processes from steps S3-1 to S3-3 only once and then execute step S4. The matrix calculation block MAB may repeatedly execute the processes from steps S3-1 to S3-3 until processing of the input information entered in step S2 is completed. The matrix calculation block MAB may read all parameters stored in the memory block MB in step S3-1 and repeatedly execute only the processes of steps S3-2 and S3-3. As an example, when large-sized input information is entered in step S2, the input information may be divided into multiple pieces and the processes from steps S3-1 to S3-3 may be repeatedly executed for each piece of input information.

[0127] In step S4, the arithmetic device 100 executes a decoding process. The decoding process is a process of generating output information from the machine learning model M using the intermediate data generated in the pre-filling process. During the decoding process, the arithmetic device 100 repeatedly executes calculations of the multilayer perceptron MLP and the attention ATT. Before executing calculations of the multilayer perceptron MLP and the attention ATT, the arithmetic device 100 may read the parameters of the machine learning model M stored in step S1 from the memory block MB.

[0128] In step S4-1, the matrix operation block MAB of the arithmetic device 100 reads parameters of the machine learning model M from the memory block MB. The matrix operation block MAB may read at least a portion of the parameters of the machine learning model M stored in the memory block MB. The matrix operation block MAB may read only parameters corresponding to the input information input in step S2. The matrix operation block MAB may read only parameters to be used in the subsequent step S4-3 or S4-4. The matrix operation block MAB may sequentially read at least a portion of the parameters of the machine learning model M stored in the memory block MB when they are needed for processing.

[0129] The matrix calculation block MAB does not need to read out parameters that have already been read out in the pre-filling process. If all parameters have been read out in the pre-filling process, the matrix calculation block MAB does not need to execute step S4-1.

[0130] In step S4-2, the matrix operation block MAB of the arithmetic device 100 reads intermediate data from the memory block MB. The matrix operation block MAB may read at least a portion of the intermediate data stored in the memory block MB. The matrix operation block MAB may read all of the intermediate data stored in the memory block MB. The matrix operation block MAB may read at least one of the intermediate data generated in the prefilling process and the intermediate data generated in the decoding process. The matrix operation block MAB may read at least a portion of the intermediate data generated in the prefilling process. The matrix operation block MAB may read at least a portion of the intermediate data generated in the decoding process. The matrix operation block MAB may read at least a portion of new intermediate data generated in a previous decoding process. The matrix operation block MAB may sequentially read at least a portion of the intermediate data stored in the memory block MB when it is needed for processing.

[0131] In step S4-3, the matrix operation block MAB of the arithmetic device 100 generates intermediate data. The matrix operation block MAB may generate the intermediate data by calculating the attention ATT of the machine learning model M. The matrix operation block MAB may generate the intermediate data corresponding to the output information generated in the pre-filling process. The matrix operation block MAB may generate the intermediate data corresponding to the output information generated in the decoding process. The matrix operation block MAB may generate the intermediate data using the parameters read in step S4-1.

[0132] The matrix operation block MAB may generate new intermediate data using the intermediate data read out in step S4-2. The matrix operation block MAB may generate new intermediate data using at least a portion of the intermediate data generated in the pre-filling process. The matrix operation block MAB may generate new intermediate data using at least a portion of the intermediate data generated in the decoding process. The matrix operation block MAB may generate new intermediate data using at least a portion of the intermediate data generated in the pre-filling process and at least a portion of the intermediate data generated in the decoding process.

[0133] The matrix operation block MAB may store the generated new intermediate data in the memory block MB. The matrix operation block MAB may store at least a part of the new intermediate data in the memory block MB. The matrix operation block MAB may store the new intermediate data in the memory block MB connected to the matrix operation block MAB. At least a part of the intermediate data stored in the memory block MB may be read out in the next decoding process to be executed.

[0134] In step S4-4, the matrix operation block MAB of the arithmetic device 100 generates output information to be output from the machine learning model M. The matrix operation block MAB may generate the output information by calculating a multilayer perceptron MLP of the machine learning model M. The matrix operation block MAB may generate the output information using the parameters read in step S4-1. The matrix operation block MAB may generate the output information using the intermediate data read in step S4-2 and the intermediate data generated in step S4-3.

[0135] The matrix calculation block MAB may store the generated output information in its local memory LM1 or in the memory block MB connected to the matrix calculation block MAB.

[0136] The matrix operation block MAB may output the generated output information. The matrix operation block MAB may output output information including a predetermined number of tokens. The matrix operation block MAB may output the output information at predetermined time intervals. The output information output from the matrix operation block MAB may be output to the outside of the arithmetic device 100. As an example, the output information may be transferred to the host memory HOSTM.

[0137] The matrix calculation block MAB may repeatedly execute the processes from steps S4-1 to S4-4. The matrix calculation block MAB may repeatedly execute the processes from steps S4-1 to S4-4 until a predetermined condition is satisfied. As an example, the matrix calculation block MAB may repeatedly execute the processes from steps S4-1 to S4-4 until terminal information of the output information is generated in step S4-4. As an example, the terminal information of the output information may be an EOS token (End of Sequence Token). As another example, the matrix calculation block MAB may repeatedly execute the processes from steps S4-1 to S4-4 until the number of repetitions reaches a predetermined number. The matrix calculation block MAB may read all parameters stored in the memory block MB in step S4-1, and repeatedly execute only the processes from steps S4-2 to S4-4.

[0138] As another example, the arithmetic device 100 may execute a decoding process of the state space model. As another example, the arithmetic device 100 may input the last token generated by the prefill process to the state space model. The state space model may read the state of the state space model stored in the memory block MB and execute forward processing on the input token. The state space model may generate a token following the input token and update the state of the state space model. The arithmetic device 100 may acquire the token output by the state space model as output information. The arithmetic device 100 may overwrite the state stored in the memory block MB with the state of the updated state space model.

[0139] (Multilayer Perceptron calculation) The calculation of the multilayer perceptron MLP will be explained in more detail below. As an example, the calculation of the multilayer perceptron MLP in the decoding process will be explained.

[0140] In this embodiment, the multilayer perceptron MLP takes x as the input of a 32x8192 matrix, and W up Let be the parameters of an 8192 × 32768 matrix, and W down Let be the parameters of a 32768 × 8192 matrix, and f be the activation function, defined by equations (1) and (2).

[0141]

number

[0142] However, the activation function f may be a ReLU function or a SiLU function. The ReLU function may be a function defined by equation (3). The SiLU function may be a function defined by equation (4).

[0143]

number

[0144] A characteristic of multilayer perceptron (MLP) calculations is that one of the inputs in a matrix multiplication calculation is a long, narrow matrix, such as a 32x8192 matrix. 32 corresponds to the batch size. A small batch size has the advantage of reducing token-to-token latency and memory usage.

[0145] Conventional processors cannot perform such matrix multiplications quickly. up ,W down Because the size of is large, the parameter W up ,W down This is because the speed at which the data is read is slower than the speed at which it is calculated.

[0146] The calculation device 100 calculates the parameter W up ,W down In the arithmetic device 100, a plurality of matrix operation blocks MAB share the task of performing matrix multiplication.

[0147] As an example, the input information x and the parameter W up The product of x·W up The process may be divided into 96 matrix operation blocks MAB as follows: The input information x is a 32 × 8192 matrix. The parameter W up is an 8192 x 32768 matrix.

[0148] Two blocks L2B per logic die LD have the same input information x. Also, the two blocks L2B have the same parameter W up is divided into two columns, up (1,2) Each parameter has W up (1,2) is an 8192x16384 matrix.

[0149] In addition, x·W up The calculation can be realized by performing the following processes (1) to (2) in parallel. Here, the 0th block L2B[0] is the parameter W up (1,2) 0, and the first block L2B[1] has parameter W up (1,2) Let's say we have 1.

[0150] (1) Block L2B[0] is the input information x and parameter W up (1,2) Matrix multiplication with 0 x W up (1,2) Calculate 0. The result is a 32x16384 matrix.

[0151] (2) Block L2B[1] receives input information x and parameter W up (1,2) Matrix multiplication with 1 x W up (1,2)Calculate 1. The result is a 32x16384 matrix.

[0152] Four blocks L1B per block L2B have the input information x duplicated. Also, the four blocks L1B have the parameter W up (1,2) is divided into four columns, up (1,8) Each parameter has W up (1,8) is an 8192x4096 matrix.

[0153] The 16 matrix operation blocks MAB per block L1B divide the input information x into four columns and (1,4) Each has input information x (1,4) is a 32 × 2048 matrix. The 16 matrix operation blocks MAB are up (1,8) The parameter W is divided into four parts in each matrix direction. up (4,32) Each parameter has W up (4,32) is a 2048x1024 matrix.

[0154] Each of the 16 matrix operation blocks MAB receives input information x (1,4) and parameter W up (4,32) matrix multiplication with x (1,4) ·W up (4,32) The result is a 32x1024 matrix.

[0155] Next, the calculation device 100 divides the 16 matrix calculation blocks MAB per block L1B into groups of four. For each group, the calculation device 100 calculates the calculation result x (1,4) ·W up (4,32) Since there are four groups, the result is x W up (1,8) is a 32 × 4096 matrix. The calculation device 100 calculates the result x·W up(1,8) may be stored in the memory L1BM.

[0156] The calculation device 100 calculates the calculation results x·W for each of the eight blocks L1B. up (1,8) Arrange and combine. Calculation result x W up is a 32 × 32768 matrix. The calculation device 100 converts the result x·W up may be divided and stored in memory blocks MB connected to the matrix calculation blocks MAB.

[0157] As another example, the output f(y) of the activation function f and the parameter W down The product of x·W down The processing may be divided into 96 matrix operation blocks MAB as follows: The output f(y) of the activation function f is a 32x32768 matrix. The parameter W down is a 32768x8192 matrix.

[0158] Two blocks L2B per logic die LD divide the output f(y) of activation function f into two columns and generate the output y (1,2) Each has output y (1,2) is a 32x16384 matrix. The two blocks L2B are separated by a parameter W down is divided into two in the row direction, down (2,1) Each parameter has W down (2,1) is a 16384x8192 matrix.

[0159] Four L1B blocks per L2B output y (1,2) The output y is divided into four columns (1,8) Each has output y (1,8) is a 32x4096 matrix. The four blocks L1B are down (2,1) is divided into four in the row direction, down (8,1) Each parameter has W down (8,1)is a 4096x8192 matrix.

[0160] The 16 matrix operation blocks MAB per block L1B output y (1,8) The output y is divided into four columns (1,32) Each has output y (1,32) is a 32 × 1024 matrix. The 16 matrix operation blocks MAB are down (8,1) The parameter W is divided into four parts in each matrix direction. down (32,4) Each parameter has W down (32,4) is a 1024x2048 matrix.

[0161] Each of the 16 matrix operation blocks MAB outputs y (1,32) and parameter W down (32,4) matrix product y (1,32) ·W down (32,4) The result is a 32x2048 matrix.

[0162] Next, the calculation device 100 divides the 16 matrix calculation blocks MAB per block L1B into groups of four. For each group, the calculation device 100 calculates the calculation result y (1,32) ·W down (32,4) Since there are four groups, the result is y (1,8) ·W down (8,1) is a 32×8192 matrix. The calculation device 100 calculates the result y (1,8) ·W down (8,1) may be stored in the memory L1BM.

[0163] The four L1B blocks per L2B are the calculation results y (1,8) ·W down (8,1) Add the result y (1,2) ·W down(2,1) is a 32×8192 matrix. The calculation device 100 calculates the result y (1,2) ·W down (2,1) may be stored in the memory L2BM.

[0164] The two blocks L2B per logic die LD are calculated by the calculation results y (1,2) ·W down (2,1) Add the result y W down The calculation device 100 calculates the calculation results y·W of each of the two blocks L2B so that they can be used in the subsequent matrix multiplication. down One or both of these may be divided and stored in memory blocks MB connected to each matrix calculation block MAB.

[0165] The above calculations may be performed serially or in parallel. For example, after the matrix multiplication calculations are partially completed, the summation may be performed in parallel. For example, after the matrix multiplication calculations are completed in multiple blocks L1B, the summation of the matrices resulting from those calculations may be calculated.

[0166] The above is an outline of the process of calculating the matrix product using the arithmetic device 100. The parameter W of the machine learning model M up ,W down Since the amount of data is large, in this embodiment, the parameter W is stored in the multiple matrix calculation blocks MAB included in the calculation device 100. up ,W down is divided and allocated to the memory blocks MB connected to the matrix calculation blocks MAB, and stored in the memory blocks MB connected to the matrix calculation blocks MAB. up ,W down are divided so as not to overlap each other. This enables the arithmetic device 100 to read the parameters of the machine learning model M from the memory block MB at high speed.

[0167] matrix product x W up In the calculation of(1,4) ·W up (4,32) The input information x (1,4) is a 32x2048 matrix. The parameter W up (4,32) is a 2048x1024 matrix.

[0168] The matrix operation block MAB has a register called a matrix register. As an example, the matrix register may be included in a matrix calculator MAU of the matrix operation block MAB. In this embodiment, the matrix register can store a 32×64 matrix. First, the matrix operation block MAB writes one 32×64 matrix M to be multiplied into the matrix register. Next, the matrix operation block MAB inputs the other matrices to be multiplied, namely, a 64×2 matrix B and a 32×2 matrix C, into the matrix register. The 32×2 matrix C is also called an "accumulator." The matrix register calculates D=M·B+C and outputs a 32×2 matrix D. Matrices M and B may be quantized using FP8. Matrices C and D may be quantized using FP16.

[0169] Calculations using the matrix registers are performed by each matrix operation block MAB once per cycle. As an example, we will explain how to calculate the matrix product of two 64x64 matrices. Let A and B be the two matrices to be multiplied.

[0170] Here, we introduce a notation that uses subscripts to represent submatrices. i:j,k:l Let be a (ji) × (lk) matrix obtained by extracting rows i to j-1 and columns k to l-1 of matrix A. As an example, A 0:32,0:64 is a submatrix extracted from the upper half of matrix A. Also, A 0:32,0:32 is a submatrix obtained by extracting the upper left half of matrix A.

[0171] First, the matrix operation block MAB is 0:32,0:64 is written to the matrix register. Next, the matrix operation block MAB calculates equation (5) for integers i between 0 and 31, where 0 is the zero matrix.

[0172]

number

[0173] Next, the matrix operation block MAB calculates A 32:64,0:64 is written to the matrix register. Next, the matrix operation block MAB calculates equation (6) for integer i between 0 and 31.

[0174]

number

[0175] This completes the multiplication of two 64x64 matrices. If the matrix is ​​larger than 64x64, simply input the previous calculation result instead of the zero matrix.

[0176] In a large-scale language model, which is an example of a machine learning model M, a matrix called an activation (such as the x or y mentioned above) is written to a matrix register, and the parameter W up ,W down The calculation device 100 performs a process of writing 64×2 values ​​into the matrix register per cycle. up ,W down can be read from the memory block MB at high speed. Therefore, by using the arithmetic device 100, the inference process of the machine learning model M can be executed at high speed.

[0177] When executing inference processing of the machine learning model M using the arithmetic device 100, data movement between matrix operation blocks MAB can be reduced by appropriately dividing the parameters of the machine learning model M. The arithmetic device 100 is configured so that each of the multiple processing elements PE can treat the memory block MB as a local memory. Therefore, by storing the parameters used by each processing element PE in the memory block MB connected to that processing element PE, the inference processing of the machine learning model M can be executed at high speed.

[0178] So far, we have explained the calculations of the multilayer perceptron MLP in the decoding process. In the calculations of the multilayer perceptron MLP in the decoding process, the input information x is a 32 x 8192 matrix, and the number of rows, 32, is the batch size. On the other hand, in the calculations of the multilayer perceptron MLP in the prefilling process, calculations can be performed with a predetermined number of tokens as the processing unit for one piece of input information (in other words, a batch size of 1). In other words, in the calculations of the multilayer perceptron MLP in the decoding process, the number of rows, 32, of the input information x corresponds to the batch size, but in the calculations of the multilayer perceptron MLP in the prefilling process, the number of rows, 32, of the input information x corresponds to the number of tokens input to the machine learning model M.

[0179] (Attention calculation) We will now explain the calculation of attention ATT in more detail. As an example, we will explain the calculation of a multi-layer perceptron (MLP) in the decoding process.

[0180] The calculation of the attention ATT may include a part similar to the calculation of the multilayer perceptron MLP and a part that calculates the attention itself. The part that calculates the attention itself may include a process for calculating a key-value cache. Below, the explanation of the part similar to the calculation of the multilayer perceptron MLP will be omitted.

[0181] In this embodiment, the attention ATT takes as input a query vector Q, a key vector K, and a value vector V. The query vector Q is a 32×8192 matrix. The key vector K is a 32×L×1024-dimensional vector. The value vector V is a 32×L×1024-dimensional vector. Here, L is an integer called the context length. The context length L increases by 1 each time the machine learning model M infers one token. Therefore, the calculation of the attention ATT needs to be configured to be able to accommodate various values ​​of L.

[0182] The arithmetic device 100 divides each of the query vector Q, key vector K, and value vector V input to the attention ATT into 64 heads. As an example, the arithmetic device 100 divides the query vector Q into 64 parts, resulting in 64 32×128 matrices q. The arithmetic device 100 assigns the 64 matrices q to the 64 heads. Furthermore, the arithmetic device 100 divides the key vector K into eight parts, resulting in eight 32×L×128 tensors k. The arithmetic device 100 overlaps the eight tensors k by eight, and assigns them to the 64 heads. Furthermore, the arithmetic device 100 divides the value vector V into eight 32×L×128 tensors v, and overlaps the eight tensors k by eight, and assigns them to the 64 heads.

[0183] Hereinafter, the matrix q, tensor k, and tensor v assigned to the head can be regarded as 32 128-dimensional column vectors q', an L × 128-dimensional tensor k', and an L × 128-dimensional tensor v', respectively. The calculation device 100 calculates Equation (7) for the 32 q', k', and v'.

[0184]

number

[0185] however,· T (The superscript T) indicates transposition. In the following, the notation for the Softmax function will be omitted.

[0186] The calculation of the attention mechanism itself consists of a primitive multiplication of an 8x128 matrix and a 128xL matrix, and a primitive multiplication of an 8xL matrix and an Lx128 matrix, per eight heads. Since L is usually larger than 8, the speed of reading the 128xL and Lx128 matrices from the memory block MB becomes an issue. By appropriately dividing the query vector Q, key vector K, and value vector V, the calculation speed can be increased to the same level as that of a multilayer perceptron (MLP). Here, the key vector K and value vector V are each 32xLx1024 tensors, and the query vector Q is a 32x8192 matrix. Therefore, the division prioritizes the larger key vector K and value vector V.

[0187] Two blocks L2B per logic die LD are tensors K (1,1,2) tensor K (1,1,2) is a 32×L×512 tensor. The two blocks L2B are tensors V obtained by dividing the key vector V into two. (1,1,2) tensor V (1,1,2) is a 32×L×512 tensor. Furthermore, the two blocks L2B are the matrix Q obtained by dividing the query vector Q into two. (1,2) Each matrix Q (1,2) is a 32x4096 matrix.

[0188] Four L1B blocks per L2B are used to store the tensor K (1,1,2) Tensor K divided into four parts (1,1,8) tensor K (1,1,8) is a 32×L×128 tensor. Also, the four blocks L1B are tensor V (1,1,2) Tensor V divided into four parts (1,1,8) tensor V (1,1,8) is a 32×L×128 tensor. Furthermore, the four blocks L1B are (1,2) is divided into four parts, (1,8) Each matrix Q (1,8) is a 32x1024 matrix.

[0189] The 16 matrix operation blocks MAB per block L1B are used to calculate the tensor K (1,1,8) Tensor K divided into 16 parts (1,16,8) tensor K (1,16,8) is a 32 × (L / 16) × 128 tensor. Also, the 16 matrix operation blocks MAB are (1,1,8) Tensor V divided into 16 parts (1,16,8) tensor V (1,16,8) is a 32×L / 16×128 matrix. Furthermore, the 16 matrix operation blocks MAB calculate the matrix Q (1,8) have overlapping.

[0190] For each of the four blocks L1B, the following processes (1) to (6) are repeated 32 times.

[0191] (1) Block L1B is a 32x1024 matrix Q (1,8) Block L1B extracts one row (i.e., a 1024-dimensional vector) from L1. Block L1B regards the extracted 1024-dimensional vector as an 8x128 matrix q.

[0192] (2) Block L1B is a 32×L×128 tensor K (1,1,8) Extract the L×128 matrix k corresponding to the matrix q from

[0193] (3) Block L1B is q@k T and obtain an 8×L matrix t.

[0194] (4) Block L1B calculates the Softmax function for each row of matrix t and defines the result as s.

[0195] (5) Block L1B is a 32×L×128 tensor V (1,1,8) Extract the L×128 matrix v corresponding to the matrix q from

[0196] (6) Block L1B calculates s@v and obtains an 8x128 matrix. Block L1B regards the 8x128 matrix as a 1024-dimensional vector.

[0197] Each of the four blocks L1B combines 32 1024-dimensional vectors obtained by repeating the process 32 times to obtain a 32 x 1024 matrix. Since the arithmetic unit 100 has eight blocks L1B in the logic die LD, a total of eight 32 x 1024 matrices are obtained. The arithmetic unit 100 combines the eight 32 x 1024 matrices to obtain a 32 x 8192 matrix. The 32 x 8192 matrix obtained in this way is the calculation result of the attention main body.

[0198] In the calculation of the above block L1B, the matrix calculation block MAB in block L1B repeats the following steps (1) to (8) 32 times.

[0199] (1) The 16 matrix operation blocks MAB have duplicated 8x128 matrices q.

[0200] (2) The matrix operation block MAB writes the lower half of the matrix q, an 8x64 matrix, into the matrix register.

[0201] (3) Memory block MB stores an (L / 16) × 128 matrix k. Matrix operation block MAB reads a 2 × 64 matrix from matrix k per cycle and calculates its product with the lower half of matrix q. The result is an 8 × (L / 16) matrix. This calculation takes L / 32 cycles.

[0202] (4) The matrix operation block MAB writes the upper half of the matrix q, an 8x64 matrix, into the matrix register.

[0203] (5) The matrix operation block MAB calculates the product of matrix k and the upper half of matrix q in the same way as in process (3). The calculation result is an 8 × (L / 16) matrix. The matrix operation block MAB calculates the sum of two 8 × (L / 16) matrices. Note that the calculations of processes (2) to (5) may be performed by calculating the matrix products one by one and then adding them up, or may be performed all at once using a matrix multiply-and-accumulate operation.

[0204] (6) The matrix operation block MAB calculates the Softmax function for each row of the 8 × (L / 16) matrix and sets the calculation result as s. At this time, the matrix operation block MAB needs to communicate with other matrix operation blocks MAB in the same block L1B.

[0205] (7) The matrix operation block MAB calculates the 8 × (L / 16) matrix s and the (L / 16) × 128 matrix v. Specifically, the matrix operation block MAB repeats the following process ceil(L / 1024) times. First, it writes the 8 × 64 portion of matrix s to the matrix register. Next, it reads the 64 × 2 portion of matrix v from the memory block MB per cycle and inputs it to the matrix register. At this time, it performs accumulation appropriately. As a result, an 8 × 128 matrix is ​​obtained.

[0206] (8) The 16 matrix operation blocks MAB treat an 8x128 matrix as a 1024-dimensional vector and calculate the sum.

[0207] As a result of repeating the processes (1) to (8) 32 times, a 32×1024 matrix is ​​obtained for block L1B, and a 32×8192 matrix is ​​obtained for the arithmetic device 100 as a whole.

[0208] In the calculation of block L1B, it is important that matrix k and matrix v are read in parallel from each matrix calculation block MAB. The above calculation procedure can be realized if matrix k and matrix v are appropriately divided and stored in memory block MB. The division method of matrix k and matrix v is naturally derived from the above calculation procedure. That is, matrix k and matrix v are divided into channels by two blocks L2B, then divided into channels by four blocks L1B within block L2B, and the context length L is divided by 16 matrix calculation blocks MAB within block L1B.

[0209] The matrices k and v are generated each time a new token is inferred, and may be written to the appropriate memory block MB each time. For example, when a 1024-dimensional vector k is generated, vector k may be divided into eight, resulting in 128-dimensional vectors that may be assigned to eight blocks L1B. When matrix k corresponds to the nth token, the 128-dimensional vector may be assigned to the matrix operation block MAB that corresponds to the remainder when n is divided by 16. The matrix operation block MAB may write the 128-dimensional vector assigned to the matrix operation block MAB to the memory block MB connected to the matrix operation block MAB.

[0210] Up to this point, the calculation of attention ATT in the decoding process has been described. Below, as another example, the calculation of attention ATT in the prefilling process will be described. Note that here, the calculation of attention ATT in the prefilling process will be described, focusing on the differences from the calculation of attention ATT in the decoding process. Unless otherwise specified, the calculation of attention ATT in the prefilling process may be configured in the same way as the calculation of attention ATT in the decoding process.

[0211] The calculation of attention ATT in the pre-filling process differs from the calculation of attention ATT in the decoding process in the following ways:

[0212] (1) Treat the 32x8192 matrix q as a 32x8x8x128 tensor.

[0213] (2) Eight blocks L1B per logic die LD correspond to a 32×8×8×128 tensor obtained by dividing a 32×8×8×128 tensor into eight parts.

[0214] (3) The 16 matrix operation blocks MAB in block L1B have overlapping 32 × 8 × 128 tensors.

[0215] (4) The matrix operation block MAB performs the following processing: <1> ~ <7> Repeat 8 times.

[0216] <1> The matrix operation block MAB writes the lower half of the matrix q, a 32x64 matrix, into the matrix register.

[0217] <2> The memory block MB stores an (L / 16) x 128 matrix k. The matrix operation block MAB reads 2 x 64 matrices from matrix k per cycle and calculates the product with the lower half of matrix q. The result is 32 x (L / 16). This calculation takes L / 8 cycles.

[0218] <3> The matrix operation block MAB writes the upper half of the matrix q, a 32x64 matrix, into the matrix register.

[0219] <4> The matrix operation block MAB processes <2> Similarly, the product of matrix k and the upper half of matrix q is calculated. The calculation result is a 32 × (L / 16) matrix. The matrix operation block MAB calculates the sum of two 32 × (L / 16) matrices. <1> ~ <4> The calculation of may be performed by adding up the matrix products after each calculation, or may be performed at once by a matrix multiplication and addition operation.

[0220] <5> The matrix operation block MAB calculates the Softmax function for each row of the 32 × (L / 16) matrix and sets the calculation result as s. At this time, the matrix operation block MAB needs to communicate with other matrix operation blocks MAB in the same block L1B.

[0221] <6> The matrix operation block MAB calculates a 32 × (L / 16) matrix s and an (L / 16) × 128 matrix v. Specifically, the matrix operation block MAB repeats the following process ceil(L / 1024) times. First, it writes a 32 × 64 portion of matrix s to the matrix register. Next, it reads a 64 × 2 portion of matrix v from the memory block MB per cycle and inputs it to the matrix register, repeating this process 64 times. At this time, it performs accumulate appropriately. As a result, a 32 × 128 matrix is ​​obtained.

[0222] <7> The 16 matrix operation blocks MAB treat a 32x128 matrix as a 4096-dimensional vector and calculate the sum.

[0223] process <1> ~ <7> As a result of repeating the above eight times, a 32×1024 matrix is ​​obtained for block L1B. The calculation device 100 obtains a 32×8192 matrix overall.

[0224] The matrix operation block MAB may execute the pre-filling process in step S3 and the decoding process in step S4 in parallel. In other words, while the matrix operation block MAB executes the pre-filling process in some batches, it may execute the pre-filling process in some other batches.

[0225] As an example, assume that the batch size of the matrix operation block MAB is 17, 16 prefill processes have been completed, and one prefill process is incomplete. In this case, the matrix operation block MAB may execute 16 decoding processes and one prefill process in parallel. Specifically, the matrix operation block MAB may input a total of 32 tokens to the machine learning model M, consisting of 16 tokens in one prefill process and one token each in the 16 decoding processes. In the calculation of the attention main body, the matrix operation block MAB may execute a calculation for a decoding process with a batch size of 16 and a calculation for a prefill process with 16 tokens.

[0226] To calculate the attention body of the Lth token, the Softmax function can be configured as follows:

[0227] Softmax((q of Lth token)@(k of 1st to Lth token) T @(v of 1st to Lth tokens))

[0228] For example, the throughput of the inference process of the machine learning model can be improved by executing the inference process of the machine learning model using the computing device 100. The throughput may include, for example, the throughput per user (e.g., the number of tokens that can be processed per second) or the throughput per cost (e.g., the number of tokens that can be processed per currency unit).

[0229] Throughput per user is the throughput seen from the user's perspective. As an example, consider an information processing system that interacts with users using a machine learning model. Some information processing systems of this type start outputting answers after receiving all input from the user, and the answers are displayed intermittently on the screen. You could say that throughput per user is the speed at which answers are displayed on the screen.

[0230] The throughput per user is mainly determined by the following factors: Processor memory bandwidth and capacity Parallelism of tensor parallelism Network bandwidth used for tensor parallel processing Batch size

[0231] The batch size can be determined appropriately when designing an information processing system. When the batch size is determined appropriately and the network bandwidth is sufficient, the throughput per user depends on the time required to read the machine learning model parameters from memory. For example, the throughput per user is proportional to the speed at which the machine learning model parameters can be read from memory.

[0232] The use of the arithmetic device 100 can speed up the reading of machine learning model parameters. Furthermore, by parallelizing the reading of machine learning model parameters using tensor parallel processing or the like, the throughput per user can be significantly improved. For example, compared to GPGPUs (General Purpose Graphics Processing Units) equipped with HBM (High Bandwidth Memory), the throughput per die can be improved by approximately 5 to 10 times. Furthermore, as a result, the cost per die can be reduced to approximately one-fifth.

[0233] Furthermore, according to this embodiment, the throughput per user is improved, making it easier to support large machine learning models. For example, because SRAM has limited storage capacity, when attempting to execute inference processing for a large machine learning model, a large number of processors must be allocated to each machine learning model. Furthermore, improved throughput per user reduces the latency or bandwidth requirements of the inter-chip network. As a result, the overall system cost can be reduced.

[0234] Some or all of the inference processing of the machine learning model in the above-described embodiments may be implemented by hardware, or may be implemented by software (program) information processing executed by a CPU, GPU (Graphics Processing Unit), or the like. When implemented by software information processing, software that realizes at least some of the functions of the inference processing of the machine learning model in the above-described embodiments may be stored on a non-transitory storage medium (non-transitory computer-readable medium) such as a CD-ROM (Compact Disc-Read Only Memory) or USB (Universal Serial Bus) memory, and the software information processing may be executed by loading the software into a computer. The software may also be downloaded via a communication network. Furthermore, all or part of the software processing may be implemented in a circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA, thereby allowing the software information processing to be executed by hardware.

[0235] The storage medium that stores the software may be a removable medium such as an optical disk, or a fixed medium such as a hard disk, memory, etc. The storage medium may be provided inside the computer (main storage device, auxiliary storage device, etc.) or outside the computer.

[0236] 21 is a block diagram showing an example of the hardware configuration of a computer on which the above-described arithmetic device 100 is installed. In the following, the explanation will be given assuming that the arithmetic device 100 is installed in the computer. In FIG. 21, the computer may be realized as a computer 500 including, as an example, the arithmetic device 100, a main storage device 30 (memory), an auxiliary storage device 40 (memory), a network interface 50, and a device interface 60, which are connected via a bus 510.

[0237] Although the computer 500 in FIG. 21 includes one of each component, it may also include multiple of the same component. Also, while FIG. 21 shows one computer 500, the software may be installed on multiple computers, and each of the multiple computers may execute the same or different parts of the software's processing. In this case, a distributed computing configuration may be used in which each computer communicates with the other computers via a network interface 50 or the like to execute processing. In other words, a system may be configured in which one or more computers 500 execute instructions stored in one or more storage devices to achieve a function. Furthermore, a configuration may be used in which information transmitted from a terminal is processed by one or more computers 500 provided on a cloud, and the processing results are then transmitted to the terminal.

[0238] Various computations may be executed in parallel using one or more computation devices 100 installed in the computer 500, or using multiple computers 500 via a network. Furthermore, various computations may be distributed to processing elements PE, which are an example of multiple computation cores within the computation device 100, and executed in parallel. Furthermore, some or all of the processes, means, etc. disclosed herein may be implemented by at least one of a processor and a storage device provided on a cloud that can communicate with the computer 500 via a network. Thus, each device in the above-described embodiment may be implemented in the form of parallel computing using one or multiple computers.

[0239] The computing device 100 may be an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.) that performs at least one of computer control and calculation. The computing device 100 may be either a dedicated processing circuit designed to perform a specific calculation, or a computing device that includes both a general-purpose processor and a dedicated processing circuit. The computing device 100 may also include an optical circuit, or may include a calculation function based on quantum computing.

[0240] The arithmetic device 100 may perform arithmetic processing based on data or software input from each device or the like configured internally of the computer 500, and may output arithmetic results or control signals to each device or the like. The arithmetic device 100 may control each component constituting the computer 500 by executing an OS (Operating System) of the computer 500, an application, or the like.

[0241] The inference process of the machine learning model in the above-described embodiment may be realized by one or more arithmetic devices 100. Here, the arithmetic device 100 may refer to one or more electronic circuits arranged on one chip, or may refer to one or more electronic circuits arranged on two or more chips or two or more devices. When multiple electronic circuits are used, the respective electronic circuits may communicate with each other via wire or wirelessly.

[0242] The main memory device 30 may store instructions executed by the arithmetic device 100 and various data, and information stored in the main memory device 30 may be read by the arithmetic device 100. The auxiliary memory device 40 is a memory device other than the main memory device 30. Note that these memory devices refer to any electronic component capable of storing electronic information, and may be semiconductor memory. The semiconductor memory may be either volatile memory or non-volatile memory. A memory device for saving various data, etc. in the computer 500 may be realized by the main memory device 30 or the auxiliary memory device 40, or may be realized by an internal memory built into the arithmetic device 100.

[0243] When the computer 500 is configured with at least one storage device (memory) and at least one arithmetic device 100 connected (coupled) to this at least one storage device, at least one arithmetic device 100 may be connected to one storage device. Also, at least one storage device may be connected to one arithmetic device 100. Also, a configuration may be included in which at least one arithmetic device 100 out of multiple arithmetic devices 100 is connected to at least one storage device out of multiple storage devices. Also, this configuration may be realized by storage devices and arithmetic devices 100 included in multiple computers 500. Furthermore, a configuration in which a storage device is integrated with the arithmetic device 100 (for example, a cache memory including an L1 cache and an L2 cache) may be included.

[0244] The network interface 50 is an interface for connecting to the communication network 600 wirelessly or via a wire. The network interface 50 may be an appropriate interface, such as one that conforms to an existing communication standard. Information may be exchanged with an external device 710 connected via the communication network 600 through the network interface 50. The communication network 600 may be any one of a wide area network (WAN), a local area network (LAN), a personal area network (PAN), etc., or a combination thereof, as long as information is exchanged between the computer 500 and the external device 710. An example of a WAN is the Internet, an example of a LAN is IEEE802.11 or Ethernet (registered trademark), and an example of a PAN is Bluetooth (registered trademark) or NFC (Near Field Communication), etc.

[0245] The device interface 60 is an interface such as a USB that directly connects to an external device 720 .

[0246] The external device 710 is a device connected to the computer 500 via a network. The external device 720 is a device directly connected to the computer 500.

[0247] For example, the external device 710 or the external device 720 may be an input device. The input device is, for example, a device such as a camera, a microphone, a motion capture device, various sensors, a keyboard, a mouse, or a touch panel, and provides acquired information to the computer 500. Alternatively, the external device 710 or the external device 720 may be a device including an input unit, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.

[0248] Furthermore, the external device 710 or the external device 720 may be, for example, an output device. The output device may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) panel, or may be a speaker that outputs sound or the like. Alternatively, the output device may be a device including an output unit, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.

[0249] Furthermore, the external device 710 or the external device 720 may be a storage device (memory). For example, the external device 710 may be a network storage or the like, and the external device 720 may be a storage such as an HDD.

[0250] Furthermore, external device 710 or external device 720 may be a device having some of the functions of the components of computer 500. In other words, computer 500 may transmit some or all of the processing results to external device 710 or external device 720, or may receive some or all of the processing results from external device 710 or external device 720.

[0251] In this specification (including the claims), when the expression "at least one of a, b, and c" or "at least one of a, b, or c" (including similar expressions) is used, it includes any of a, b, c, ab, ac, bc, or abc. It may also include multiple instances of any element, such as aa, abb, aabbcc, etc. Furthermore, it also includes the addition of elements other than the enumerated elements (a, b, and c), such as having d, as in abcd.

[0252] In this specification (including claims), when expressions such as "using data as input / based on / according to / in response to data" (including similar expressions) are used, unless otherwise specified, this includes cases where the data itself is used, or where data that has been processed in some way (e.g., data with noise added, normalized data, features extracted from data, intermediate representations of data, etc.) is used. Furthermore, when a statement is made that a result is obtained "using data as input / based on / according to / in response to data" (including similar expressions), this includes cases where the result is obtained based solely on the data, or where the result is influenced by other data, factors, conditions, and / or states other than the data itself, unless otherwise specified. Furthermore, when a statement is made that "data is output" (including similar expressions), this includes cases where the data itself is used as output, or where data that has been processed in some way (e.g., data with noise added, normalized data, features extracted from data, intermediate representations of various data, etc.) is used as output, unless otherwise specified.

[0253] When the terms "connected" and "coupled" are used in this specification (including the claims), they are intended as open-ended terms that encompass any of direct connection / coupling, indirect connection / coupling, electrically connection / coupling, communicatively connection / coupling, functionally connection / coupling, and physically connection / coupling. These terms should be interpreted appropriately according to the context in which they are used, but any form of connection / coupling that is not intentionally or naturally excluded should be interpreted as being included in these terms without limitation.

[0254] In this specification (including the claims), the expression "A configured to B" may include the physical structure of element A having a configuration capable of performing operation B, and the permanent or temporary setting / configuration of element A being configured / set to actually perform operation B. For example, if element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and is configured to actually perform operation B by setting a permanent or temporary program (instruction). Also, if element A is a dedicated processor, dedicated arithmetic circuit, etc., it is sufficient that the circuit structure, etc. of the processor is implemented to actually perform operation B, regardless of whether control instructions and data are actually attached.

[0255] Whenever words implying containing or possessing (e.g., "comprising / including," "having," etc.) are used in this specification (including the claims), they are intended to be open-ended terms that include the inclusion or possession of things other than the object designated by the object of the term. When the object of such words implying containing or possessing does not specify a quantity or suggests a singular number (e.g., expressions using the articles "a" or "an"), the expression should be construed as not being limited to a specific number.

[0256] In this specification (including the claims), even if expressions such as "one or more" and "at least one" are used in some places and expressions that do not specify a quantity or that imply a singular number (expressions using the articles "a" or "an") are used in other places, the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or that imply a singular number (expressions using the articles "a" or "an") should be interpreted as not necessarily being limited to a specific number.

[0257] In this specification, when a particular advantage / result is described as being obtained with respect to a particular configuration of an embodiment, it should be understood that the same advantage / result can also be obtained with one or more other embodiments having the same configuration, unless otherwise stated. However, it should be understood that the presence or absence of the effect generally depends on various factors, conditions, and / or circumstances, and that the effect is not necessarily obtained with the configuration. The effect is merely obtained by the configuration described in the embodiment when various factors, conditions, and / or circumstances are satisfied, and the effect does not necessarily occur in a claimed invention that defines the same or a similar configuration.

[0258] When terms such as "maximize" and "maximization" are used in this specification (including the claims), they include finding a global maximum, finding an approximation of a global maximum, finding a local maximum, and finding an approximation of a local maximum, and should be interpreted appropriately according to the context in which the term is used. They also include finding approximations of these maxima probabilistically or heuristically. Similarly, when terms such as "minimize" and "minimization" are used, they include finding a global minimum, finding an approximation of a global minimum, finding a local minimum, and finding an approximation of a local minimum, and should be interpreted appropriately according to the context in which the term is used. They also include finding approximations of these minima probabilistically or heuristically. Similarly, when terms such as "optimize" and "optimization" are used, they include finding a global optimum, finding an approximation of a global optimum, finding a local optimum, and finding an approximation of a local optimum, and should be interpreted appropriately according to the context in which the term is used. It also includes finding approximations of these optimum values ​​probabilistically or heuristically.

[0259] In this specification (including claims), when multiple pieces of hardware perform a predetermined process, the pieces of hardware may cooperate to perform the predetermined process, or some of the hardware may perform all of the predetermined process. Furthermore, some of the hardware may perform part of the predetermined process, and other hardware may perform the rest of the predetermined process. In this specification (including claims), when an expression such as "one or more pieces of hardware perform process A, and the one or more pieces of hardware perform process B" (including similar expressions) is used, the hardware performing process A and the hardware performing process B may be the same or different. In other words, it is sufficient that the hardware performing process A and the hardware performing process B are included in the one or more pieces of hardware. Note that hardware may include electronic circuits, devices including electronic circuits, etc.

[0260] In this specification (including the claims), when multiple storage devices (memories) store data, each of the multiple storage devices may store only a portion of the data, or may store the entire data. Also, a configuration in which only some of the multiple storage devices store data may be included.

[0261] Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, modifications, substitutions, partial deletions, etc. are possible within the scope of the conceptual idea and spirit of the present invention, which is derived from the content defined in the claims and their equivalents. For example, when numerical values ​​or formulas are used in the above-described embodiments, they are shown for illustrative purposes and do not limit the scope of the present disclosure. Furthermore, the order of each operation shown in the embodiments is also illustrative and does not limit the scope of the present disclosure.

[0262] The disclosed technology may take the following forms as described below.

[0263] (Appendix 1) A plurality of computing units; one or more first memories connected to the plurality of computing units; one or more second memories connected to the plurality of computing units; the one or more second memories are stacked on the plurality of computing units; At least a part of the data stored in the one or more second memories is used by the computing unit without going through the one or more first memories. Computing device.

[0264] (Appendix 2) a first die including the plurality of arithmetic units and the one or more first memories; a second die comprising the one or more second memories; 10. The computing device of claim 1.

[0265] (Appendix 3) the first die and the second die are stacked on a substrate; the first die and the second die are connected to each other in a face-to-face connection manner; 10. The computing device of claim 2.

[0266] (Appendix 4) the first die and the second die are connected to each other by hybrid bonding; 4. The computing device of claim 3.

[0267] (Appendix 5) the first die and the second die are stacked on a substrate; the first die and the second die are connected to each other by a back-to-face connection method; 10. The computing device of claim 2.

[0268] (Appendix 6) Of the first die and the second die, the die farther from the substrate is connected to the die closer to the substrate via a through via provided in the die. 6. The computing device of claim 5.

[0269] (Appendix 7) the first die and the plurality of second dies are stacked on a substrate; 10. The computing device of claim 2.

[0270] (Appendix 8) The second die is one or more redundant memories; a switching circuit that disconnects a defective memory from the second die and connects the one or more redundant memories to the first die; 8. The computing device according to claim 2, wherein the first and second inputs are input to the first and second inputs.

[0271] (Appendix 9) the plurality of computing units include one or more first computing units and one or more second computing units; a first memory interface that connects at least one of the first computing units to the one or more second memories; a second memory interface that connects at least one of the second computing units to the one or more second memories; 9. The computing device according to any one of claims 1 to 8.

[0272] (Appendix 10) the first memory interface connects at least one of the first computing units to a predetermined second memory among the one or more second memories; the second memory interface connects at least one of the second computing units to another predetermined second memory among the one or more second memories; 10. The computing device of claim 9.

[0273] (Appendix 11) the one or more second memories include one or more DRAMs; 11. The computing device according to claim 1.

[0274] (Appendix 12) the one or more DRAMs are scratchpad memories of the plurality of arithmetic units; 12. The computing device of claim 11.

[0275] (Appendix 13) a plurality of second hierarchical blocks each including the plurality of arithmetic units and the one or more first memories; one or more third memories connected to the plurality of second-level blocks; the plurality of arithmetic units included in each of the plurality of second hierarchical blocks are stacked on the one or more second memories; 13. The computing device according to any one of claims 1 to 12.

[0276] (Appendix 14) a plurality of first hierarchical blocks each including two or more arithmetic units included in the plurality of arithmetic units; Each of the plurality of second hierarchical blocks comprises the plurality of first hierarchical blocks. 14. The computing device of claim 13.

[0277] (Appendix 15) the first hierarchy block includes one or more third computing units provided in common to the two or more computing units; 15. The computing device of claim 14.

[0278] (Appendix 16) a plurality of third hierarchical blocks each including the plurality of second hierarchical blocks and the one or more third memories; one or more fourth memories; 16. The computing device according to any one of Supplementary Note 13 to Supplementary Note 15.

[0279] (Appendix 17) each of the plurality of arithmetic units includes one or more fifth memories having a capacity smaller than that of the one or more first memories; 17. The computing device according to any one of claims 1 to 16.

[0280] (Appendix 18) a third die stacked on the second die and including one or more third memories; 9. The computing device according to any one of Supplementary Note 2 to Supplementary Note 8.

[0281] (Appendix 19) The computing device has a SIMD architecture. 19. The computing device according to any one of claims 1 to 18.

[0282] (Appendix 20) the one or more second memories store parameters of a machine learning model and intermediate data calculated using the machine learning model; the plurality of computing units acquire the parameters and the intermediate data from the one or more second memories, and execute a first process using the acquired parameters and intermediate data. 19. The computing device according to claim 1.

[0283] (Appendix 21) the one or more second memories store intermediate data newly generated by the execution of the first process; the plurality of computing units acquire the parameters, the intermediate data, and the newly generated intermediate data from the one or more second memories, and execute a second process using the acquired parameters, the intermediate data, and the newly generated intermediate data. 21. The computing device of claim 20.

[0284] (Appendix 22) Each of the first process and the second process includes a process of generating one or more pieces of output information using the machine learning model. 22. The computing device of claim 21.

[0285] (Appendix 23) the plurality of computing units acquiring the parameters and the intermediate data from the one or more second memories; the plurality of computing units execute processing using the acquired parameters and the intermediate data; the one or more second memories store intermediate data newly generated by the processing in the one or more second memories as part of the intermediate data; Repeatedly execute until a specified condition is met. 23. The computing device according to any one of Supplementary Note 20 to Supplementary Note 22.

[0286] (Appendix 24) The machine learning model includes at least one of a transformer or a state space model. 24. The computing device according to any one of Supplementary Note 20 to Supplementary Note 23.

[0287] (Appendix 25) the machine learning model includes the Transformer; The intermediate data includes at least a portion of key vectors and value vectors used in the attention mechanism of the Transformer. 25. The computing device of claim 24.

[0288] (Appendix 26) the machine learning model includes the state space model, the intermediate data includes at least a portion of states in the state space model; 25. The computing device of claim 24.

[0289] (Appendix 27) A computing method executed by a computing device comprising a plurality of computing units, one or more first memories connected to the plurality of computing units, and one or more second memories connected to the plurality of computing units, wherein the one or more second memories are stacked on the plurality of computing units, and at least a portion of data stored in the one or more second memories is used by the computing units without passing through the one or more first memories, the one or more second memories store parameters of a machine learning model and intermediate data calculated using the machine learning model; the plurality of computing units acquire the parameters and the intermediate data from the one or more second memories, and execute a first process using the acquired parameters and intermediate data. Calculation method.

[0290] In addition, the processing performed by the arithmetic device described in any one of Supplementary Notes 20 to 26 may be realized by the arithmetic device described in any one of Supplementary Notes 1 to 19 executing instructions generated by a compiler device. [Explanation of symbols]

[0291] 30 Main memory 40 Auxiliary storage 50 Network Interface 60 Device Interfaces 100, 101 Arithmetic device 102 Main storage 200 calculator 300 Host 500 computers 510 Bus 600 Communication Network 710, 720 External device AD address signal CLK Clock signal CMD Command signal CNTL Control Signal DB Data Bus DSYNC Data Sync Signal GRF0, GRF1 registers HOSTM Host Memory IALU integer arithmetic unit L1B, L2B blocks L1BM, L2BM memory L1D L1 data cache L2D L2 data cache L3D L3 data cache LD Logic Die LM0, LM1 local memory LP0-LP16 ports MAB Matrix Operation Block MAU matrix operator MB Memory Blocks MCNT Memory Controller MD Memory Die MI / F memory interface MP0-MP16 ports M-REG register MUX1, MUX2 multiplexers PDM Memory PE Processing Element SW switching circuit T-REG register

Claims

1. Multiple arithmetic units, One or more first memories connected to the aforementioned plurality of arithmetic units, One or more second memories connected to the aforementioned plurality of arithmetic units, Each of the following is a plurality of second-level blocks, each comprising the plurality of arithmetic units and the one or more first memories, It comprises one or more third memories connected to the plurality of second-level blocks, Each of the aforementioned plurality of second-level blocks contains the plurality of arithmetic units, which are stacked on the one or more second memories. At least a portion of the data stored in the one or more second memories is used by the arithmetic unit without going through the one or more first memories. Computing device.

2. A first die comprising the plurality of arithmetic units and the one or more first memories, A second die comprising one or more second memories, The first die and the second die are stacked on a substrate. The first die and the second die are interconnected using a face-to-face connection method. The computing device according to claim 1.

3. The first die and the second die are interconnected by hybrid bonding. The arithmetic device according to claim 2.

4. A first die comprising the plurality of arithmetic units and the one or more first memories, A second die comprising one or more second memories, The aforementioned second die is One or more redundant memory units, The system includes a switching circuit that disconnects the connection between the faulty memory among the one or more second memories and the second die, and connects the one or more redundant memories to the first die. The computing device according to claim 1.

5. The aforementioned plurality of arithmetic units include one or more first arithmetic units and one or more second arithmetic units. A first memory interface connecting at least one of the first arithmetic units to one or more of the second memories, The system comprises a second memory interface connecting at least one of the second arithmetic units to one or more of the second memories, The computing device according to claim 1.

6. The first memory interface connects at least one of the first arithmetic units to a predetermined second memory among the one or more second memories, The second memory interface connects at least one of the second arithmetic units to another predetermined second memory among the one or more second memories. The arithmetic device according to claim 5.

7. The aforementioned one or more second memories include one or more DRAMs, The one or more DRAMs mentioned above are scratchpad memories for the plurality of arithmetic units. The computing device according to claim 1.

8. Each comprises a plurality of first-level blocks, each containing two or more arithmetic units included in the plurality of arithmetic units, Each of the aforementioned plurality of second-level blocks comprises the aforementioned plurality of first-level blocks. The computing device according to claim 1.

9. The first hierarchical block comprises one or more third arithmetic units that are commonly provided to the two or more arithmetic units. The computing device according to claim 8.

10. Each of the multiple third-level blocks comprises the multiple second-level blocks and the one or more third-level memories, A fourth memory comprising one or more, The computing device according to claim 1.

11. The aforementioned computing device comprises a SIMD architecture. The computing device according to any one of claims 1 to 10.

12. The aforementioned one or more second memories store the parameters of the machine learning model and the intermediate data calculated using the machine learning model. The plurality of arithmetic units obtain the parameters and intermediate data from the one or more second memories, and use the obtained parameters and intermediate data to execute the first process. The machine learning model includes at least one of a transformer or a state-space model. The computing device according to any one of claims 1 to 10.

13. The machine learning model includes the transformer, The aforementioned intermediate data includes at least a portion of the key vectors and value vectors used in the attention mechanism of the transformer. The computing device according to claim 12.

14. The machine learning model includes the state-space model, The aforementioned intermediate data includes at least a portion of the states in the state-space model, The computing device according to claim 12.

15. The aforementioned computing device comprises a SIMD architecture. The computing device according to claim 12.

16. Multiple arithmetic units, One or more first memories connected to the aforementioned plurality of arithmetic units, It comprises one or more second memories connected to the aforementioned plurality of arithmetic units, The one or more second memories are stacked on the plurality of arithmetic units, At least a portion of the data stored in the one or more second memories is used by the arithmetic unit without going through the one or more first memories. Equipped with a SIMD architecture, Computing device.

17. A first die comprising the plurality of arithmetic units and the one or more first memories, A second die comprising one or more second memories, The first die and the second die are stacked on a substrate. The first die and the second die are interconnected using a face-to-face connection method. The computing device according to claim 16.

18. The first die and the second die are interconnected by hybrid bonding. The arithmetic device according to claim 17.

19. A first die comprising the plurality of arithmetic units and the one or more first memories, A second die comprising one or more second memories, The aforementioned second die is One or more redundant memory units, The system includes a switching circuit that disconnects the connection between the faulty memory among the one or more second memories and the second die, and connects the one or more redundant memories to the first die. The computing device according to claim 16.

20. The aforementioned plurality of arithmetic units include one or more first arithmetic units and one or more second arithmetic units. A first memory interface connecting at least one of the first arithmetic units to one or more of the second memories, The system comprises a second memory interface connecting at least one of the second arithmetic units to one or more of the second memories, The computing device according to claim 16.

21. The first memory interface connects at least one of the first arithmetic units to a predetermined second memory among the one or more second memories, The second memory interface connects at least one of the second arithmetic units to another predetermined second memory among the one or more second memories. The arithmetic device according to claim 20.

22. The aforementioned one or more second memories include one or more DRAMs, The one or more DRAMs mentioned above are scratchpad memories for the plurality of arithmetic units. The computing device according to claim 16.

23. The aforementioned one or more second memories store the parameters of the machine learning model and the intermediate data calculated using the machine learning model. The plurality of arithmetic units obtain the parameters and intermediate data from the one or more second memories, and use the obtained parameters and intermediate data to execute the first process. The machine learning model includes at least one of a transformer or a state-space model. The computing device according to any one of claims 16 to 22.

24. The machine learning model includes the transformer, The aforementioned intermediate data includes at least a portion of the key vectors and value vectors used in the attention mechanism of the transformer. The arithmetic device according to claim 23.

25. The machine learning model includes the state-space model, The aforementioned intermediate data includes at least a portion of the states in the state-space model, The arithmetic device according to claim 23.

26. Multiple arithmetic units, One or more first memories connected to the aforementioned plurality of arithmetic units, It comprises one or more second memories connected to the aforementioned plurality of arithmetic units, The one or more second memories are stacked on the plurality of arithmetic units, At least a portion of the data stored in the one or more second memories is used by the arithmetic unit without going through the one or more first memories. The aforementioned one or more second memories store the parameters of the machine learning model and the intermediate data calculated using the machine learning model. The plurality of arithmetic units obtain the parameters and intermediate data from the one or more second memories, and use the obtained parameters and intermediate data to execute the first process. The machine learning model includes at least one of a transformer or a state-space model. Computing device.

27. A first die comprising the plurality of arithmetic units and the one or more first memories, A second die comprising one or more second memories, The first die and the second die are stacked on a substrate. The first die and the second die are interconnected using a face-to-face connection method. The computing device according to claim 26.

28. The first die and the second die are interconnected by hybrid bonding. The arithmetic device according to claim 27.

29. A first die comprising the plurality of arithmetic units and the one or more first memories, A second die comprising one or more second memories, The aforementioned second die is One or more redundant memory units, The system includes a switching circuit that disconnects the connection between the faulty memory among the one or more second memories and the second die, and connects the one or more redundant memories to the first die. The computing device according to claim 26.

30. The aforementioned plurality of arithmetic units include one or more first arithmetic units and one or more second arithmetic units. A first memory interface connecting at least one of the first arithmetic units to one or more of the second memories, The system comprises a second memory interface connecting at least one of the second arithmetic units to one or more of the second memories, The computing device according to claim 26.

31. The first memory interface connects at least one of the first arithmetic units to a predetermined second memory among the one or more second memories, The second memory interface connects at least one of the second arithmetic units to another predetermined second memory among the one or more second memories. The computing device according to claim 30.

32. The aforementioned one or more second memories include one or more DRAMs, The one or more DRAMs mentioned above are scratchpad memories for the plurality of arithmetic units. The computing device according to claim 26.

33. The machine learning model includes the transformer, The aforementioned intermediate data includes at least a portion of the key vectors and value vectors used in the attention mechanism of the transformer. The computing device according to claim 26.

34. The machine learning model includes the state-space model, The aforementioned intermediate data includes at least a portion of the states in the state-space model, The computing device according to claim 26.

35. The system comprises a plurality of arithmetic units, one or more first memories connected to the plurality of arithmetic units, and one or more second memories connected to the plurality of arithmetic units, wherein the one or more second memories are stacked on the plurality of arithmetic units, and at least a portion of the data stored in the one or more second memories is an arithmetic method executed by an arithmetic device used by the arithmetic units without going through the one or more first memories. The aforementioned one or more second memories store the parameters of the machine learning model and the intermediate data calculated using the machine learning model. The plurality of arithmetic units obtain the parameters and intermediate data from the one or more second memories, and use the obtained parameters and intermediate data to execute the first process. The machine learning model includes at least one of a transformer or a state-space model. Calculation method.

36. The one or more second memories mentioned above store the intermediate data newly generated by the execution of the first process. The plurality of arithmetic units retrieve the parameters, the intermediate data, and the newly generated intermediate data from the one or more second memories, and use the retrieved parameters, the intermediate data, and the newly generated intermediate data to execute a second process. The calculation method according to claim 35.

37. Each of the first and second processes includes a process for generating one or more output pieces of information using the machine learning model. The calculation method according to claim 36.

38. The plurality of arithmetic units obtain the parameters and the intermediate data from the one or more second memories, The plurality of arithmetic units perform processing using the acquired parameters and the intermediate data, The one or more second memories store the intermediate data newly generated by the process as part of the intermediate data in the one or more second memories. This is repeated until the specified conditions are met. The calculation method according to any one of claims 35 to 37.

39. The machine learning model includes the transformer, The aforementioned intermediate data includes at least a portion of the key vectors and value vectors used in the attention mechanism of the transformer. The calculation method according to claim 38.

40. The machine learning model includes the state-space model, The aforementioned intermediate data includes at least a portion of the states in the state-space model, The calculation method according to claim 38.

Images