Flexible address mapping method and apparatus for utilizing in-processing memory

Abstract

To provide a flexible address mapping method and apparatus for utilizing processing-in-memory. [Solution] A method for performing a matrix operation involving multiple parameters using a system-on-chip (SoC) processor and a processing-in-memory (PIM) processor in memory, comprising the steps of: calculating a map ID based on information about the matrix, at least one of information about the memory and information about the processing-in-memory (PIM) processor in memory; allocating a huge page to represent a portion of the multiple parameters; and adding the map ID to a page table entry of the huge page for the matrix operation in the system-on-chip (SoC) processor.

Description

Technical Field

[0001] The present invention relates to a flexible address mapping method for utilizing processing-in-memory and an apparatus therefor, and more specifically, to a flexible address mapping technique for on-device large language model inference utilizing processing-in-memory.

[0002] Meanwhile, the application of the present invention was conducted with the support of the following South Korean research and development project.

[0003] Project Unique Number: 2710004147 Project Number: 00340008 Department Name: Ministry of Science and ICT Project Management (Specialized) Institution Name: National Research Foundation of Korea Research Project Name: Individual Basic Research (Ministry of Science and ICT) Research Subject Name: Chiplet-based Integrated Accelerator Architecture for Flexible HPC / AI Computing Project Performing Institution Name: Seoul National University, Republic of Korea Research Period: May 1, 2024 ~ April 30, 2025 Project Unique Number: 2710018731 Project Number: 00405857 Department Name: Ministry of Science and ICT Project Management (Specialized) Institution Name: National Research Foundation of Korea Research Project Name: Collective Research Support Research Subject Name: Sustainable Generative Artificial Intelligence Computing Platform Research Laboratory Project Performing Institution Name: Seoul National University, Republic of Korea Research Period: August 1, 2024 ~ July 31, 2025 Project Unique Number: 2710018171 Project Number: 00456287 Department Name: Ministry of Science and ICT Project Management (Specialized) Institution Name: Institute for Information & Communications Technology Planning & Evaluation Research Project Name: Global Talent Development in the Digital Field (R&D) Research Project Title: Integrated Design of Accelerator, Network, Memory, Storage Hardware, and System Software for Training and Inference of Large-Scale AI Models Organization responsible for carrying out the project: Seoul National University Industry-Academia Cooperation Group Research period: July 1, 2024 ~ December 31, 2024 [Background technology]

[0004] Large language model (LLM) inference on a wide variety of edge devices, including mobile phones and laptop computers, is becoming increasingly important. Such "on-device" inference offers numerous advantages, including protection of sensitive privacy information stored on the device, faster response times compared to servers, and personalization of the model. Because on-device LLM inference processes only the individual requests of the device user, most of the computation consists of general matrix-vector multiplication (GEMV), where memory bandwidth becomes a bottleneck.

[0005] Processing-in-memory (PIM) is a method that utilizes the wide bandwidth within memory by integrating arithmetic units into the memory itself, and has recently gained attention as a solution that can overcome memory bandwidth bottlenecks. For example, "near-bank" DRAM PIM, which integrates arithmetic units into each bank of dynamic random access memory (DRAM), is considered highly likely to be implemented in actual devices, and various prototypes and commercialized products support this assessment.

[0006] Ideally, PIM should be implemented on edge devices to accelerate on-device LLM inference, where memory is the primary bottleneck. However, this presents a problem: existing SoC (system-on-chip) processors, such as graphics processing units (GPUs) or neural processing units (NPUs), and the PIM processor must share model parameters. The data mapping methods used by SoC processors and those used for PIM calculations are different. Due to these differences in mapping methods, existing methods have employed redundant storage in memory of both the original parameters and copies mapped using different methods in order to utilize PIM. However, this method is unsuitable for edge devices with limited DRAM space because it redundantly uses memory space.

[0007] Another proposed alternative is to dynamically transform and store copies of the mapped parameters. For example, only the original parameters could be stored, and copies of the parameters could be dynamically mapped, transformed, and stored using other methods as needed. However, this would increase both the response time and processing time of the processing (e.g., LLM inference processing) because the mappings are transformed and stored dynamically each time.

[0008] Ultimately, in order for SoC processors to perform complex calculations using PIM, existing methods have included (i) storing duplicate copies of parameters mapped to other types in memory, or (ii) dynamically converting data mappings. However, these methods have the problem of incurring costs in terms of storage space and / or processing time. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] Republic of Korea Registered Patent No. 10-2695927 Gazette [Overview of the Initiative] [Problems that the invention aims to solve]

[0010] The objective of the present invention is to solve the problem that when an SoC processor and a PIM processor must access the same matrix parameter data to accelerate on-device LLM inference using PIM, they require different memory configurations.

[0011] According to embodiments of the present invention, the objective is to provide a system that combines practicality and scalability by proposing flexible DRAM address mapping in units of huge pages and the necessary changes to the memory system for this purpose.

[0012] Another objective of the present invention is to efficiently implement matrix operations used in LLM inference using flexible DRAM address mapping, and to apply it to optimizing a wide variety of data arrangements that can be represented by flexible address mapping, so that it can be extended to applications other than LLM inference.

[0013] On the other hand, the problems that the present invention aims to solve or the objectives of the present invention are not limited in any way to those described above, and the matters described above should be understood as illustrative examples. [Means for solving the problem]

[0014] According to an embodiment of the present invention, there is provided a method for performing a matrix operation including a plurality of parameters by a system on chip (SoC) processor and a processing in memory (PIM) processor in a memory, the method may include: calculating a map ID based on at least one of information on the matrix, information on the memory, and information on the processing in memory (PIM) processor in the memory; allocating a huge page for indicating a part of the plurality of parameters; and adding the map ID to a page table entry of the huge page for the matrix operation in the system on chip (SoC) processor.

[0015] Further, the information on the matrix may include information on dimensions of the matrix, and the map ID may be a value representing a degree to which a bit controlling bank interleaving in the memory is separated from a chunk column.

[0016] Furthermore, the step of calculating the map ID may be performed by a mapping selector, and the step of allocating the huge page and the step of adding the map ID to the page table entry of the huge page may be performed by a memory allocator.

[0017] Still further, before the step of calculating the map ID, the method may further include a step of passing a memory allocation request by a user program to the mapping selector together with the information on the matrix.

[0018] Still further, after the step of adding the map ID to the page table entry of the huge page, the method may further include a step of returning a virtual address for the huge page allocated by the memory allocator to the user program.

[0019] Furthermore, the matrix operations may include matrix-matrix multiplication and matrix-vector multiplication, the matrix-matrix multiplication may be performed by the system-on-chip (SoC) processor, and the matrix-vector multiplication may be performed by the processing-in-memory (PIM) processor.

[0020] Furthermore, the matrix operations may be matrix operations for weight values used in inference of a large language model (LLM).

[0021] Furthermore, the large language model (LLM) inference may include a prefill process and a decode process, the prefill process may include the matrix-matrix multiplication performed by the system-on-chip (SoC) processor, and the decode process may include the matrix-vector multiplication performed by the processing-in-memory (PIM) processor.

[0022] Furthermore, the large language model (LLM) inference may be performed by a single edge device.

[0023] Furthermore, the information related to the matrix may include at least one of information related to the dimension of the matrix and information related to the data type of the matrix.

[0024] Furthermore, the information related to the memory may include at least one of information related to the huge page size of the memory, information related to the number of channels of the memory, information related to the number of ranks of the memory, and information related to the number of banks of the memory.

[0025] Furthermore, the information related to the processing-in-memory (PIM) processor may include information related to a chunk column.

[0026] Furthermore, the map ID may be determined differently depending on whether partitioning is required, by comparing the size of the huge page and the row size of the matrix to the total number of banks of the memory.

[0027] Furthermore, the process may further include, after the step of adding the map ID to the page table entry of the huge page, receiving a virtual address from a user program for storing or loading the matrix; determining a physical address corresponding to the virtual address; and determining the DRAM address of the memory corresponding to the physical address based on the physical address and the map ID.

[0028] Furthermore, the step of determining the DRAM address of the memory corresponding to the physical address is performed by the memory controller, the multiplexer of the memory controller receives the map ID as input, and the memory controller may have a register for storing the map ID.

[0029] In addition to these, the mapping between the DRAM address of the memory and the physical address may be abstracted into the user program.

[0030] According to one embodiment of the present invention, a computing device is provided, the computing device comprising a system-on-chip (SoC) processor and a memory having a processing-in-memory (PIM) processor, and when a command relating to a matrix operation stored in the memory is performed by the system-on-chip (SoC) processor and the PIM processor, the computing device may perform the following operations: calculate a map ID based on information about the matrix, information about the memory, and information about the processing-in-memory (PIM) processor; allocate a huge page to indicate a portion of the plurality of parameters; and add the map ID to a page table entry of the huge page for the matrix operation in the system-on-chip (SoC) processor.

[0031] Furthermore, according to one embodiment of the present invention, a computer program is provided which is stored on a computer-readable medium, and when performed by a computing device, the computer program performs a matrix operation involving a plurality of parameters by a system-on-chip (SoC) processor and a processing-in-memory (PIM) processor in memory, and the operation may include calculating a map ID based on information about the matrix, information about the memory, and information about the processing-in-memory (PIM) processor, allocating a huge page to represent a portion of the plurality of parameters, and adding the map ID to a page table entry of the huge page for the matrix operation in the system-on-chip (SoC) processor. [Effects of the Invention]

[0032] According to embodiments of the present invention, by calculating a map ID for DRAM address mapping for data sharing between the SoC processor and the PIM processor, it becomes possible to use a new address mapping instead of an existing mapping for the huge page storing the shared data.

[0033] According to embodiments of the present invention, unlike existing research that designs PIM from accelerators alone, a novel method for integrating PIM into existing systems can be presented, and in particular, LLM inference, a core application of on-device AI, can be improved.

[0034] According to embodiments of the present invention, only minimal system and hardware modifications are required to enable application to edge devices. For example, in the case of an operating system, only a modification is needed to allow mapping information to be stored in the available space of existing page table entries, and in the case of a memory controller, only simple logic can be added to receive the mapping information along with the address and to enable the use of other address mapping techniques based on the mapping information.

[0035] According to embodiments of the present invention, edge device manufacturers can adopt the mapping method presented herein without extensive system modifications. In particular, for mobile phone manufacturers that are actively developing and releasing on-device LLM services, this method can not only overcome the difficulty of adopting PIM but also improve the quality of LLM services by incorporating PIM.

[0036] On the other hand, the effects obtained by the present invention are not limited in any way to those described above, and other technical effects not mentioned should be clearly understandable to an ordinary engineer from the description below. [Brief explanation of the drawing]

[0037] [Figure 1]This is a configuration diagram of a device comprising an external processor to the memory and an internal processor to the memory according to one embodiment of the present invention. [Figure 2] This diagram shows an example of how the Transformer Layer works. [Figure 3] This figure shows an example of an LLM inference process that is divided into a prefill phase and a decode phase. [Figure 4] This graph shows the degree to which LLM inference time can be reduced by utilizing PIM. [Figure 5] This diagram illustrates an example of a configuration in which matrix parameters are placed on DRAM for calculations performed by a PIM processor. [Figure 6] This graph shows the response time delay caused when rearranging data used by the PIM processor and the SoC processor. [Figure 7] This is a flowchart showing a method for performing matrix operations involving multiple parameters using an SoC processor and a PIM processor in memory according to one embodiment of the present invention. [Figure 8] This is a flowchart showing a method for performing matrix operations involving multiple parameters using an SoC processor and a PIM processor in memory according to one embodiment of the present invention. [Figure 9] This block diagram schematically shows the process of performing matrix operations involving multiple parameters using an SoC processor and a PIM processor in memory according to one embodiment of the present invention. [Figure 10] This is program code illustrating an example implementation of an algorithm for calculating a map ID according to an embodiment of the present invention. [Figure 11] This figure shows optimized DRAM address mappings for two exemplary PIM architectures. [Figure 12] This figure shows an example implementation of modifying a page table entry to pass the map ID to the memory controller. [Figure 13]This figure shows a modified memory controller required to change DRAM address mappings based on map IDs. [Figure 14] This figure shows the technical advantages of the flexible address mapping method according to an embodiment of the present invention compared to existing technologies. [Figure 15] This figure shows the technical advantages of the flexible address mapping method according to an embodiment of the present invention compared to existing technologies. [Figure 16] This figure shows the technical advantages of the flexible address mapping method according to an embodiment of the present invention compared to existing technologies. [Modes for carrying out the invention]

[0038] The objectives, technical configuration, and associated effects of the present invention will be more clearly understood by the following detailed description based on the drawings attached to the specification of the present invention. Hereinafter, embodiments of the present invention will be described in detail based on the attached drawings.

[0039] The embodiments disclosed herein should not be construed as limiting the scope of the invention. It goes without saying that the descriptions herein, including the embodiments, have various applications for a person ordinary in the art. Therefore, any embodiments described in the detailed description section are merely illustrative examples for the purpose of further illustrating the invention, and are not intended to limit the scope of the invention to these embodiments.

[0040] The functional blocks shown in the drawings and described below are merely examples of feasible implementations. Other functional blocks may be used in other embodiments, without departing from the ideas and scope of the detailed description. Although one or more functional blocks of the present invention are shown as individual blocks, one or more of the functional blocks of the present invention may be combinations of various hardware and software components that perform the same function.

[0041] Furthermore, unless otherwise specified, the words and phrases "to possess," "to have," "to include," and "to encompass" a certain component are to be interpreted as open-ended terms (i.e., "to include but not to limit"), merely indicating the existence of that component and not being understood as excluding further components.

[0042] Furthermore, whereever it is stated in this disclosure that one component is “linked” or “connected” to another component, it should be understood that the first component is directly linked or connected to the other component, or that there may be other components between them.

[0043] Various embodiments of the present invention are described below with reference to the accompanying drawings. However, these should not be understood as limiting the scope to any particular embodiment of the present invention, but rather as including various modifications, equivalents, and / or alternatives to embodiments of the present invention.

[0044] Figure 1 is a configuration diagram of a device 100 equipped with an external processor (e.g., SoC processor) and an internal processor (e.g., PIM processor) of the memory according to one embodiment.

[0045] Referring to Figure 1, the device 100 according to one embodiment may include a memory 110, a processor 120, an input / output interface 130, and a communication interface 140. The device 100 may also be a computing device.

[0046] The memory 110 may include a PIM processor 112. The PIM processor within the memory 110 is distinct from the external processor 120 shown in Figure 1.

[0047] The memory 110 may be, for example, DRAM, but this is merely illustrative, and the content of the present invention is not limited in any way to it. However, for the sake of ease of explanation, the following description will proceed assuming that the memory 110 in this specification is DRAM.

[0048] Memory 110 may store data acquired from an external device or data generated by itself. Memory 110 may also store commands / data that cause the processor 120 to operate. Furthermore, memory 110 may store commands / data that cause its own PIM processor 112 to operate. For example, memory 110 may store various information for realizing a method of performing matrix operations including multiple parameters using the SoC processor and the PIM processor in memory, as described later.

[0049] The processor 120 is an arithmetic unit that controls the overall operation. The processor 120 may execute commands stored in the memory 110. The processor 120 may be a single-core processor or a multi-core processor containing multiple cores. Alternatively, the processor 120 may be an SoC processor.

[0050] The input / output interface 130 may include a hardware interface or a software interface for inputting or outputting information.

[0051] The communication interface 140 enables the transmission and reception of information over a communication network. For this purpose, the communication interface 140 may include a wireless communication module or a wired communication module.

[0052] The device 100 may be implemented by a wide variety of devices capable of performing calculations via the processor 120 and / or PIM processor 112 and sending and receiving information via a network. For example, it can be implemented in the form of a server, computer device, portable communication device, smartphone, portable multimedia device, notebook PC, tablet PC, etc., but the present invention is not limited to such examples.

[0053] Figure 2 is a diagram illustrating an example of the transformer layer mechanism, and Figure 3 is a diagram illustrating an example of an LLM inference process that is divided into a prefill phase and a decode phase.

[0054] Most modern large-scale language model (LLM) inference can commonly consist of multiple transformer decoder layers. As shown in Figure 2, in the mechanism of the transformer layer, the core operations of the transformer decoder layer may include linear operations and self-attention operations. Of these, linear operations account for a significant portion of the on-device inference time. In on-device inference, only requests from the user of the device are processed, but the linear operations in on-device LLM inference that only process requests from the device user have different characteristics depending on the phase of the LLM inference process.

[0055] As shown in Figure 3, LLM inference is divided into a prefill phase, which processes multiple tokens at once, and a decode phase, which generates a single token at once. In the former case, matrix multiplication (GEMM: general matrix-matrix multiplication) is performed, and in the latter case, matrix multiplication (GEMV: general matrix-vector multiplication) is performed. The prefill process is performed only once for each request, but the decode process is performed repeatedly for each token generated, so the decode phase accounts for most of the total inference time. Therefore, matrix-vector multiplication can be said to be the core operation that determines the performance of LLM inference.

[0056] Matrix-vector multiplication is a low-intensity operation with minimal data reuse, making the device's memory bandwidth a bottleneck. Therefore, processing such operations with an external SoC processor (e.g., processor 120) outside of the memory (e.g., memory 110) incurs enormous costs for sending and receiving data, leading to a decrease in overall computational efficiency. To mitigate these drawbacks, an internal processor (e.g., PIM processor 112) within the memory may be utilized. PIM is a technique that enables the use of internal bandwidth by integrating arithmetic units within the memory. For example, among the various types of PIM, "near-bank" DRAM PIM, which places an arithmetic unit for each DRAM bank, can be considered. Near-bank DRAM PIM is considered to be the closest to commercialization.

[0057] Figure 4 is a graph showing the degree to which LLM inference time can be reduced by utilizing PIM.

[0058] As can be clearly seen from Figure 4, by utilizing a PIM processor, it is possible to accelerate memory bottleneck calculations that were difficult to accelerate with existing SoC processors alone, thereby significantly reducing the time required for inference. For example, as shown in Figure 4, comparing the case of performing LLM inference using a GPU as the SoC processor 401, the case of performing LLM inference using an ideal NPU 402, and the case of performing LLM inference using a GPU (a type of SoC processor) and a PIM processor in combination 403, it can be seen that the execution speed decreases as we move towards the latter. Furthermore, it can be confirmed that the factor that affects such execution speed is the decoding process rather than the prefill process.

[0059] Figure 5 is an illustrative diagram showing a configuration in which matrix parameters are placed on DRAM for calculations performed by a PIM processor, and Figure 6 is a graph showing the response time delay caused when rearranging data used by the PIM processor and the SoC processor.

[0060] As can be clearly seen from Figure 5, in general, in order to perform matrix and vector operations using PIM, the matrix parameters used for LLM inference must be placed inside the DRAM in a specific form. For example, this may differ from the DRAM address mapping commonly used by SoC processors. For example, some parameters 501a, 502a, and 503a of the weight matrix may be stored in memory such as DRAM based on a specific pattern. For example, some parameters 501b and 502b may be stored in bank 0, while other parameters 503b may be stored in bank 1.

[0061] LLM inference requires not only matrix multiplication with matrix operations that are computationally intensive and favorably handled by the SoC processor, but also matrix multiplication with vector operations that are computationally intensive and favorably handled by the PIM processor, for the same LLM parameters (e.g., parameters included in the weighting matrix). Therefore, performing LLM inference on heterogeneous platforms consisting of SoC processors and PIM processors may require considering what hardware is performing the operation at the time a particular operation is performed (e.g., the SoC processor or the PIM processor) and converting the data layout accordingly.

[0062] On the other hand, as shown in Figure 6, performing the aforementioned data rearrangement operation presents a problem in that it causes delays in response time, such as TTFT.

[0063] Data rearrangement may involve, for example, storing data in memory in a format suitable for the PIM processor, and then dynamically rearranging the data in a prefill step to store it in a format suitable for the SoC processor. However, in this case, the cost of data rearrangement extends the time it takes for the first token to be generated, i.e., the TTFT (time-to-first-token), which significantly hinders the inference response.

[0064] In on-device LLM inference, TTFT is known to be a more important metric that impacts the user experience than the total time taken for inference. This is because the rate at which the LLM generates tokens is generally faster than the rate at which the user can read or hear the results generated by the LLM. In other words, the user's perception is more sensitive to the time it takes for the first word to appear than to the total time it takes for inference to finish.

[0065] Therefore, embodiments of the present invention provide a flexible address mapping method that allows both the SoC processor and the PIM processor to check multiple parameters included in matrix operations without performing such data rearrangement.

[0066] Hereinafter, exemplary embodiments of the method of the present invention will be described with reference to Figures 7 and 8. The steps disclosed in Figures 7 and 8 are merely preferred embodiments for achieving the objectives of the present invention, and it goes without saying that some steps may be added or omitted as needed, and any one step may be included in any of the others. The order of the operations disclosed in Figures 7 and 8 is merely an order arranged for the sake of understanding, and such an order is not limited to a chronological order, and the order may be changed and the operations performed at the designer's discretion.

[0067] Figure 7 is a flowchart illustrating a method 700 for performing matrix operations involving multiple parameters using an SoC processor and a PIM processor in memory according to one embodiment of the present invention. For example, Figure 7 is a flowchart of the operation performed in conjunction with the memory 110 by the SoC processor 120 and PIM processor 112 of a device 100 according to one embodiment of the present invention shown in Figure 1.

[0068] In step S710, the map ID may be calculated based on information about the matrix, and at least one of the information about the memory and the information about the PIM processor. In one embodiment, the information about the matrix may include information about the dimension of the matrix. The map ID may also be a value that represents the degree to which the bits governing bank interleaving in memory are separated from the chunk column.

[0069] In step S720, a huge page may be allocated to represent some of the multiple parameters.

[0070] In step S730, a map ID may be added to the page table entry of the huge page for matrix operations in the SoC processor.

[0071] The step of calculating the map ID (S710) may be performed by a mapping selector, and the steps of allocating a huge page (S720) and adding the map ID to the page table entry of the huge page (S730) may be performed by a memory allocator. The specific operation of this will be described later with reference to Figure 9.

[0072] On the other hand, although not shown in Figure 7, method 700 may further include a step in which a memory allocation request from a user program is passed to the mapping selector along with information about the matrix, prior to the step of calculating the map ID (S710). For example, the user program may pass the memory allocation request to the mapping selector using a PIM memory allocation function such as pimalloc(dim, dtype).

[0073] Although not shown in Figure 7, method 700 may further include, after the step of adding a map ID to the page table entry of the huge page (S730), the step of returning to the user program a virtual address for the huge page allocated by the memory allocator. Such a virtual address can be used in the mapping process between virtual address VA, physical address PA, and DRAM address DA in the exemplary embodiment shown in Figure 8, which will be described later.

[0074] Figure 8 is a flowchart illustrating a method 800 for performing matrix operations involving multiple parameters using an SoC processor and a PIM processor in memory according to one embodiment of the present invention. For example, Figure 8 is a flowchart of the operation performed in conjunction with the memory 110 by the SoC processor 120 and PIM processor 112 of a device 100 according to one embodiment of the present invention, as shown in Figure 1. The method 800 in Figure 8 may be performed after the method 700 in Figure 7.

[0075] In step S810, after the step of adding a map ID to the page table entry of the huge page (S730), a virtual address for storing or loading a matrix may be received from the user program.

[0076] In step S820, the physical address corresponding to the virtual address may be determined.

[0077] In step S830, the DRAM address of the memory corresponding to the physical address may be determined based on the physical address and the map ID.

[0078] Figure 9 is a schematic block diagram showing the process of performing matrix operations involving multiple parameters by an SoC processor and a PIM processor in memory according to one embodiment of the present invention. For example, Figure 9(a) corresponds to the embodiment in Figure 7, and Figures 9(b) and (c) correspond to the embodiment in Figure 8.

[0079] As shown in Figure 9, the data allocation process may follow the flow shown in Figure 9(a). Once the dimensions of the matrix that the SoC processor and the PIM processor must share and the microarchitecture of the PIM processor are identified, a mapping selector may calculate a MapID for that matrix. Details of an exemplary algorithm for determining the MapID are described later with reference to Figure 10.

[0080] The mapping selector requests the operating system to allocate memory in units of huge pages, and the modified memory allocation function (e.g., pimalloc) may add a map ID to the page table entry (PTE) of the huge page. Details of the process of allocating huge pages and adding a map ID to the page table entry will be described later with reference to Figures 11 and 12.

[0081] For programs running on the SoC processor, changes in DRAM address mapping are abstracted and invisible, as shown in Figures 9(b) and (c). Therefore, user programs access data from the virtual address (VA) layer, just as before, and existing user programs can run as is without considering the conversion between the underlying physical address (PA) and DRAM address (DA).

[0082] For example, when a user program requests to store the parameters of a weighted value matrix (e.g., Figure 9(b)) or to load it (e.g., Figure 9(c)), both of these requests begin by passing the virtual address VA (corresponding to S810 in Figure 8). The physical address PA corresponding to the virtual address VA can be determined using the page table (corresponding to S820 in Figure 8), and the DRAM address DA in memory can be determined based on the physical address VA and the MapID (corresponding to S830 in Figure 8).

[0083] Figure 10 shows program code illustrating an example of an algorithm for calculating a map ID according to an embodiment of the present invention.

[0084] The matrix_config information may include at least one of the following: matrix_config→dim information regarding the matrix dimensions and matrix_config→dtype information regarding the matrix data type.

[0085] The memory information memory_config may include at least one of the following: information about the huge page size of memory memory_config→hpage_size, information about the number of memory channels memory_config→n_ch, information about the number of memory ranks memory_config→n_rank, and information about the number of memory banks memory_config→n_bank.

[0086] The PIM processor information file, pim_config, may also include information about chunk columns, pim_config→chunk_col.

[0087] The map ID `map_id` may be determined differently depending on whether partitioning is required, by comparing the size of the huge pages (`hpage_size / total_bank_count`) relative to the total number of memory banks and the row size of the matrix (`row_size`). For example, the map ID `map_id` may be determined differently depending on the conditional statements shown in lines 25-27 of Figure 10.

[0088] Figure 11 shows optimized DRAM address mappings for two exemplary PIM architectures. For example, Figure 11(a) is an example for the AiM architecture, and Figure 11(b) is an example for the HBM-PIM architecture.

[0089] Referring to Figure 11, the process of determining the DRAM address mapping that arranges the LLM parameters included in the weighted value matrix in a way that is suitable for the PIM processor's calculations during the LLM inference process, and representing the type of mapping by a map ID, is shown.

[0090] As described above, under the constraint of not modifying the program running on the SoC processor, the DRAM address mapping pattern that achieves an optimized matrix arrangement for calculations by the PIM processor can be identified and represented by a MapID. As can be clearly seen from Figure 11, the optimized DRAM address mapping will be influenced by the PIM microarchitecture and the matrix dimensions, and may be generalized and represented by a MapID for various cases. When the LLM model is first uploaded to memory, the mapping selector according to the embodiment of the present invention will identify the DRAM address mapping required by each matrix based on the PIM architecture and the dimensions of the LLM parameter matrix, and this may be represented in the form of a MapID.

[0091] Figure 12 shows an example implementation of modifying the page table entry to pass the map ID to the memory controller. As mentioned above, memory allocation can be performed on a huge page basis for the matrix LLM parameters, and the map ID can be added to the huge page and passed to the memory controller.

[0092] Since the SoC processor can perform a wide variety of calculations in addition to LLM inference, the DRAM address mapping may be changed only for LLM parameters shared by the SoC processor and the PIM processor.

[0093] In edge devices such as smartphones and laptop computers, the channel, rank, bank, and column bits that determine the location within DRAM may all be present within the 21 bits that make up a single 2MB huge page. Therefore, by allocating memory for LLM parameters using huge pages, it is possible to change the DRAM address mapping locally without causing significant changes, while still maintaining the overall DRAM address mapping of the device.

[0094] To pass the MapID determined by the mapping selector to the memory controller, the operating system's memory allocation function can be modified by adding the MapID to the remaining area of ​​unused page table entries (PTEs) through the use of huge pages, as shown in Figure 12.

[0095] Figure 13 shows a modified memory controller required to change the DRAM address mapping based on the map ID.

[0096] As shown in Figure 13, changing the DRAM address mapping on a per-huge-page basis can be achieved by simply modifying an existing memory controller. In the memory controller, a hardware module that allows other mappings based on the MapID can be added to the PA-to-DA mapping area that changes physical addresses PA to DRAM addresses DA, as shown in Figure 13.

[0097] Thus, the memory controller can realize embodiments of the present invention with minimal modifications while retaining its existing form. For example, such modifications require only the addition of a multiplexer and registers, and can be implemented with little increase in the area and power consumption of the memory controller. In other words, the step of determining the DRAM address of the memory corresponding to the physical address is performed by the memory controller, the memory controller's multiplexer receives the map ID as input, and the memory controller has registers for storing the map ID, so that the rest of the existing form of the memory controller can be retained.

[0098] According to embodiments of the present invention, the matrix operation may be a weighted matrix operation used in LLM inference, and may include matrix multiplication and matrix-vector multiplication. According to the flexible address mapping method using map ID according to the embodiments of the present invention described above, matrix multiplication may be performed by the SoC processor, and matrix-vector multiplication may be performed by the PIM processor.

[0099] Matrix multiplication corresponds to the prefill process, and matrix multiplication corresponds to the decode process. In other words, LLM inference may include both a prefill process and a decode process, but the prefill process may include matrix multiplication performed by the SoC processor, and the decode process may include matrix multiplication performed by the PIM processor.

[0100] Thus, embodiments of the present invention assign LLM parameters that must be shared by two different types of processors to huge pages, and use PIM-optimized address mapping for these huge pages instead of the SoC processor's existing DRAM address mapping. Through this, the SoC processor can access the LLM parameters without the cost of data relocation and without any additional program modifications. Furthermore, the PIM processor can also maximize the acceleration of computations where memory is a bottleneck because the data is arranged in a form optimized for the PIM architecture.

[0101] Figures 14 to 16 show the technical advantages of the flexible address mapping method according to the embodiment of the present invention compared to existing technologies.

[0102] Figure 14(a) abstractly represents a method in which the same data is stored redundantly for each mapping. As shown in Figure 14(a), the parameters of the weighting matrix used in the user program differ for each channel CH0, CH1 and each bank Bank 0, Bank 1, for the data placement pattern SoC Opt. optimized for the SoC processor and the data placement pattern PIM Opt. optimized for the PIM processor. Because these different data placement patterns are stored redundantly, there is a limitation in that the method shown in Figure 14(a) is difficult to adopt when performing on-device LLM inference.

[0103] Figure 14(b) abstractly represents a method for dynamically rearranging data. In the mechanism shown in Figure 14(b), although data is not continuously stored redundantly, there are costs associated with dynamically changing the data layout. Consequently, this also increases both the inference response time and processing time.

[0104] Figure 14(c) shows that such a method overcomes the shortcomings of existing technologies by using a flexible address mapping method according to an embodiment of the present invention. The flexible address mapping method according to an embodiment of the present invention is also called Flexible Address Mapping for Cooperative Inference (FACIL).

[0105] Figure 15 is a graph showing the improvement in LLM inference response time (TTFT) compared to the baseline, and Figure 16 is a graph showing the improvement in LLM inference processing time compared to the baseline.

[0106] Here, we evaluated the performance of four edge devices with distinct characteristics. Specifically, we measured performance using actual device measurements and a PIM simulator, and verified the effectiveness of the invention by measuring the response time and processing time of LLM inference on the Alpaca dataset and the RealHumanEval dataset, which can be considered examples of a virtual secretary and automated completion using LLM, respectively.

[0107] As can be clearly seen from Figure 15, the embodiments of the present invention achieve more than a twofold reduction in response time compared to the SoC-PIM baseline. The graph labeled "FACIL" in Figure 15 is a graph for the embodiments of the present invention. Compared to the baseline labeled "Hybrid (Static)", the response time per request, as measured by TTFT, is reduced by an average of 2.37 times and 2.63 times for each dataset.

[0108] In LLM services where users and devices interact, such as chatbots and virtual assistants, response time has the greatest impact on the user experience. This is because the rate at which LLMs generate tokens is generally faster than the rate at which users can read or listen to the generated results. Therefore, in such cases, response time, represented by TTFT, can be a more crucial performance metric for on-device inference than overall processing time.

[0109] Furthermore, as shown in Figure 16, the embodiments of the present invention demonstrate approximately 20% reduction in processing time compared to the SoC-PIM baseline, and reaffirm the effectiveness of PIM. For example, the embodiments of the present invention can reduce the processing time per query by approximately 20% in both datasets. In addition, by achieving a processing time reduction of more than 3.5 times compared to using only the SoC processor, the effectiveness of PIM in accelerating on-device LLM inference can be reaffirmed.

[0110] As described above, embodiments of the present invention propose a method for enabling an SoC processor and a PIM processor to share LLM parameters without additional cost. For example, a method is proposed in which data shared by hardware of different types is allocated to huge pages and the DRAM address mapping is converted in that unit. Through this, calculations performed in the SoC processor can be performed without additional program modifications, and calculations that are memory bottlenecks can be accelerated by the PIM. Since there is no cost for rearranging the data mapping to convert LLM parameters, both the response time and processing time of LLM inference can be reduced compared to existing technologies.

[0111] In particular, the effectiveness of the embodiments of the present invention is even more pronounced when LLM inference is performed on a single edge device. On-device LLM inference has several advantages compared to server-based LLM inference because it can be utilized within a single device without transmitting personal information externally. The release of on-device LLM services from various manufacturers, such as Apple Intelligence from Apple Inc. and Samsung Galaxy AI from Samsung Electronics of South Korea, corroborates this importance.

[0112] Edge devices such as mobile phones and laptops have limited computing and memory resources due to their limited form factor, and therefore generally take longer to perform LLM inference compared to servers. Consequently, applying PIM to improve the memory bottleneck that most computations face can not only significantly improve the user experience but also yield further benefits such as reduced battery consumption.

[0113] As a result, according to embodiments of the present invention, the core problems that arise when applying PIM to edge devices to perform LLM inference can be solved, and LLM inference can be accelerated using PIM without any additional costs, which can lead to substantial improvements in user experience and performance on billions of edge devices used by people all over the world, including mobile phones and laptop computers.

[0114] These improvements are even more pronounced when attempting to accelerate on-device LLM inference, which forms the core of numerous services such as chatbots, voice assistants, and auto-completion, using PIM.

[0115] The above description has focused on LLM inference when applying embodiments of the present invention. However, this is merely illustrative, and it goes without saying that the invention is generally applicable not only to LLM but also to other applications where a PIM-suitable mapping can be achieved by changing the DRAM address mapping.

[0116] Furthermore, while the embodiments of the present invention focus on matrix-vector multiplication, which is the core operation of LLM inference, they can also be applied to other operations that can achieve a data arrangement suitable for PIM operations using DRAM address mapping. Needless to say, they can also be applied with minimal modifications to other applications that utilize matrix-vector multiplication.

[0117] It goes without saying that the various embodiments of the present invention described above can be realized individually or in combination with other embodiments.

[0118] The various embodiments and terms used herein are not intended to limit the technical features described herein to any particular embodiment, but should be understood to include a wide variety of modifications, equivalents, or substitutions of such embodiments. In connection with the description of the drawings, similar or related components may be denoted by similar reference numerals. The singular form of a noun corresponding to an item may include one or more of such items unless otherwise specifically noted herein or if it would be clearly inconsistent with the context.

[0119] In this disclosure, each of the phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” may encompass any one of the items listed together in the corresponding phrase, or any possible combination thereof. Phrases such as “first,” “second,” or “first,” “second,” are used solely to distinguish one component from another, and such phrases do not limit other aspects of the component (e.g., importance, order, or sequence). When a component (e.g., the first) is referred to as “combined,” “connected,” or “linked” with or without the terms “functionally” or “communicatively” to another component (e.g., the second), it means that the first component can be combined, connected, or linked to the other component directly (e.g., by wire), wirelessly, or through a third component.

[0120] As used in this disclosure, the term "module" may comprise a unit implemented by hardware, software, or firmware, and is interrelationshiply interchangeable with terms such as logic, logic block, component, or circuit. A module can be a single component or the smallest unit or part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0121] Various embodiments of this disclosure may be implemented as software (e.g., a program) containing one or more commands stored in a storage medium (e.g., memory) readable by a device (e.g., an electronic device). The storage medium may include random access memory (RAM), memory buffers, hard drives, databases, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), and / or the like.

[0122] Furthermore, the processor of the embodiments of this disclosure can invoke and execute at least one of one or more commands stored from a storage medium. This enables the device to be operated to perform at least one function in accordance with the invoked at least one command. Such one or more commands may include compiler-generated code or interpreter-executable code. The processor may be a general-purpose processor, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), and / or the like.

[0123] The storage medium readable by the device may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and this expression does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0124] The various embodiments of the methods disclosed in this disclosure may be provided embedded in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a device-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or via an application store (e.g., the Play Store), or directly or online between two user devices (e.g., smartphones) (e.g., by download or upload). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated on a device-readable storage medium, such as a manufacturer's server, an application store's server, or a server's memory.

[0125] In a wide variety of embodiments, each of the components described above (e.g., a module or a program) may comprise one or more individuals. In a wide variety of embodiments, one or more of the aforementioned components or operations may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those functions performed by the components of the multiple components before integration. In a wide variety of embodiments, operations performed by a module, program or other component may be executed sequentially, in parallel, repeatedly, or heuristically, or one or more operations may be executed in a different order, omitted, or one or more other operations may be added. [Explanation of Symbols]

[0126] 100 devices 110 memory 112 PIM processor 120 Processors 130 Input / Output Interfaces 140 Communication Interfaces

Claims

1. A method for performing matrix operations involving multiple parameters using a system-on-chip (SoC) processor and a processing-in-memory (PIM) processor in memory, A step of calculating a map ID based on information relating to the matrix, information relating to the memory, and information relating to the processing-in-memory (PIM) processor in the memory. The steps include: allocating a huge page to represent some of the aforementioned parameters; The steps include adding the map ID to the page table entry of the huge page for the matrix operation in the system-on-chip (SoC) processor, Methods that include...

2. The information relating to the matrix includes information relating to the dimensions of the matrix, The method according to claim 1, wherein the map ID is a value that represents the degree to which the bit governing bank interleaving in the memory is separated from the chunk column.

3. The step of calculating the aforementioned map ID is performed by a mapping selector. The method according to claim 2, wherein the steps of allocating the huge page and adding the map ID to the page table entry of the huge page are performed by a memory allocator.

4. The method according to claim 3, further comprising the step of passing a memory allocation request from a user program, along with information about the matrix, to the mapping selector prior to the step of calculating the map ID.

5. The method according to claim 3, further comprising the step of adding the map ID to the page table entry of the huge page, and then returning to the user program a virtual address for the huge page allocated by the memory allocator.

6. The aforementioned matrix operations include matrix multiplication and matrix multiplication. The matrix multiplication is performed by the system-on-chip (SoC) processor. The method according to claim 1, wherein the multiplication of the matrix and the vector is performed by the processing-in-memory (PIM) processor.

7. The method according to claim 6, wherein the matrix operation is a matrix operation of weight values ​​used in large language model (LLM) inference.

8. The aforementioned large-scale language model (LLM) inference includes a prefill process and a decode process, The prefill process includes matrix multiplication performed by the system-on-chip (SoC) processor, The method according to claim 7, wherein the decoding process includes multiplication of a matrix and a vector performed by the processing-in-memory (PIM) processor.

9. The method according to claim 8, wherein the large-scale language model (LLM) inference is performed on a single edge device.

10. The method according to claim 1, wherein the information relating to the matrix includes at least one of information relating to the dimensions of the matrix and information relating to the data type of the matrix.

11. The method according to claim 10, wherein the information relating to the memory includes at least one of the following: the huge page size of the memory, the number of channels of the memory, the number of ranks of the memory, and the number of banks of the memory.

12. The method according to claim 11, wherein the information relating to the processing-in-memory (PIM) processor includes information relating to chunk columns.

13. The method according to claim 12, wherein the map ID is determined differently depending on whether a partition is required, by comparing the size of the huge page and the row size of the matrix with respect to the total number of banks of the memory.

14. The steps include: adding the map ID to the page table entry of the huge page, receiving a virtual address from a user program for storing or loading the matrix; The steps include determining the physical address corresponding to the virtual address, A step of determining the DRAM address of the memory corresponding to the physical address based on the physical address and the map ID, The method according to claim 1, further comprising:

15. The step of determining the DRAM address of the memory corresponding to the physical address is performed by the memory controller. The multiplexer of the memory controller receives the map ID as input, The method according to claim 14, wherein the memory controller comprises a register for storing the map ID.

16. The method according to claim 14, wherein the mapping between the DRAM address of the memory and the physical address is abstracted to the user program.

17. A computing device, System-on-a-chip (SoC) processor, Memory equipped with a Processing-in-Memory (PIM) processor, Equipped with, When a command relating to matrix operations stored in the memory is executed by the system-on-chip (SoC) processor and the PIM processor, the computing device, An operation to calculate a map ID based on information relating to the matrix, information relating to the memory, and information relating to the processing-in-memory (PIM) processor, An operation to allocate a huge page to represent some of the aforementioned multiple parameters, The operation of adding the map ID to the page table entry of the huge page for the matrix operation in the system-on-chip (SoC) processor, A computing device that performs [this task].

18. A computer program stored on a computer-readable medium, When the computer program is executed by the computing device, it performs matrix operations involving multiple parameters using a system-on-chip (SoC) processor and a processing-in-memory (PIM) processor in memory, and the said operation is: An operation to calculate a map ID based on information relating to the matrix, information relating to the memory, and information relating to the processing-in-memory (PIM) processor, An operation to allocate a huge page to represent some of the aforementioned multiple parameters, The operation of adding the map ID to the page table entry of the huge page for the matrix operation in the system-on-chip (SoC) processor, A computer program that includes [this].

Images