Method for deploying large language model and computing device

Abstract

The invention provides a method for deploying a large language model and a computing device, and the method comprises the steps: executing a weight unit off-line division process of a first stage: obtaining a to-be-deployed large language model, and before reasoning is executed, according to cold and hot feature statistics of each weight unit of the large language model, calculating the weight unit of the to-be-deployed large language model; determining an initial storage position for dividing each weight unit into a video memory or an expansion space of the DIMM to be stored; and executing a weight unit position on-line adjustment process of a second stage: in the process of executing reasoning, predicting the activated probability of each weight unit on line according to the condition of the currently executed layer, and adjusting the storage position of the related weight unit according to the activated probability.

Description

Technical Field

[0001] This invention relates to the field of neural network processing technology, more specifically to the field of deep learning technology, and more specifically to a method and computing device for deploying large language models. Background Technology

[0002] Large Language Models (LLMs) such as OPT, LLaMA, and the Qwen series excel in tasks like code generation, machine translation, and chatbots. However, these models, with billions of parameters, typically require server-grade GPUs with large-capacity, high-bandwidth storage, leading to high costs. For example, deploying an LLaMA2-70B requires five NVIDIA A100 GPUs, with a total cost exceeding $50,000.

[0003] To reduce costs, researchers have turned to consumer-grade GPUs, such as NVIDIA's RTX 4090. While these GPUs offer powerful computing capabilities, their limited memory space cannot meet the storage requirements of LLM (Language Modeling). Therefore, researchers have adopted an offloading strategy, transferring LLM parameters to DIMM-based host memory. Existing offloading schemes treat host memory as an extended memory space for the GPU, but the bandwidth difference between host memory and GPU memory (VRAM) exceeds 15 times, resulting in 99% of LLM runtime being spent on data transfer between the two. To alleviate the data transfer burden, existing work utilizes activation sparsity to reduce data loading. Since activation functions in LLM (such as ReLU) can zero out specific activation values, there is no need to load the corresponding parameters. Based on activation sparsity, neurons are divided into hot neurons and cold neurons. Approximately 20% of hot neurons are responsible for 80% of the computation, with a computational intensity 16 times that of cold neurons. Therefore, when deploying large language models on consumer-grade GPUs, storing hot neurons in GPU VRAM and offloading cold neurons to host memory can effectively alleviate data loading costs. Nevertheless, data transfer still accounts for 90% of the total inference latency, becoming the main bottleneck.

[0004] It should be noted that the background information presented here is only for illustrating relevant information about the present invention to aid in understanding the technical solution of the present invention, and does not imply that the relevant information is necessarily prior art. The relevant information was submitted and disclosed together with the present invention, and should not be considered prior art unless there is evidence that the relevant information was disclosed before the filing date of the present invention. Summary of the Invention

[0005] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a method and computing device for deploying large language models.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] According to a first aspect of the present invention, a computing device for deploying a large language model is provided, comprising a CPU, a GPU connected via PCIe, and multiple DIMMs, wherein: the DIMM includes a memory space, a general matrix-vector multiplication unit, and an activation function calculation unit, and a portion of the memory space serves as an extended space for GPU data storage; the CPU includes a scheduler, which controls the GPU and DIMM to collaboratively participate in the storage and processing of tasks related to the large language model, wherein the weight parameters corresponding to neurons of the large language model are divided into weight units according to a preset granularity, hot and cold feature statistics of the weight units are obtained, the globally hot weight units determined according to the hot and cold feature statistics are loaded into the GPU's video memory, the remaining weight units are stored in the extended space, and the general matrix-vector multiplication unit and / or activation function calculation unit are scheduled to complete calculations related to at least some of the weight units stored in the DIMM. This solution can achieve at least the following beneficial technical effects: by setting matrix-vector multiplication and activation function calculation units in the DIMM, the DIMM can not only participate in storage but also participate in some matrix-vector multiplication and activation function calculations in the large language model, without all data being transferred to the GPU for processing, effectively reducing the impact of data transfer between the GPU and memory on the total inference latency. Compared to the traditional approach of unloading large model weights from CPU memory and performing all calculations on the GPU, this invention offers higher computational efficiency and better resource utilization, improving overall performance at a lower hardware cost.

[0008] Optionally, the DIMM also includes an interconnect processing unit for supporting high-speed dedicated communication between different DIMMs. This interconnect processing unit does not occupy the PCIe bandwidth between the GPU and the DIMM, and the communication bandwidth is greater than or equal to 25GB / s. This solution can achieve at least the following beneficial technical effects: The method of this invention establishes a dedicated high-speed communication channel between DIMMs without occupying the PCIe bandwidth between the GPU and the DIMM, thereby improving the efficiency of data transmission within the DIMM.

[0009] Optionally, the CPU can establish a scheduler for unified management of the GPU and DIMM. The scheduler includes: an instruction queue for recording the instructions to be executed by the GPU and DIMM, and the scheduler sends instructions to the GPU or DIMM according to the recorded instructions; a weight unit mapper for determining the initial storage location of each weight unit in the GPU's video memory and the DIMM's extended space based on the hot and cold characteristics of the weight units during the offline partitioning process; and for predicting the activation probability of each weight unit online during the online adjustment process of the weight unit position, adjusting the storage location of the relevant weight units according to the activation probability, and generating storage location adjustment instructions for the relevant weight units and storing them in the instruction queue; and a predictor for predicting the activation probability of weight units within a specified number of downstream layers.

[0010] According to a second aspect of the present invention, a method for deploying a large language model is provided. The method deploys the large language model on a computing device as described in the first aspect, comprising: performing a first-stage offline weight unit partitioning process, including: acquiring the large language model to be deployed; and before performing inference, determining the initial storage location for each weight unit in video memory or the extended space of a DIMM based on statistical analysis of the hot and cold characteristics of each weight unit in the large language model; and performing a second-stage online weight unit position adjustment process, including: during inference, predicting the activation probability of each weight unit online based on the currently executed layers, and adjusting the storage location of the relevant weight units based on the activation probability. This method can achieve at least the following beneficial technical effects: in the initial first stage, a dataset can be input for analysis and testing to statistically analyze the usage of each weight unit of the large language model during forward propagation, thereby partitioning the initial storage location according to the macroscopic hot and cold characteristics of each weight unit. Subsequently, during the inference process of the large language model, since the processing of different layers or different input data have different effects on the activation of different weight units, the storage location of the weight units is adjusted online again based on the actual inference situation to better ensure the access speed of weight units with higher activation probability and improve the overall processing efficiency.

[0011] Optionally, the offline partitioning process of the weight units in the first stage includes: S11, obtaining the large language model to be deployed, and before inference, obtaining the hot and cold feature statistics of each weight unit of the large language model accessed during processing by inputting multiple samples of the dataset used for testing; S12, establishing an offline partitioning strategy of weight units between GPU and DIMM and a latency model of the inference latency of the large language model, which includes an objective function and modeling constraints. The objective function is to minimize the inference latency based on the partitioned storage location. The modeling constraints include storage constraints, which require that the space occupied by the weight units partitioned to GPU storage should be less than the size of the GPU memory and the space occupied by the weight units partitioned to DIMM storage should be less than the size of the extended space; S13, using linear programming, under the modeling constraints, based on the hot and cold feature statistics, solving the optimal allocation strategy of each weight unit corresponding to the objective function in the GPU memory and the extended space of DIMM as the initial storage location. This scheme can achieve at least the following beneficial technical effects: In this stage, neuron activity is recorded through testing for future scheduling optimization. Through hot and cold feature statistics, objective functions, and modeling constraints, hot weight units are tended to be loaded into GPU memory and cold weight units are loaded into DIMM storage, which reduces the adjustment cost of online partitioning and scheduling in subsequent inference processes.

[0012] Optionally, the hot / cold feature statistics include the following information for each weight unit: activation frequency statistics, which records the activation frequency of the weight unit in multiple inference processes; activation time statistics, which records the activation time of the weight unit in the inference process and identifies weight units that are continuously activated during the inference process; and activation dependency statistics, which records the activation dependency relationship between the weight unit and other weight units in the same layer and different layers. This scheme can achieve at least the following beneficial technical effects: by statistically analyzing hot / cold features from multiple aspects, the hot / cold features of each weight unit can be better characterized from different dimensions, thereby more effectively preventing large-scale adjustments to the weight unit positions during subsequent online adjustments, reducing adjustment overhead and access speed fluctuations.

[0013] Optionally, the online adjustment process of weight unit positions in the second stage includes: S21, predicting the activation probability of weight units within a specified number of downstream layers based on the correlation between tokens and the dependency between layers currently being executed by the large language model; S22, during the inference process of the large language model, selecting one or more local hot weight units in the specified number of layers and loading them into the GPU's video memory in descending order of activation probability, based on the activation probability of weight units within the specified number of layers. This scheme can achieve at least the following beneficial technical effects: During real-time inference, this scheme can predict the activation probability of relevant weight units within a specified number of layers in real time, and can load a small number of potentially frequently accessed local hot weight units in the layers that will be used recently into the video memory, thereby avoiding loading too many local hot weight units into the video memory too early, and allowing the hot weight units that will be used recently to be loaded into the video memory in real time, thus improving the overall processing efficiency.

[0014] Optionally, S22 further includes: determining, in addition to the locally hot weight units loaded into the GPU's video memory, the remaining weight units stored in the DIMM and their activation probabilities; and adjusting the distribution among multiple DIMMs based on the weight units stored in the DIMMs and their activation probabilities, so that weight units with higher activation probabilities are evenly distributed among the multiple DIMMs. This scheme can achieve at least the following beneficial technical effects: on the one hand, this scheme adjusts locally hot weight units to the GPU's video memory in real time to improve overall processing efficiency; on the other hand, this scheme also evenly distributes the less hot weight units among the DIMMs to achieve overall computational balance and avoid a single DIMM storing too many less hot weight units, which could cause access congestion and affect overall efficiency.

[0015] Optionally, in S22, during the inference process of the large language model, the input projection layer in the attention mechanism is configured to perform computation collaboratively on the GPU and DIMM. The input projection layer is used to map the input data into query vectors, key vectors, and value vectors. This scheme can achieve at least the following beneficial technical effects: the input projection layer is computed collaboratively on the GPU and DIMM, which can make full use of the computing resources of both, reduce the generation time of QKV vectors in the early stage, and better ensure processing efficiency.

[0016] Optionally, in S22, the attention computation in the attention mechanism is configured to be performed in the DIMM. This scheme can achieve at least the following beneficial technical effects: since the attention computation in the attention mechanism is performed on the DIMM, the additional data transmission overhead can be minimized, and the overall processing efficiency can be improved.

[0017] Optionally, in S22, during the inference process of the large language model, the output projection layer in the attention mechanism is configured to be computed on the GPU, and each time the output projection layer is computed, the CPU is used to perform an online adjustment process for the weight unit positions. This scheme can achieve at least the following beneficial technical effects: during the projection computation of the output projection layer, since the DIMM is completely idle, the host uses this time to perform an online adjustment process, dynamically reconfiguring the storage location of the weight units according to the prediction results, which can both mask the reconfiguration latency and improve the efficiency of subsequent inference. Attached Figure Description

[0018] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0019] Figure 1 This is a schematic diagram of a computing device for deploying a large language model according to an embodiment of the present invention;

[0020] Figure 2 A schematic diagram illustrating the computational principle of deploying a large language model on a computing device according to an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0022] As mentioned in the background section, to reduce costs, when deploying large language models on consumer-grade GPUs, hot neurons are stored in GPU memory while cold neurons are offloaded to host memory, effectively alleviating data loading costs. However, data transfer still accounts for 90% of the total inference latency, becoming the main bottleneck. Therefore, this invention sets up matrix-vector multiplication and activation function calculation units in a Dual In-line Memory Module (DIMM). This allows the DIMM to not only participate in storage but also in some matrix-vector multiplication and activation function calculations within the large language model, without transferring all data to the GPU. This effectively reduces the impact of data transfer between the GPU and memory on the total inference latency. Compared to the traditional method of offloading large model weights only from CPU memory and completing all calculations on the GPU, this invention offers higher computational efficiency and better resource utilization, improving overall performance at a lower hardware cost.

[0023] According to one embodiment of the present invention, Figure 1 This diagram schematically illustrates the modular structure of a computing device for deploying large language models. The device includes a GPU and multiple DIMMs connected via PCIe, as well as a CPU. Wherein:

[0024] GPU, or Graphics Processing Unit, includes video memory (graphics memory).

[0025] DIMM, including memory space ( Figure 1 (Not shown), a general matrix-vector multiplication unit, an activation function calculation unit, and / or an interconnect processing unit are included, with a portion of the memory space serving as an extended space for GPU data storage. The general matrix-vector multiplication unit can be used to perform matrix-vector multiplication calculations. The activation function calculation unit can be used to perform activation function calculations (such as ReLU, LeakyReLU, etc.). The interconnect processing unit is used to support high-speed dedicated communication between different DIMMs. This interconnect processing unit does not occupy the PCIe bandwidth between the GPU and the DIMM, and the communication bandwidth is greater than or equal to 25GB / s. Compared to conventional memory, this invention adds a proximity processing unit in the DIMM (such as the central cache location) to perform matrix-vector multiplication calculations, perform activation function calculations, and support efficient communication between DIMMs. Based on the constructed proximity processing unit, the overall architecture of the GPU and DIMM is constructed, and the GPU graphics storage unit and the DIMM are connected using PCIe.

[0026] The CPU, or Central Processing Unit, houses the scheduler, which manages the overall architecture of the GPU and DIMM. The scheduler includes an instruction queue, a weighted unit mapper, a runtime detector, and a predictor, or a combination thereof.

[0027] The instruction queue is used to record the instructions that the GPU and DIMM will execute. The scheduler sends instructions to the corresponding device (GPU or DIMM) based on the recorded instructions.

[0028] A weight unit mapper is used to determine the initial storage location of each weight unit in the GPU's video memory and DIMM expansion space based on the hot / cold characteristics of the weight units during the offline weight unit partitioning process; and during the online weight unit position adjustment process, it predicts the activation probability of each weight unit online and adjusts the storage location of the relevant weight units according to the activation probability. It also generates storage location adjustment instructions for the relevant weight units and stores them in an instruction queue. It should be noted that the weight unit of this invention is the smallest management unit obtained by partitioning the weight parameters of the large oracle model according to the granularity set by the user. Depending on the different granularities set by the user, a weight unit can be all the weight parameters corresponding to a neuron, or a row or column of the weight matrix corresponding to a neuron, etc.

[0029] A runtime detector is used to detect the workload of each component, the GPU and the DIMM, during operation. This workload can be used to adjust the computational load on both.

[0030] A predictor is used to predict the activation probability of weighted units within a specified number of downstream layers. For example, the first or second downstream layer.

[0031] According to an embodiment of the present invention, a method for deploying a large language model is provided. Deploying the large language model in the computing device described in the above embodiment includes the following steps: S1, performing a first-stage offline weight unit partitioning process, including: acquiring the large language model to be deployed; before performing inference, determining the initial storage location for each weight unit to be partitioned into video memory or the extended space of a DIMM based on statistical analysis of the hot and cold characteristics of each weight unit in the large language model; S2, performing a second-stage online weight unit position adjustment process, including: during inference, predicting the activation probability of each weight unit online based on the currently executed layers, and adjusting the storage location of the relevant weight units based on the activation probability.

[0032] To better understand the present invention, each step will be described in detail below with reference to specific embodiments.

[0033] According to an embodiment of the present invention, step S1 includes:

[0034] S11. Obtain the large language model to be deployed. Before performing inference, obtain the statistics of hot and cold features accessed by each weight unit of the large language model during processing by inputting multiple samples of the dataset used for testing.

[0035] The hot / cold feature statistics include the following information for each weighted unit: activation frequency statistics, activation time statistics, and activation dependency statistics. Activation frequency statistics record the activation frequency of the weighted unit across multiple inference processes. Activation time statistics record the activation time of the weighted unit during inference and identify weighted units that remain continuously activated during inference. Activation dependency statistics record the activation dependencies between the weighted unit and other weighted units in the same and different layers.

[0036] S12. Establish an offline partitioning strategy for weight units between GPU and DIMM and a latency model for inference latency of large language models. This includes an objective function and modeling constraints. The objective function is to minimize inference latency based on the partitioned storage location. The modeling constraints include storage constraints, which require that the space occupied by weight units partitioned to GPU storage should be less than the size of GPU memory and the space occupied by weight units partitioned to DIMM storage should be less than the size of the expansion space.

[0037] The illustrative, established time delay model includes:

[0038] Objective function setting: The inference latency of a large language model is determined by the large computational latency of GPU and DIMM. The computational latency of GPU and DIMM is proportional to the number of weight units stored therein that are activated during inference.

[0039] Constraints: The amount of storage required for the weight units stored in the GPU and DIMM must be less than or equal to the amount of storage that the device can provide.

[0040] S13. Using linear programming, under modeling constraints and based on hot and cold feature statistics, solve for the optimal allocation strategy of each weight unit corresponding to the objective function in the GPU's video memory and the DIMM's extended space as the initial storage location. Linear programming methods can include Pulp, CVXPY, gurobi Optimizer, or COIN-OR. The initial storage location is the offline partitioning result; according to the offline partitioning result, the weight units can be stored in the corresponding GPU video memory or DIMM. Linear programming can be used to determine the globally hot and globally cold weight units. In the initial storage location, the globally hot weight units are stored in the GPU's video memory, and the globally cold weight units are stored in the DIMM's extended space.

[0041] According to an embodiment of the present invention, step S2 includes:

[0042] S21. Based on the correlation between tokens and the dependency between layers in the current execution of the large language model, predict the activation probability of weight units within a specified number of downstream layers.

[0043] According to one embodiment of the present invention, large language models are typically based on the Transformer architecture. A token is the smallest semantic or syntactic unit after the input text has been segmented using a word segmentation algorithm. That is, it is the basic discrete unit for large language models to process and understand natural language text. The correlation between tokens refers to the strength of the semantic, syntactic, or contextual dependencies (such as co-occurrence frequency, attention weights) between different tokens in the large language model. For example, the tokens "cat" and "dog" may have a high correlation due to semantic similarity. Inter-layer dependencies refer to the activation dependencies between the weight units of the currently executing layer and the layers within a specified downstream layer. Inter-layer dependencies involve the correlation of different layers in the neural network in selecting activated weight units; for example, by analyzing that when the 6th weight unit of layer 30 is activated, the 0th weight unit of layer 31 also has a high probability of being activated.

[0044] S22. During the inference process of the large language model, based on the activation probability of the weight units within a specified layer, one or more local hot weight units of the specified layer are selected and loaded into the GPU's video memory in descending order of activation probability.

[0045] According to one embodiment of the present invention, a specified number (e.g., 10 or 20) of weight units can be loaded from the DIMM to the GPU in each online adjustment process. A specified number of weight units from a specified layer are selected as local hot weight units and loaded into the GPU's video memory in descending order of activation probability. Alternatively, the activation probability threshold can be used to determine the local hot weight units, with weight units whose activation probability exceeds the threshold being selected as local hot weight units.

[0046] According to one embodiment of the present invention, S22 further includes: determining, in addition to the local hot weight units loaded into the GPU's video memory, the remaining weight units stored in the DIMM and their activation probabilities; and adjusting the distribution among multiple DIMMs according to the weight units stored in the DIMMs and their activation probabilities, so that weight units with higher activation probabilities are evenly distributed among the multiple DIMMs. The weight units stored in all DIMMs can be sorted in descending order of activation probability. If a DIMM has a higher activation probability, the weight units with higher activation probabilities are swapped with those with lower activation probabilities to achieve balanced distribution. Alternatively, the potential activation number of weight units in each DIMM with activation probabilities exceeding a predetermined threshold can be analyzed, the average of the total potential activation numbers can be calculated, and some weight units with activation probabilities exceeding the predetermined threshold in DIMMs with potential activation numbers exceeding the average can be replaced in other DIMMs with potential activation numbers less than the average to achieve balanced distribution.

[0047] According to one example of the invention, Figure 2This example schematically illustrates the partitioning and calculation principles of weight parameters during the overall operation of a large language model inference using a hybrid architecture combining GPU and DIMM. In this example, the entire weight parameter of a neuron is used as a single weight unit; therefore, the weight unit is referred to simply as a neuron.

[0048] In the first stage of LLM inference, during the processing of the input text, due to significant computational demands, the entire prompting stage of the Transformer layer is processed on the GPU, followed by key-value caching (KV caching) during the large language model inference process. During this stage, the CPU scheduler on the host machine records neuron activity for future scheduling optimization. After this stage, the initial offline neuron mapping is derived using offline analysis information, loading only selected hot neurons into GPU memory. Offline analysis information refers to the statistical data and knowledge obtained before formally deploying the model for inference (the online stage) through pre-analysis (i.e., "offline") of the model's weight structure and a large amount of training text. This includes statistics on the hot and cold characteristics of neurons, activation patterns (recording the frequency with which any two (or groups of) neurons are activated simultaneously or sequentially when processing the same text; higher frequencies indicate stronger functional correlation), token correlations (recording which neurons are most frequently activated by a specific token (or token sequence), and inter-layer dependencies (describing the causal relationships of neuron activation between different neural network layers). This reduces the adjustment costs of online partitioning and scheduling during subsequent inference. During analysis, a publicly available dataset was used as the model to be run. The dataset was input into the model for multiple forward propagations. During this process, the activation state of each neuron was recorded, i.e., which neurons were activated and the number of times they were activated; the corresponding input token sequence at the time of activation was also recorded. This recorded information was then analyzed to count the number of times any two neurons were jointly activated in the same batch of data processing, and to analyze the correlation between specific tokens or token combinations and neuron activation, in order to extract stable dependency patterns across layers of neurons.

[0049] In the second stage, QKV vector generation (QKV generation for short) is completed collaboratively by the GPU and DIMM near memory during each layer of inference. The output of QKV generation is collected in the DIMM near memory for further attention computation. This method has two advantages. First, since the GPU typically completes computation tasks faster due to its superior computing power, the latency of transferring data from the GPU to the DIMM can be hidden by the computation in the DIMM, thus not affecting the overall system runtime. Second, since the attention computation is performed on the DIMM near memory, merging the results of QKV generation at the DIMM end can minimize additional data transfer overhead. The projection layer, due to its higher computational requirements, is processed separately by the GPU. During the projection computation, since the DIMM is completely idle, the CPU uses this time to dynamically reconfigure hot / cold partitions based on the prediction results and redistribute neurons among the DIMMs. Adjusting the hot / cold neuron partitions online before neuron computation can improve inference efficiency, which requires predicting neuron partitions in advance. This invention combines inter-token and inter-layer prediction strategies to achieve accurate prediction of activated neurons during the token generation process and analyze local hot neurons. For example, based on the prediction criteria, local hot neurons 3, 6, and 9 will be used in subsequent calculations, but neuron 6 is not yet stored in the GPU's video memory, so it is loaded into the GPU's video memory. Based on the prediction results, local hot neurons are loaded into the GPU's video memory, and load balancing is achieved among multiple DIMMs while minimizing the remapping of non-local hot neurons. Then, the multilayer perceptron (MLP) is offloaded to the GPU and DIMMs. Finally, the output of each layer is collected in the DIMM as input for subsequent layers.

[0050] In summary, some embodiments of the present invention can achieve at least one or more of the following beneficial effects:

[0051] (1) The method of this invention improves the performance of large models at a lower cost. By constructing a hybrid architecture of GPU and DIMM near-memory, and adopting an offline weight position partitioning and online adjustment mechanism, the prediction of hot and cold weight units and the online dynamic adjustment of their positions are realized, ensuring the computational balance between GPU and DIMM, as well as between the near-memory units of each DIMM. Compared with the traditional method of only unloading large model weights from CPU memory and computing them entirely on the GPU, this invention has higher computational efficiency and better resource utilization, improving overall performance at a lower hardware cost.

[0052] (2) Break through the PCIe bandwidth bottleneck in the traditional offloading scheme, realize heterogeneous collaboration between computing resources and storage resources, reduce hardware costs by 80% (compared to the A100 server scheme), and reduce the computation latency of cold weight units by utilizing the high bandwidth inside the DIMM of the near-memory computing unit.

[0053] (3) Compared with the random task partitioning strategy, the offline weight partitioning based on statistical information improves the partitioning accuracy by about 50%, thereby reducing the online optimization overhead and improving the end-to-end inference latency.

[0054] (4) Load balancing is achieved by combining the correlation between tokens and the dependency between layers to predict neuron activation. Compared with the static allocation scheme, the inference speed can be increased by up to 13.75 tokens per second in the test model, and the load difference between DIMMs is reduced from 2.5 times to 1.2 times.

[0055] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0056] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0057] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can include, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0058] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computing device for deploying large language models, comprising a CPU, a GPU connected via PCIe, and multiple DIMMs, wherein: DIMM includes memory space, general matrix-vector multiplication unit, and activation function computation unit. A portion of the memory space serves as an extended space for GPU data storage. The CPU, including a scheduler, controls the GPU and DIMM to collaboratively participate in the storage and processing of large language model-related tasks. Specifically, the CPU divides the weight parameters corresponding to neurons of the large language model into weight units according to a preset granularity, obtains the hot and cold feature statistics of the weight units, loads the globally hot weight units determined based on the hot and cold feature statistics into the GPU's video memory, and stores the remaining weight units in the extended space. Furthermore, the CPU schedules the general matrix-vector multiplication unit and / or activation function calculation unit to complete the calculations related to at least some of the weight units stored in the DIMM.

2. The computing device according to claim 1, characterized in that, The DIMM also includes an interconnect processing unit for supporting high-speed dedicated communication between different DIMMs. This interconnect processing unit does not occupy the PCIe bandwidth between the GPU and the DIMM, and the communication bandwidth is greater than or equal to 25GB / s.

3. The computing device according to claim 1 or 2, characterized in that, The CPU can establish a scheduler for unified management of the GPU and DIMM, and the scheduler includes: The instruction queue is used to record the instructions that the GPU and DIMM will execute. The scheduler sends instructions to the GPU or DIMM based on the recorded instructions. The weight unit mapper is used to determine the initial storage location of each weight unit in the GPU's video memory and DIMM expansion space based on the hot and cold characteristics of the weight units during the offline weight unit partitioning process; and during the online weight unit position adjustment process, it predicts the activation probability of each weight unit online, adjusts the storage location of the relevant weight units according to the activation probability, and generates the storage location adjustment instructions of the relevant weight units and stores them in the instruction queue. A predictor is used to predict the activation probability of weighted units within a specified number of downstream layers.

4. A method for deploying a large language model, characterized in that, Deploying a large language model on a computing device as described in any one of claims 1-3 includes: The first stage of the weight unit offline partitioning process includes: obtaining the large language model to be deployed; and determining the initial storage location for each weight unit to be partitioned into video memory or the extended space of DIMM based on the statistical hot and cold features of each weight unit of the large language model before performing inference. The second phase of the online adjustment process for weight unit positions includes: during the inference process, predicting the activation probability of each weight unit online based on the current layer being executed, and adjusting the storage position of the relevant weight units according to the activation probability.

5. The method according to claim 4, characterized in that, The offline weight unit partitioning process for the first phase includes: S11. Obtain the large language model to be deployed. Before performing inference, obtain the statistics of hot and cold features accessed by each weight unit of the large language model during processing by inputting multiple samples of the dataset used for testing. S12. Establish an offline partitioning strategy for weight units between GPU and DIMM and a latency model for inference latency of large language models. This includes an objective function and modeling constraints. The objective function is to minimize inference latency based on the partitioned storage location. The modeling constraints include storage constraints, which require that the space occupied by weight units partitioned to GPU storage should be less than the size of GPU memory and the space occupied by weight units partitioned to DIMM storage should be less than the size of the extended space. S13. Using linear programming, under modeling constraints, and based on hot and cold feature statistics, the optimal allocation strategy of each weight unit corresponding to the objective function in the GPU's video memory and DIMM expansion space is solved as the initial storage location.

6. The method according to claim 5, characterized in that, The cold and hot feature statistics include the following information for each weighted unit: Activation frequency statistics record the activation frequency of the weighted unit in multiple inference processes; Activation time statistics record the activation time of the weight unit during the inference process and identify weight units that are continuously activated during the inference process. Activation dependency statistics record the activation dependencies between the weight unit and other weight units in the same and different layers.

7. The method according to claim 4, characterized in that, The online adjustment process for the weighted unit position in the second phase includes: S21. Based on the correlation between tokens and the dependency between layers in the current execution of the large language model, predict the activation probability of weight units within a specified number of downstream layers. S22. During the inference process of the large language model, based on the activation probability of the weight units within a specified layer, one or more local hot weight units of the specified layer are selected and loaded into the GPU's video memory in descending order of activation probability.

8. The method according to claim 7, characterized in that, S22 further includes: In addition to the local hot weight units loaded into the GPU's video memory, determine the remaining weight units stored in the DIMM and their activation probabilities. Based on the weight units stored in the DIMM and their activation probabilities, the distribution is adjusted among multiple DIMMs so that weight units with higher activation probabilities are evenly distributed among the multiple DIMMs.

9. The method according to claim 8, characterized in that, In S22, during the inference process of the large language model, the input projection layer in the attention mechanism is configured to perform computation collaboratively on the GPU and DIMM. The input projection layer is used to map the input data into query vectors, key vectors, and value vectors.

10. The method according to claim 9, characterized in that, In S22, the attention computation in the attention mechanism is configured to be performed in the DIMM.

Images