Large model deployment method for enhancing lama.cpp framework

Abstract

The invention discloses a large model deployment method for enhancing a lama.cpp framework, and belongs to the technical field of large language model reasoning optimization. The method comprises the following steps: firstly, enhancing a lama.cpp framework to support operator-level tensor equipment distribution; secondly, constructing a 0-1 integer linear programming model with model throughput maximization as a target, defining three decision variables of operator allocation, whole layer allocation and operator cohesion by the model, and setting constraint conditions of single operator allocation, equipment resource capacity, allocation sequence and the like; and finally, solving the planning problem by adopting a branch and bound method to obtain an optimal operator-level resource allocation scheme, and deploying each operator of the model to a GPU or a CPU according to the scheme. According to the method, the video memory and the memory of the equipment can be finely managed, and operators which have great influence on the throughput are preferentially deployed in the GPU, so that the reasoning throughput and the video memory utilization rate of a large model are remarkably improved under the condition of limited resources.

Description

Technical Field

[0001] This invention belongs to the field of large language model inference optimization technology, specifically involving a method for deploying large models that enhances the llama.cpp framework. Background Technology

[0002] To efficiently deploy large language models and achieve high-speed inference on hardware platforms, various large language model inference frameworks have been proposed in the industry. Among them, frameworks such as vLLM, SGLang, and TensorRT-LLM fully leverage the computing power of GPUs by deeply integrating GPU computing libraries such as CUDA, achieving extremely high inference throughput on servers with sufficient high-end GPU resources. However, these frameworks typically rely heavily on GPUs and have high requirements for GPU memory capacity and computing performance, making them difficult to effectively deploy and run on edge computing or ordinary computing nodes where GPU resources are limited (such as insufficient memory, older GPU models, or limited quantities).

[0003] Llama.cpp is a high-performance large language model inference framework implemented in pure C / C++. Its significant advantage lies in its support for dynamically offloading parts of the model to the system's CPU memory for computation, thus enabling hybrid collaborative inference between the GPU and CPU. This approach greatly reduces the total GPU memory requirements, making it possible to deploy large models in resource-constrained hardware environments and broadening the application scenarios for large models.

[0004] However, the existing Llama.cpp framework (up to version b4827) suffers from a significant coarse-grained resource allocation strategy. Currently, the framework only supports allocation at the level of complete model layers, meaning all operators within a layer must reside entirely on the same device (GPU or CPU). While this "layer-by-layer allocation" strategy is simple to implement, it fails to fully utilize the heterogeneous computing resources of the device. Specifically, within a Transformer layer, different operators (such as qkv computation, FFN, etc.) have significantly different computational characteristics and contributions to overall inference throughput. Due to the limitation of allocating entire layers, when the memory of a single GPU is insufficient to accommodate the entire layer, even if the GPU still has remaining memory, the framework will be forced to offload the entire layer (including critical operators essential to throughput) to the CPU, resulting in idle GPU computing resources and potentially making the CPU a performance bottleneck. Conversely, forcing the entire layer onto the GPU may fail due to insufficient memory.

[0005] Therefore, the coarse-grained resource allocation mechanism of the existing Llama.cpp framework limits its potential to further optimize model throughput under limited resources. It cannot perform fine-grained resource scheduling based on operator importance, which prevents the inference performance of large models such as iFlytek Spark from reaching the theoretical upper limit allowed by the device hardware resources. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides an enhanced method for deploying large models within the llama.cpp framework. This method solves the problem that the existing llama.cpp framework allocates computing resources at the whole-layer granularity, which fails to maximize the inference throughput of large models when device resources are limited.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for deploying large models that enhances the llama.cpp framework includes:

[0009] Step S1: Construct a 0-1 integer linear programming model with the objective of maximizing model throughput. The programming model includes decision variables for representing operator-level device allocation, resource and order constraints to ensure the feasibility of the scheme, and a utility function consisting of operator GPU allocation score and cohesion reward within the same device.

[0010] Step S2: Solve the planning model using the branch and bound method to obtain the optimal operator-level resource allocation scheme;

[0011] Step S3: Enhance the llama.cpp framework to support operator-level tensor allocation, and according to the optimal operator-level resource allocation scheme, deploy each operator of the large model to the corresponding GPU or CPU in the enhanced llama.cpp framework.

[0012] Furthermore, in step S1, the decision variables include operator allocation variables, whole-layer allocation variables, and operator-device cohesion variables.

[0013] Furthermore, the operator allocation variable indicates that a specific operator of a specific layer is allocated to a specific device, the whole-layer allocation variable indicates whether a specific layer is allocated to a specific device, and the operator-device cohesion variable indicates whether the qkv operator of a specific layer is deployed on the same GPU as other operators of the same layer.

[0014] Furthermore, in step S1, the constraints include single operator allocation constraints, definition and implementation constraints of the whole-layer allocation variables and the operator's cohesion variables within the same device, resource constraints, and allocation order constraints.

[0015] Furthermore, the single operator allocation constraint ensures that each operator must be allocated to one and only one device; the device resource capacity constraint ensures that the total memory usage of operators allocated to each device does not exceed the available capacity of that device; and the allocation order constraint is used to regulate the allocation order of operators between layers.

[0016] Furthermore, in step S1, the utility function serves as the optimization objective of the planning model. The utility function is composed of the weighted sum of the scores allocated to the GPU by the operator, the scores allocated to the GPU across the entire layer, and the reward scores for the cohesion between the operator and the device.

[0017] Furthermore, step S2 includes:

[0018] Initialization process: Create a list of active nodes and add the original planning problem as the initial node to the list, solve the linear relaxation problem of the node, record the relaxation solution and the corresponding utility score, and set the initial optimal solution to empty;

[0019] Node processing loop: When the list of active nodes is not empty, the following steps are executed in a loop: Select and remove the node with the best relaxation solution from the list; if the utility score corresponding to the relaxation solution of the node is lower than the current best score, then prune the node; if it is not pruned and the relaxation solution of the node is an integer solution, then compare it with the current best solution and update it; if it is not pruned and the relaxation solution is a non-integer solution, then first try to add cutting plane conditions to tighten it. If it cannot be tightened further, then branch: Select a decision variable with a value closest to 0.5, and generate two new child nodes by constraining the variable to 0 or 1 respectively, and add them to the list of active nodes;

[0020] Output result: When the list of active nodes is empty, the currently stored optimal solution is the optimal operator-level resource allocation scheme.

[0021] Furthermore, in step S3, a processing module is added to the parameter loading section of the llama.cpp framework to specify the list of buffers to be used when creating a tensor with specific parameters.

[0022] In a second aspect, the present invention provides an electronic device comprising: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method for deploying large models of an enhanced llama.cpp framework.

[0023] Thirdly, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enable the processor to implement the aforementioned method for deploying large models of the enhanced llama.cpp framework.

[0024] The beneficial effects of this invention are as follows:

[0025] Significant throughput improvements were achieved: By refining resource allocation granularity from the "layer" level to the "operator" level and constructing an optimization model aimed at maximizing throughput, the system can intelligently prioritize the deployment of key operators (such as qkv computation) that contribute the most to throughput on GPUs. In actual deployment, compared to the traditional layer-based allocation scheme, this invention enabled iFlytek's large model to achieve considerable throughput improvements in both the pre-filling and decoding stages.

[0026] Improved hardware resource utilization: This invention overcomes the memory fragmentation problem caused by whole-layer allocation through fine-grained operator-level allocation, allowing limited GPU memory to be utilized more fully and evenly. Experimental data shows that after adopting the solution of this invention, the GPU memory utilization rate is further improved compared with the layer-based allocation scheme, avoiding resource idleness.

[0027] Enhanced deployment capabilities in resource-constrained environments: The planning model designed in this invention strictly considers the constraints of device video memory and system memory capacity, and introduces reasonable allocation order constraints (such as avoiding device skipping), ensuring that the generated allocation scheme is not only optimal in performance on resource-constrained hardware platforms (such as servers with multiple old GPUs), but also feasible and stable, expanding the application potential of large models in scenarios such as edge computing.

[0028] Optimization process automation and intelligence: This method formalizes the resource allocation problem into a complete mathematical programming problem and uses the branch and bound method to solve it, realizing automatic optimization of allocation schemes, avoiding reliance on human experience for trial and error tuning, and improving deployment efficiency and consistency of results. Attached Figure Description

[0029] Figure 1 Flowchart for solving the planning problem;

[0030] Figure 2 This is a schematic diagram comparing the pre-filled throughput based on the present invention and existing technologies;

[0031] Figure 3 This is a schematic diagram comparing the decoding throughput of the present invention with that of existing technologies. Detailed Implementation

[0032] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0033] As mentioned earlier, this invention uses the iFlytek Spark Open Source-13B model as an example. The iFlytek Spark Open Source-13B model is a large language model consisting of an input processing layer and 40 Transformers layers with the same structure. Each Transformer layer can be divided into seven operators that implement different computations: qkv computation, attn_proj, attn_norm, ffn_norm, ffn_gate, ffn_up, and ffn_down. Each operator has parameter tensors that depend on its computation and need to be stored. In the original llama.cpp framework, the input and output processing layers are allocated to the CPU by default, while the Transformers layer can be allocated entirely to either the GPU or the CPU. The purpose of this invention is to find and implement an optimal operator-level resource allocation scheme based on the Spark Open Source-13B model information and device resource conditions, enabling the iFlytek large model to achieve higher throughput on resource-limited platforms. Its core principle is: based on the differences between the structural features of the iFlytek large model and the standard llama2 model, llama.cpp adds support for the iFlytek large model. In the tensor creation part, a processing branch is added to handle layers that are not fully allocated to the same GPU or CPU. Based on the operator allocation scheme, the tensor corresponding to each operator is loaded onto the corresponding device. A comparative experiment on operator allocation is conducted. By assigning different operators of a certain layer to the GPU for testing and inference, the importance order of different operators allocated to the GPU on throughput can be obtained, identifying the operator with the greatest impact on throughput. Experiments show that the qkv computation operator has the highest throughput for the model, and the order of influence of other operators on throughput is ffn(gate / up / down) > attn_proj > normal(attn / ffn). A 0-1 integer linear programming problem is constructed and solved using the branch-and-split method. Specifically, the method includes:

[0034] Step S1: Construct a 0-1 integer linear programming model with the objective of maximizing model throughput. The programming model includes decision variables for representing operator-level device allocation, resource and order constraints to ensure the feasibility of the scheme, and a utility function consisting of operator GPU allocation score and cohesion reward within the same device.

[0035] The decision variables include: operator allocation variables indicating the allocation of a specific operator of a specific layer to a specific device; whole-layer allocation variables indicating whether a specific layer is allocated entirely to a specific device; and operator cohesion variables indicating whether the qkv operator of a specific layer is deployed on the same GPU as other operators in the same layer.

[0036] The operator allocation variable x[(l, o, d)] has a value of 1, which means that the operator o of layer l is allocated to device d (a GPU or CPU).

[0037] The variable whole_layer[(l, d)] is used for whole-layer allocation. A value of 1 indicates that layer l has been fully allocated to device d.

[0038] The operator cohesion variable z_colocate[(l, op_other, gpu_k)] has a value of 1, which means that the qkv operator of layer l and another operator op_other are deployed together on a certain GPU, namely gpu_k.

[0039] The constraints include:

[0040] Single operator assignment constraint ensures that each operator must be assigned to one and only one device:

[0041]

[0042] The definition of the whole-layer allocation variable and the operator's cohesion variable within the device implements constraints; among them, the definition of the whole-layer allocation variable whole_layer[(l, d)] is implemented through constraint conditions, as shown in formulas (2)-(3):

[0043]

[0044]

[0045] Where op_list represents the list of operators, and len represents the length of the list of operators.

[0046] The operator-device cohesion variable z_colocate[(l, op_other, gpu_k)] is defined through constraints, as shown in formulas (4)-(6):

[0047]

[0048]

[0049]

[0050] Equipment resource capacity constraints ensure that the total memory usage of operators allocated to each device does not exceed the available capacity of that device; as shown in formula (7):

[0051]

[0052] Where fixed_overhead(d) represents the fixed computational overhead of device d, device_capacity(d) represents the available capacity of device d, and memory(o) represents the storage space occupied by operator o;

[0053] Allocation order constraints are used to regulate the allocation order of inter-layer operators, including the priority of whole-layer allocation constraints, that is, if layer l is not allocated to the GPU in its entirety, then layer l+1 cannot be allocated to the GPU in its entirety either, as shown in formula (8):

[0054]

[0055] To avoid device jump constraints, if operator o of layer l or the entire layer l is allocated to a high-index GPU, then operator o of layer l+1 or the entire layer l+1 cannot be allocated to a low-index GPU, as shown in formulas (9)-(10):

[0056]

[0057]

[0058] In this context, gpu_higher_idx represents the GPU with the higher index, and gpu_lower_idx represents the GPU with the lower index.

[0059] If an operator o in layer l is assigned to the CPU, then operator o in layer l+1 cannot be assigned to the GPU, as shown in formula (11):

[0060] .

[0061] The utility function serves as the objective function for the planning problem. The utility function consists of operator allocation score, whole-layer allocation score, and operator cohesion reward score. The specific calculation of the utility score using the utility function is shown in formula (12):

[0062]

[0063] in Let represent the score assigned to operator o on the GPU, and w represent the base score assigned to each layer on the GPU. represents the base score for the order, n represents the total number of layers in the model, c represents the cohesion reward score for the same device, and l_idx represents the sequence number corresponding to layer l.

[0064] Step S2: Solve the planning model using the branch and bound method to obtain the optimal operator-level resource allocation scheme; wherein, the solution requires initialization first, the initialization process includes model information acquisition, hardware resource acquisition, and utility function coefficient assignment:

[0065] (1) Model information acquisition:

[0066] Information about the Spark Open Source-13B model, including the number of layers, is obtained from a local gguf file. The exact size of each tensor is calculated based on its shape, quantization type, and the GGML_QUANT_SIZES data provided by gguf, which includes block size and byte count information. This allows for the calculation of the tensor size corresponding to each operator.

[0067] (2) Acquisition of hardware resource information;

[0068] The GPU memory is obtained using the pynvml library, and the system available memory is obtained using the psutil library. Users can specify the GPU to use; if not specified, all obtained GPUs will be used. The fixed overhead for each device is obtained from a user-provided list of fixed computational overheads, with CPU computational overhead set to 0. When the fixed overhead is a fixed context length, it represents the difference between the actual GPU memory usage during runtime and the CUDA buffer calculated by the llama.cpp framework.

[0069] (3) Assigning values ​​to the coefficients of the utility function;

[0070] Assign appropriate values ​​to the coefficients of the utility function. The throughput is determined by the magnitude of the impact of operator o, and The values ​​should be relatively large to ensure that the qkv operator is allocated to the GPU as much as possible. A recommended set of utility function coefficients is as follows: =20000, =500, = =400, =150, = =100, w=1000, =150, c=150.

[0071] (4) Initialization of quantities that need to be maintained during the iterative solution process;

[0072] Create a list of active nodes and add the original programming problem as a node to the list. Solve the linear relaxation problem corresponding to the original programming problem, record the relaxation solution S and utility score E, and store them as node information in the node. Set the initial optimal utility score. =0, initial optimal solution Empty.

[0073] After the initialization process is complete, proceed to... Figure 1 The iterative solution process shown below includes the following specific steps:

[0074] Step 1: Check if the node list is not empty. If the node list is empty, end the solution process and find the current optimal solution. This is the final solution to the planning problem. If the list is not empty, proceed to step 2.

[0075] Step 2: Select the node with the largest utility score E corresponding to the relaxation solution and remove this node from the node list.

[0076] Step 3: If the utility score E of this node is less than the current optimal utility score If the result is positive, prune the node and return to step 1; otherwise, proceed to step 4.

[0077] Step 4: If the relaxation solution S corresponding to this node is an integer solution, then prune this node. At this point, if the utility score E of this node is greater than the current optimal utility score... Then let , Then return to step 1. If S is a non-integer solution, proceed to step 5.

[0078] Step 5: Attempt to add a cutting plane. If a valid cutting plane is found, solve the relaxation problem corresponding to the new planning problem after adding the cutting plane, and return to the judgment in Step 3. If no valid cutting plane is found, branching is required, and proceed to Step 6.

[0079] Step 6: Select the decision variable y that is closest to 0.5 in the relaxation solution, and then add constraints y=0 and y=1 respectively to form two new nodes. Solve the linear relaxation problem corresponding to the two new nodes. If there is a feasible relaxation solution, record the relaxation solution S and the corresponding utility score E.

[0080] Step S3: Enhance the llama.cpp framework to support operator-level tensor allocation. Based on the optimal operator-level resource allocation scheme, deploy each operator of the iFlytek large model to the corresponding GPU or CPU within the enhanced llama.cpp framework. Specifically, in the llama.cpp model parameter loading section, add a processing module that specifies the buffer list to be used when creating a specific parameter tensor. When creating parameters, determine whether it is a special layer that is not allocated to the same device. The processing of non-special layers remains unchanged. In special layers, for tensors that need to be allocated on the GPU, use the corresponding GPU's buffer list to create parameter tensors and place the corresponding parameters on the GPU. The processing of other tensors remains the same as in the original llama.cpp framework. In this way, the llama.cpp framework can achieve operator-level resource allocation.

[0081] The Spark Open Source-13B model was deployed on an Nvidia 2×1080Ti server. The layer allocation scheme was as follows: layers 0-15 were deployed on GPU0, layers 16-31 on GPU1, and the remaining layers were deployed on the CPU. Model information was obtained from the local iFlytek large model gguf file using the method described in the technical solution section. Available device storage information was also obtained. The fixed computational overhead was set to 1053.44MB for GPU0 and 459.44MB for GPU1, and the recommended utility function coefficients were used. The operator allocation results obtained are shown in Table 1.

[0082] Table 1

[0083]

[0084] Inference tests were performed using the enhanced llama.cpp framework, applying both the original hierarchical allocation method and the solved operator-level allocation scheme. The inference runs used the default parameters of llama.cpp. The throughput comparison results between the hierarchical and operator-level allocation schemes are shown below. Figure 2 , Figure 3 As can be seen, deploying the iFlytek large model using the operator-level allocation scheme obtained by solving the planning problem resulted in a significant improvement in model throughput during both the pre-filling and decoding processes compared to the layer-based allocation scheme. Furthermore, the GPU1 memory utilization rate was 93.6% during layer-based allocation inference, compared to 98.7% during inference using the operator-level allocation scheme, also showing an improvement.

[0085] In a second aspect, the present invention provides an electronic device comprising: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method for deploying large models of an enhanced llama.cpp framework.

[0086] Thirdly, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enable the processor to implement the aforementioned method for deploying large models of the enhanced llama.cpp framework.

[0087] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for deploying large models using an enhanced llama.cpp framework, characterized in that, include: Step S1: Construct a 0-1 integer linear programming model with the objective of maximizing model throughput. The programming model includes decision variables for representing operator-level device allocation, resource and order constraints to ensure the feasibility of the scheme, and a utility function consisting of operator GPU allocation score and cohesion reward within the same device. Step S2: Solve the planning model using the branch and bound method to obtain the optimal operator-level resource allocation scheme; Step S3: Enhance the llama.cpp framework to support operator-level tensor allocation, and according to the optimal operator-level resource allocation scheme, deploy each operator of the large model to the corresponding GPU or CPU in the enhanced llama.cpp framework.

2. The method for deploying large models of the enhanced llama.cpp framework according to claim 1, characterized in that, In step S1, the decision variables include operator allocation variables, whole-layer allocation variables, and operator-device cohesion variables.

3. The method for deploying large models of the enhanced llama.cpp framework according to claim 2, characterized in that, The operator allocation variable indicates that a specific operator of a specific layer is allocated to a specific device. The whole-layer allocation variable indicates whether a specific layer is allocated to a specific device. The operator-device cohesion variable indicates whether the qkv operator of a specific layer is deployed on the same GPU as other operators in the same layer.

4. The method for deploying large models of the enhanced llama.cpp framework according to claim 1, characterized in that, In step S1, the constraints include single operator allocation constraints, definition and implementation constraints of whole-layer allocation variables and operator cohesion variables within the same device, resource constraints, and allocation order constraints.

5. A method for deploying large models of an enhanced llama.cpp framework according to claim 4, characterized in that, The single operator allocation constraint ensures that each operator must be allocated to one and only one device; the device resource capacity constraint ensures that the total memory usage of operators allocated to each device does not exceed the available capacity of that device. Allocation order constraints are used to regulate the allocation order of inter-layer operators.

6. A method for deploying large models of an enhanced llama.cpp framework according to claim 1, characterized in that, In step S1, the utility function serves as the optimization objective of the planning model. The utility function is composed of the weighted sum of the scores allocated to the GPU by the operator, the scores allocated to the GPU by the entire layer, and the reward scores for the cohesion between the operator and the device.

7. A method for deploying large models of an enhanced llama.cpp framework according to claim 1, characterized in that, Step S2 includes: Initialization process: Create a list of active nodes and add the original planning problem as the initial node to the list, solve the linear relaxation problem of the node, record the relaxation solution and the corresponding utility score, and set the initial optimal solution to empty; Node processing loop: When the list of active nodes is not empty, the following steps are executed in a loop: Select and remove the node with the best relaxation solution from the list; if the utility score corresponding to the relaxation solution of the node is lower than the current best score, then prune the node; if it is not pruned and the relaxation solution of the node is an integer solution, then compare it with the current best solution and update it; if it is not pruned and the relaxation solution is a non-integer solution, then first try to add cutting plane conditions to tighten it. If it cannot be tightened further, then branch: Select a decision variable with a value closest to 0.5, and generate two new child nodes by constraining the variable to 0 or 1 respectively, and add them to the list of active nodes; Output result: When the list of active nodes is empty, the currently stored optimal solution is the optimal operator-level resource allocation scheme.

8. A method for deploying large models of an enhanced llama.cpp framework according to claim 1, characterized in that, In step S3, a processing module is added to the parameter loading section of the llama.cpp framework to specify the list of buffers to be used when creating a tensor with specific parameters.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When one or more programs are executed by the one or more processors, the one or more processors implement a large model deployment method for an enhanced llama.cpp framework as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed by a processor, enable the processor to implement a large model deployment method for an enhanced llama.cpp framework as described in any one of claims 1-8.

Images