Large language model reasoning optimization method and device, electronic equipment and storage medium
By combining white-box modeling and reinforcement learning with an optimization method that utilizes offloading strategies, the problem of high inference resource requirements for large language models is solved, improving throughput and overall performance, and adapting to changes in hardware and load.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SUN YAT SEN UNIV
- Filing Date
- 2024-09-27
- Publication Date
- 2026-06-02
Smart Images
Figure CN119150994B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, electronic device, and storage medium for optimizing reasoning in large language models. Background Technology
[0002] With the rapid increase in the number of parameters in large language models, the required GPU memory is also gradually increasing, posing a significant challenge to the use of computing and storage resources. Today's emerging large language models have billions or even trillions of parameters. During model inference, a large amount of GPU memory is needed to store model weights and intermediate computation results. However, it is undeniable that increasing the number of model parameters can bring beneficial effects to improving model performance and solving complex problems. Therefore, the enormous computing and storage resources consumed during the inference and training of large models are a problem that cannot be ignored.
[0003] For large language model inference, current approaches primarily focus on reducing computational and storage resource consumption through model compression, collaborative inference, and offloading strategies. Model compression aims to reduce the computational and storage resources required by the model by reducing model parameters or simplifying its structure, while ensuring model performance within certain limits. Collaborative inference utilizes the cooperation of multiple devices or models to jointly complete the inference task, reducing the burden on individual devices and models, thereby reducing overall resource consumption. The core idea of offloading strategies is to offload computational tasks concentrated on one device to other devices, thereby reducing device load and achieving efficient resource allocation.
[0004] Of the methods described above, the first two typically assume that the model is adapted to GPU memory, making it difficult to run models with a large number of parameters using only a single consumer-grade GPU. In recent systems using the third method, acceptable throughput cannot be achieved on a single GPU due to inefficiencies in I / O scheduling and tensor placement. Summary of the Invention
[0005] This invention provides a method, apparatus, electronic device, and storage medium for optimizing large language model inference, which solves or partially solves the problem of how to reduce the huge resources required for large model inference due to the large number of parameters in existing related technologies.
[0006] This invention provides a method for optimizing reasoning in large language models, the method comprising:
[0007] Real-time acquisition of configuration parameter data and computing resource data of large language models during model inference;
[0008] Perform static optimization analysis on the configuration parameter data and output the configuration analysis results;
[0009] By combining the configuration parameter data with the configuration analysis results, automatic tuning of the configuration parameters based on reinforcement learning is performed to obtain the optimal parameter configuration;
[0010] The computing resource data is dynamically optimized based on an unloading strategy to obtain the optimal resource configuration;
[0011] Based on the optimal parameter configuration and the optimal resource configuration, the parameter configuration and resource allocation of the large language model during model inference are dynamically optimized.
[0012] The present invention also provides a large language model inference optimization device, comprising:
[0013] The data acquisition module is used to acquire configuration parameter data and computing resource data of the large language model in real time during model inference;
[0014] The static optimization analysis module is used to perform static optimization analysis on the configuration parameter data and output the configuration analysis results.
[0015] The automatic parameter tuning module is used to combine the configuration parameter data with the configuration analysis results to automatically tune the configuration parameters based on reinforcement learning, so as to obtain the optimal parameter configuration.
[0016] The dynamic optimization processing module is used to perform dynamic optimization processing on the computing resource data based on the unloading strategy to obtain the optimal resource configuration;
[0017] The model inference optimization module is used to dynamically optimize the parameter configuration and resource allocation of the large language model during model inference based on the optimal parameter configuration and the optimal resource configuration.
[0018] The present invention also provides an electronic device, the device comprising a processor and a memory:
[0019] The memory is used to store program code and transmit the program code to the processor;
[0020] The processor is used to execute the large language model inference optimization method as described above, according to the instructions in the program code.
[0021] The present invention also provides a computer-readable storage medium for storing program code for executing the large language model inference optimization method as described in any of the preceding claims.
[0022] As can be seen from the above technical solutions, the present invention has the following advantages:
[0023] This invention provides a method for optimizing large language model inference. It involves real-time acquisition of configuration parameter data and computational resource data of the large language model during inference; static optimization analysis of the configuration parameter data, outputting the analysis results to facilitate user modification of certain parameters to optimize the inference performance of the large language model; automatic tuning of configuration parameters based on reinforcement learning, combining the configuration parameter data and the analysis results to obtain the optimal parameter configuration for further optimization of certain parameters; dynamic optimization of computational resource data based on an offloading strategy to obtain the optimal resource configuration, enabling flexible configuration under hardware resource constraints; and dynamic optimization of parameter configuration and resource allocation for the large language model during inference based on the optimal parameter and resource configurations. This invention not only improves the throughput of large model inference and enables automatic tuning of configuration parameters, but also helps users adjust certain configuration parameters during subsequent model inference to improve the overall performance of the large model inference process. Attached Figure Description
[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0025] Figure 1 A flowchart illustrating the steps of an optimization method for reasoning in a large language model.
[0026] Figure 2 This is a schematic diagram of the processing flow in a static optimization module based on white-box modeling.
[0027] Figure 3 A schematic diagram of the processing flow in the automatic parameter tuning module for a large model based on reinforcement learning algorithm;
[0028] Figure 4 This is a schematic diagram of the processing flow in a dynamic optimization module based on an unloading strategy.
[0029] Figure 5 This is a schematic diagram of a pre-defined three-level storage structure;
[0030] Figure 6 This is a computational graph representation of a large language model during model inference.
[0031] Figure 7 This is a structural block diagram of a large language model reasoning optimization device. Detailed Implementation
[0032] This invention provides a method, apparatus, electronic device, and storage medium for optimizing large language model inference, which addresses or partially addresses the problem in existing related technologies where the large number of parameters leads to high computational and storage resource requirements for model inference, and how to reduce the huge resources required for large model inference.
[0033] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.
[0034] As an example, current approaches to reducing computational and storage resource consumption for large language model inference primarily focus on model compression, collaborative inference, and offloading strategies. Model compression and collaborative inference typically assume that the model is adapted to GPU memory, making it difficult to run large models with a large number of parameters using only a single consumer GPU. Specifically, model compression aims to reduce memory footprint and computational complexity by decreasing the number or precision of model parameters, making it easier to deploy on resource-constrained devices. However, when faced with deep learning models with millions or even billions of parameters, even after compression, the model size may still exceed the memory capacity of a single consumer GPU. Collaborative inference involves distributing computational tasks across multiple devices to achieve a more efficient inference process. This approach typically assumes that all participating devices have sufficient memory and computing power to process a portion of the model. However, in reality, consumer GPUs have limited memory and cannot independently handle the complete inference task of large models. Therefore, both approaches still face challenges in practical applications and require further optimization and innovation to effectively run large-scale models in resource-constrained environments.
[0035] In state-of-the-art systems based on offload strategies, inefficiencies in input / output scheduling and tensor placement prevent the achievement of acceptable throughput on a single GPU. Specifically, input / output scheduling involves effectively managing the transfer of data from storage devices to GPU memory. Inefficient scheduling strategies can create bottlenecks in data transfer, preventing the GPU from fully utilizing its computational power and impacting overall system performance. Tensor placement refers to the proper allocation and management of computational units for deep learning models within GPU memory. Inappropriate tensor placement strategies can lead to memory fragmentation or frequent memory swapping, both of which significantly reduce system efficiency. These inefficiencies work together to underutilize GPU resources, resulting in throughput that falls short of expectations when processing large amounts of data and complex computational tasks. Therefore, even though state-of-the-art offload strategies theoretically improve performance, the inefficiencies in input / output scheduling and tensor placement remain a critical bottleneck in practical applications, requiring further optimization and improvement to achieve acceptable throughput on a single GPU.
[0036] Furthermore, if only white-box modeling techniques are used to build a simulator for a large language model, the resulting analysis is derived from pre-established linear equations, making it unable to adapt to new data or environmental changes during runtime. This means the model cannot learn online, i.e., it cannot dynamically adjust and optimize itself to adapt to new situations during system operation. Consequently, it cannot utilize effective optimization algorithms for back-optimization of configuration parameters. Back-optimization is a process that uses optimization algorithms to find optimal configuration parameters based on an objective function and constraints. Because the linear equations of the white-box model are predetermined and cannot be adjusted at runtime, the model cannot adaptively optimize configuration parameters under new data or conditions.
[0037] Therefore, one of the core inventive points of this invention is: addressing the shortcomings of existing technologies and reducing the enormous resources required for model inference, a large language model inference optimization method based on an offloading strategy is proposed. First, static optimization analysis of configuration parameters is performed based on white-box modeling. A series of linear equations combined with model and graphics card configuration information provide quantitative analysis results, facilitating user modification of certain parameters to optimize the inference performance of the large language model. Simultaneously, reinforcement learning algorithms are used to achieve automatic parameter tuning for further optimization of certain configuration parameters. Second, dynamic optimization is achieved based on the offloading strategy. By applying the offloading strategy, computing and storage resources from the CPU, GPU, and hard disk are aggregated and flexibly configured under hardware resource constraints. This invention not only improves the throughput of large model inference and enables automatic tuning of configuration parameters, but also helps users adjust certain configuration parameters during subsequent model inference to improve the overall performance of the large model inference process. The method proposed in this invention can provide strong support for optimizing the performance of large language model systems, improving system reliability and efficiency, and promoting the application of large language models in various fields.
[0038] The technical solution of this invention mainly consists of three parts: a static optimization module based on white-box modeling technology, an automatic tuning module for large model configuration parameters based on reinforcement learning algorithms, and a dynamic optimization module based on an unloading strategy. Through the processing of these three modules, an overall framework for accelerating large language model inference can be formed, which improves the throughput of large model inference to a certain extent, reduces the computational and storage resources consumed by the system during large model inference, and provides strong support for the subsequent construction of a large language model inference simulation and optimization system.
[0039] Reference Figure 1 The diagram illustrates a flowchart of a large language model inference optimization method provided by an embodiment of the present invention, which may specifically include the following steps:
[0040] Step 101: Obtain the configuration parameter data and computing resource data of the large language model during model inference in real time;
[0041] Configuration parameter data mainly includes the system's GPU configuration information, the model configuration information and model architecture information of the large language model. Computational resource data refers to the computing and storage resources from CPU, GPU and hard disk obtained through aggregation.
[0042] For processing configuration parameter data, in the specific implementation, configuration parameters such as model configuration parameters, performance parameters, and GPU configuration information can be organized into a JSON file. Specifically, model configuration information can include the model name, model architecture, and model type. Model architecture includes the number of model layers, hidden layer dimension, feedforward neural network intermediate layer dimension, maximum sequence length, number of attention heads, and vocabulary length. GPU configuration information includes the GPU name, GPU device memory size, GPU device memory bandwidth, bandwidth within GPU device nodes, bandwidth between GPU device nodes, peak GPU performance for single-precision computation, peak GPU performance for half-precision computation, peak GPU performance for 8-bit integer computation, peak GPU performance for 4-bit integer computation, and minimum message latency within GPU nodes. Model configuration information can be downloaded from the Hugging Face community (an open-source community) for large language models; refer to the config files there for organization.
[0043] Step 102: Perform static optimization analysis on the configuration parameter data and output the configuration analysis results;
[0044] This step primarily utilizes a static optimization module built using white-box modeling technology to perform static optimization analysis. In its implementation, static optimization analysis is performed on configuration parameter data, outputting configuration analysis results. This can be achieved by inputting GPU configuration information, model configuration information, and model architecture information into a pre-built multi-layered analysis model (which can also be understood as a systematic linear equation) based on white-box modeling for static optimization analysis, outputting a series of configuration analysis results. The static optimization analysis includes inference configuration analysis, parameter quantity analysis, computational quantity analysis, memory analysis, and latency analysis.
[0045] Figure 2 This paper illustrates a schematic diagram of the processing flow in a static optimization module based on white-box modeling provided in an embodiment of the present invention. To enable those skilled in the art to better understand the analysis process of step 102, the following will elaborate on the linear equations applied and the specific analysis in the static optimization module based on the five parts of theoretical performance analysis results obtained through analysis.
[0046] (1) Analysis of the main reasoning configurations of the large language model
[0047] For the main inference configuration analysis of large language models, the first step is to create an inference configuration dictionary. This dictionary mainly consists of two parts: the inference configuration part and the GPU configuration part.
[0048] Specifically, the inference configuration includes the model name, batch size per GPU, length of the input sequence, thread pool size, pipeline parallelism size, number of tokens generated, and whether a KV Cache (Key-Value Cache, a data storage technology for fast access and retrieval of data) is used.
[0049] It's worth noting that pooling technology refers to pre-storing some resources to avoid insufficient resources during use, while improving resource reuse efficiency. This is especially true in scenarios with high request volumes, significantly improving overall system performance and optimizing resource consumption. Thread pools utilize pooling technology. A thread pool starts several threads, which are in a sleep state. When specific tasks are needed, one of the threads is started. Once the task is completed, the thread returns to the sleep state. The meanings of the various parameters output by the inference configuration section are shown in Table 1 below:
[0050]
[0051] Table 1: Meaning of parameters output by the inference configuration section
[0052] The GPU configuration section includes the GPU name, GPU memory size, GPU memory bandwidth, GPU node bandwidth, and GPU floating-point operations per second.
[0053] It's worth noting that GPU memory bandwidth describes the data transfer rate within the GPU, typically expressed as the amount of data transferred per second. HBM, on the other hand, is a newer GPU memory chip that significantly improves performance and bandwidth. GPU node bandwidth describes the data transfer rate between the GPU and other system components (including the CPU and other GPUs), also typically expressed as the amount of data transferred per second. GPU floating-point operations per second (TFLOPS) are usually measured in TFLOPS. TFLOPS represents trillions of floating-point operations performed per second.
[0054] Finally, the information in the inference configuration dictionary is printed in an unsorted manner. That is, the order of the key-value pairs in the dictionary is not sorted; they are output in the order they were initially inserted into the dictionary. The meanings of the various parameters output in the GPU configuration section are shown in Table 2 below:
[0055]
[0056] Table 2: Meaning of parameters output in the GPU configuration section
[0057] It is worth noting that in the Transformer model, the feedforward neural network exists not only in the encoder but also in the decoder. The feedforward neural network consists of two linear layers. The activation function of the first linear layer is ReLU, while the second layer does not use an activation function. The output dimension of the first layer is the same as the dimension of the intermediate layers of the feedforward neural network. The output dimension of the second layer is the same as the dimension of the hidden layers in the model. These two parameters are mentioned in the detailed information of each model. The model configuration information file contains several details about the model. The information contained in each model is shown in Table 3 below:
[0058]
[0059] Table 3: Model information contained in the model configuration information file
[0060] The GPU configuration information file contains several details about the graphics card. The information for each graphics card is shown in Table 4 below:
[0061]
[0062] Table 4: Graphics card information contained in the GPU configuration information file
[0063] (2) Parameter analysis of large language models
[0064] The following section will introduce the specific calculation formulas for the parameter analysis of large language models.
[0065] The attention module has four linear layers: query, key-value projection, and output matrix. The number of parameters per layer is calculated by adding the number of parameters from the four linear layers. The result is also four times the square of the hidden layer dimension `hidden_dim`. The formula for calculating the number of parameters `params_attn` per layer is as follows:
[0066]
[0067] The number of parameters in a linear layer of a multilayer perceptron includes the intermediate and output matrices. Specifically, the number of parameters in a linear layer of a multilayer perceptron, params_mlp, is eight times the square of the hidden layer dimension of the model. The calculation formula is as follows:
[0068]
[0069] For the parameter quantity params_layernorm of each layer in the two standardized modules, if ln_ignore is True, then params_layernorm is 0; otherwise, its calculation formula is as follows:
[0070]
[0071] Here, ln_ignore indicates whether to ignore the number of parameters in the standardization module.
[0072] In the Transformer model, the number of parameters per layer of the decoder module mainly includes the number of parameters per layer of the attention module and the number of parameters in the linear layers of the multilayer perceptron. Since the primary considerations are the number of parameters per layer of the attention module and the number of parameters in the linear layers of the multilayer perceptron, the parameter `ln_ignore` is set. The default value of `ln_ignore` is `True`. If `ln_ignore` is `True`, only the number of parameters per layer of the attention module plus the number of parameters in the linear layers of the multilayer perceptron are considered. Otherwise, the number of parameters per layer of the normalization module needs to be considered. In summary, when `ln_ignore` is `False`, the formula for calculating the number of parameters per layer of the decoder module in the Transformer model, `params_per_layer`, is as follows:
[0073]
[0074] When ln_ignore is True, the calculation formula for params_per_layer is as follows:
[0075]
[0076] Regarding the number of parameters in the embedding layer, the parameter shared_embedding indicates whether the input and output embedding layers share parameters, with a default value of True.
[0077] The formula for calculating the number of input embedding layer parameters, num_params_input_embedding, is as follows:
[0078]
[0079] When the input and output embedding layers share parameters, the number of parameters in the output embedding layer is 0. Otherwise, the formula for calculating the number of parameters in the output embedding layer, num_params_output_embedding, is as follows:
[0080]
[0081] The calculation of the total number of model parameters needs to consider all layers in the model, including the embedding layers. It is assumed that the input and output embedding layers share parameters, and the parameter count of each layer in the normalization module is not considered. The specific formula for calculating the total number of model parameters, params_model, is as follows:
[0082]
[0083] Here, params_embedding represents the number of parameters in the embedding layer.
[0084] (3) Computational analysis of large language models
[0085] The computational complexity analysis of large language models depends only on the chosen model and is independent of the parallelization strategy employed. Furthermore, the computational complexity of the normalization module and other element-wise operations are ignored.
[0086] The floating-point computational cost (flops_attn) during the forward propagation of the attention module in the Transformer layer mainly includes linear computations such as querying, key-value projection, output matrix multiplication, and internal self-attention operations. The specific formula for calculating flops_attn is shown below:
[0087]
[0088] Where flops_q represents the computational cost required for the query operation; flops_k represents the computational cost required for the key projection operation; flops_v represents the computational cost required for the value projection operation; flops_output represents the computational cost required for the output matrix multiplication operation; and flops_self_attention represents the computational cost required for the self-attention internal operation.
[0089] The computational load (flops_mlp) in the multilayer perceptron module mainly consists of two floating-point operations in matrix multiplication. The specific calculation formula for flops_mlp is shown below:
[0090]
[0091] Here, both flops_fc1 and flops_fc2 refer to floating-point operations in matrix multiplication.
[0092] Regarding the computational cost per layer during forward propagation, `flops_fwd_per_layer`, the main analysis approach is to add the floating-point computational cost of the attention module in the Transformer layer during forward propagation to the computational cost of the multilayer perceptron module. The specific calculation formula for `flops_fwd_per_layer` is shown below:
[0093]
[0094] Here, flops_layernorm refers to the computational cost of the normalization module.
[0095] Analyzing the computational cost of floating-point operations during the forward propagation of a Transformer model (flops_model) requires considering the computational cost of floating-point operations in the output label logits layer. It's worth noting that the output label logits layer is typically the last layer in models used in natural language processing, and its output is a vector of the same length as the vocabulary. Each value in the vector represents the probability distribution of the corresponding word in the current context. During model training, these values can be compared with the true values to calculate the loss function. During model inference, the word with the highest probability can be selected as the model's current output.
[0096] Furthermore, activation functions are typically not used in the output label logits layer to preserve the numerical magnitude of the probability distribution. The specific calculation formula for flops_model is shown below:
[0097]
[0098] Here, flops_logits_layer refers to the amount of floating-point computation in the output marker logits layer.
[0099] (4) Memory analysis of large language models
[0100] The first part is an analysis of the memory usage and maximum batch size during the pre-filling phase.
[0101] First, the specific calculation formula for the memory size occupied by the model weights, weight_memory_per_gpu, is as follows.
[0102]
[0103] Here, `bytes_per_param` represents the storage precision of the model weights, with a default value of FP16; `embedding_dtype_bytes` represents the data type of the embedding layer, with a default value of FP16; `params_embedding` represents the sum of the input and output embedding layer parameters. By default, the input and output embedding layers share parameters, and the output embedding layer has 0 parameters. Therefore, the value of `params_embedding` is... Based on the analysis above, with the default value of ln_ignore set to True, the value of params_per_layer is... .
[0104] Next, we analyze the memory size occupied by the activation values in the pre-filling stage. For calculating the memory size of each layer's activation values, if the activation value recalculation flag is not FULL, then the memory size occupied by each layer's activation values is 0. If the activation value recalculation flag is FULL, the specific value will be affected by the batch size.
[0105] If the specified batch size is 1, the following formula can be used for calculation:
[0106]
[0107] Among them, prefill_activation_memory_batch_size_1 refers to the memory size occupied by the model activation values in the prefilling stage when the specified batch size is 1; num_layers_per_gpu refers to the number of layers placed on each GPU in a multi-layer neural network.
[0108] If a specific batch size is specified, the memory size occupied by the activation value in this case, prefill_activation_memory_per_gpu, can be calculated using the following formula:
[0109]
[0110] The maximum batch size for the prefilling stage, prefill_max_batch_size_per_gpu, can be calculated using the following formula:
[0111]
[0112] Among them, memory_left refers to the remaining available video memory; gpu_memory_in_GB refers to the total video memory capacity on each GPU; and weight_memory_per_gpu refers to the amount of video memory used to store model weights on each GPU.
[0113] The second part is an analysis of the memory usage and maximum batch size during the decoding phase.
[0114] For the decoding stage, the analysis of memory and maximum batch processing size needs to be discussed in two cases, with the use of KV Cache as the dividing criterion.
[0115] First, let's analyze the memory size occupied by the activation values during the decoding phase, `decode_activation_memory_per_gpu`. If a KV cache is used, `decode_activation_memory_per_gpu` can be calculated using the following formula:
[0116]
[0117] Because a KV Cache is used, the seq_len value here is 1, which is the newly generated token.
[0118] If a KV cache is not used, the size of the parameter seq_len is... That is, the context length is no longer the newly generated token, but... The specific calculation formula for decode_activation_memory_per_gpu at this time is as follows:
[0119]
[0120] Secondly, we analyze the memory size (kv_cache_memory_per_gpu) occupied by the KV cache during the decoding stage. When using a KV cache, we first need to calculate the memory (in bytes) required to store the Transformer layers' KV cache during model inference. The result needs to be multiplied by the number of neural network layers processed on each GPU device. The specific formula for calculating kv_cache_memory_per_gpu is shown below:
[0121]
[0122] kv_cache_dtype_bytes refers to the number of bytes occupied by the data type storing the key-value pairs in the cache.
[0123] If the KV Cache is not used, the value of kv_cache_memory_per_gpu is 0.
[0124] Next, we analyze the total memory consumption of the decoding phase, decode_memory_total. The specific calculation formula is shown below:
[0125]
[0126] When using a KV cache, the specific calculation formula for decode_memory_total is as follows:
[0127]
[0128] Without using a KV cache, the specific calculation formula for decode_memory_total is as follows:
[0129]
[0130] Next, we analyze the maximum batch size during the decoding stage: decode_max_batch_size_per_gpu. When using a KV cache, decode_max_batch_size_per_gpu can be calculated using the following formula:
[0131]
[0132] Here, decode_activation_memory_batch_size_1 refers to the memory size occupied by the model activation when processing a single batch of data; kv_cache_memory_batch_size_1 refers to the memory size occupied by the model's key-value cache when processing a single batch of data.
[0133] Without using a KV cache, decode_max_batch_size_per_gpu can be calculated using the following formula:
[0134]
[0135] (5) Delay analysis of large language models
[0136] The first part is a detailed analysis of the pre-filling stage delay. The specific calculation formula is shown below:
[0137]
[0138] Among them, latency_attn refers to the computation latency in the self-attention mechanism; compute_latency refers to the computation processing latency; memory_latency refers to the memory access latency; floats refers to the number of floating-point operations; memory refers to the amount of memory storage; latency_mlp refers to the computation latency of the feedforward neural network part in the Transformer; and latency_layernorm refers to the latency of normalizing the computation layer.
[0139] The normalization module is not a computationally intensive module, therefore `compute_latency` in the formula is 0. Also, the normalization module has no matrix weights, only vector weights, which are negligible; therefore, `memory_latency` in the formula is also 0. In summary, `latency_layernorm` is 0. It is worth noting that there are two normalization layers here.
[0140] The following calculations cover the latency of a single AllReduce communication within the tensor parallel group during the forward pass of the Transformer layer. Here, AllReduce refers to a class of algorithms aimed at efficiently integrating data results from different machines and distributing the integrated results to each machine. This integration can involve operations such as summation, finding the maximum or minimum value, etc. Furthermore, the communication volume of a single AllReduce operation is... It is worth noting that if tp_size is 1, then latency_tp_comm is 0. If tp_size is not 1, then the latency_tp_comm of tensor parallel communication can be calculated using the following formula:
[0141]
[0142] Here, latency_per_all_reduce refers to the latency during each AllReduce operation.
[0143] In the Transformer model, the latency per layer of the decoder module can be calculated using the following formula:
[0144]
[0145] The results can be substituted into latency_attn, latency_mlp, latency_layernorm, and latency_tp_comm respectively.
[0146] The delay prefill_latency_fwd_attn during the model forward propagation phase for calculating the self-attention mechanism can be calculated using the following formula:
[0147]
[0148] Here, `prefill_latency_fwd_mlp` refers to the latency in the feedforward neural network; `prefill_latency_fwd_layernorm` refers to the latency of the normalization operation in the computation layer; and `prefill_latency_fwd_tp_comm` refers to the latency of data communication between different computing devices during tensor parallelization. `latency_attn` can be calculated using the formula derived above and substituted into the equation.
[0149] prefill_latency_fwd_mlp can be calculated using the following formula:
[0150]
[0151] The result of latency_mlp can be calculated using the formula derived above and then substituted into the input.
[0152] Since latency_layernorm is 0, prefill_latency_fwd_layernorm is also 0.
[0153] prefill_latency_fwd_tp_comm can be calculated using the following formula:
[0154]
[0155] The result of latency_tp_comm can be calculated using the formula derived above and then substituted into the input.
[0156] The specific calculation formula for the input embedding layer forward propagation delay, prefill_latency_fwd_input_embedding, is as follows:
[0157]
[0158] Among them, memory_latency refers to the latency when accessing computer memory; comm_latency refers to the latency when communicating data in a distributed computing system.
[0159] For the analysis of the forward pass of the output embedding layer, considering that the operation in this part is computationally intensive, the main calculation is the unnormalized probability, i.e., logits.
[0160] The specific calculation formula for the output embedding layer forward propagation delay prefill_latency_fwd_output_embedding_loss is as follows:
[0161]
[0162] Here, flops_efficiency refers to the ratio of the actual number of operations to the theoretical peak number of operations.
[0163] The formula for calculating the prefill_latency generated by the output of the first token in the prefill phase is as follows:
[0164]
[0165] The results of latency_per_layer, prefill_latency_fwd_input_embedding, and prefill_latency_fwd_output_embedding_loss can be calculated using the formulas derived above and then substituted into the equation.
[0166] The second part is a detailed analysis of the latency during the decoding stage. The specific formula for calculating latency during the decoding stage is shown below (regardless of whether a KV cache is used, the following formula is used for calculation):
[0167]
[0168] For the standardized module, as can be seen from the previous analysis, latency_layernorm is 0.
[0169] As can be seen from the previous analysis, if tp_size is 1, then latency_tp_comm is 0; if tp_size is not 1, regardless of whether a KV cache is used, it can be calculated using the following formula:
[0170]
[0171] The computational latency of the self-attention mechanism in the decoder, decode_latency_fwd_attn, can be calculated using the following formula:
[0172]
[0173] The latency_attn can be calculated using the formula derived above and substituted into the input. decode_latency_fwd_mlp refers to the computation latency of the feedforward neural network in the decoder; decode_latency_fwd_layernorm refers to the computation latency of the layer normalization operation in the decoder; decode_latency_fwd_tp_comm refers to the communication latency in the tensor parallelization process in the decoder.
[0174] decode_latency_fwd_mlp can be calculated using the following formula:
[0175]
[0176] The result of latency_mlp can be calculated using the formula derived above and then substituted into the input.
[0177] Since latency_layernorm is 0, decode_latency_fwd_layernorm is also 0.
[0178] decode_latency_fwd_tp_comm can be calculated using the following formula:
[0179]
[0180] The result of latency_tp_comm can be calculated using the formula derived above and then substituted into the input.
[0181] For the analysis of the input embedding layer forward pass, regardless of whether a KV cache is used, the input embedding layer forward pass latency decode_latency_fwd_input_embedding in the decoder can be calculated using the following formula:
[0182]
[0183] For the analysis of the output embedding layer forward pass, if a KV cache is used, the formula for calculating the output embedding layer forward pass delay decode_latency_fwd_output_embedding_loss in the decoder is as follows:
[0184]
[0185] If a KV cache is not used, the specific calculation formula for decode_latency_fwd_output_embedding_loss is as follows:
[0186]
[0187] The following calculates the latency of the key-value cache forward pass in the Transformer layer. If no KV cache is used, the latency is 0. If a KV cache is used, the value of context_len is calculated from... Traversal to The amount of memory (kv_cache_memory_per_gpu) used per GPU for caching the key and value vectors in the self-attention mechanism is calculated according to the following formula:
[0188]
[0189] Here, context_len refers to the length of the context that the model considers when processing the input sequence.
[0190] Calculate the latency kv_cache_latency associated with using the key-value cache using the following formula:
[0191]
[0192] Among them, hbm_memory_efficiency refers to the ratio of actual available memory to total memory.
[0193] The values of kv_cache_memory_per_gpu and kv_cache_latency obtained in each loop are added to the lists kv_cache_memory_list_per_gpu and kv_cache_latency_list, respectively. The values of kv_cache_avg_latency and kv_cache_peak_latency are the average and maximum values of the list kv_cache_latency_list, respectively.
[0194] Furthermore, decode_avg_latency and decode_peak_latency can be calculated using the following formulas:
[0195]
[0196]
[0197] Here, `kv_cache_memory_list_per_gpu` refers to the list of key-value cache memory amounts on each GPU; `kv_cache_latency_list` refers to the list of key-value cache latency under different context lengths; `kv_cache_avg_latency` refers to the average latency of key-value cache operations; `kv_cache_peak_latency` refers to the maximum latency of key-value cache operations; `decode_avg_latency` refers to the average latency of the decoding process; `decode_peak_latency` refers to the maximum latency of the decoding process; and `decode_model_latency` refers to the total latency of the entire decoding stage.
[0198] Thus, the automatically generated series of analysis results can help users adjust certain configuration parameters during subsequent inference to improve the overall performance of the large language model inference process. At the same time, it can help users estimate the computing and storage resources required for inference so as to select more suitable hardware resources.
[0199] In this step, a relatively complete theoretical performance analysis model for large language models was constructed based on white-box modeling. In practical applications, a series of analysis results can be provided by combining the model and graphics card configuration information, thus enabling a better simulation of the reasoning process of large language models.
[0200] Step 103: Combine the configuration parameter data with the configuration analysis results to perform automatic optimization of the configuration parameters based on reinforcement learning, and obtain the optimal parameter configuration;
[0201] This step primarily utilizes a large-model configuration parameter auto-tuning module based on reinforcement learning algorithms to automatically create and manage tuning instances. It uses dynamic context information and feedback data to continuously optimize application configuration parameters to maximize performance metrics and adapt to changes in hardware and workload. A flowchart illustrating this process is shown below. Figure 3 As shown.
[0202] Tuning instances refer to the process of optimizing and tuning parameters in computer systems, software, or applications. The goal is to improve performance, reduce latency, increase throughput, or improve other key metrics.
[0203] Specifically, the process of automatically optimizing configuration parameters based on reinforcement learning by combining configuration parameter data and configuration analysis results to obtain the optimal parameter configuration can be achieved by executing the following steps S01 to S05:
[0204] Step S01: Extract the tunable configuration parameter information of the application from the configuration parameter data and configuration analysis results, and create a tuning instance at the same time;
[0205] The application first creates a tuning instance through the configuration parameter auto-tuning module, providing a list of parameters and their metadata during the call. This data includes the configuration parameter data from the preceding steps (i.e., model configuration parameters, performance parameters, GPU configuration information) and configuration analysis results, as well as adjustable configuration parameters for other large models. From this data, the application can extract the configuration parameters that can be tuned, along with detailed information for each parameter. The process of creating the tuning instance can be automated (done by the application itself) or manually configured by the developers.
[0206] Step S02: Filter the target parameter configurations that have the greatest impact on application performance from the tunable configuration parameter information through tuning instances;
[0207] During the instance creation process, the selector module in the automatic configuration parameter tuning module automatically selects key performance parameters (configuration parameters that have a significant impact on the overall performance of a large language model), aiming to find the configuration parameters that have the greatest impact on application performance. In this way, the automatic configuration parameter tuning module can focus on optimizing the most important parameters, thereby improving tuning efficiency and effectiveness.
[0208] Step S03: Based on the current operating status of the system, automatically recommend the corresponding configuration values of the target parameter configuration to the application through the optimization instance;
[0209] Applications interact with the tuning instance using API calls to obtain recommended configuration values. These values are optimal settings recommended based on the system's current state and performance characteristics. The current state refers to the system's current running state, while performance characteristics refer to the features and attributes exhibited by the system when performing tasks.
[0210] Step S04: Obtain the reward value based on the application's configuration numerical feedback, and dynamically adjust and update the configuration of the tuning instance based on the reward value;
[0211] Over time, the application can monitor and evaluate the effectiveness and performance of these recommended values. Once the application has collected enough data and experience, it submits reward values to the configuration parameter auto-tuning module by calling the reward setting function. These reward values reflect the actual performance and performance optimization of the configuration values. The configuration parameter auto-tuning module then uses these reward values to dynamically adjust and update the configuration of the tuning instance.
[0212] Step S05: Repeat steps S02 to S04 to optimize the configuration parameters until the optimal parameter configuration is obtained.
[0213] The above process is a continuous optimization cycle. This allows the tuning instance to gradually improve its configuration to better meet the application's performance requirements and goals.
[0214] When the system has sufficient servers or resources, the automatic configuration parameter tuning module provided in this step can support parallel exploration. Alternatively, as an optional implementation, the system can manage and control multiple servers, each of which can support multiple applications.
[0215] Applications may require tuning for different hardware types, workloads, and functionalities. To manage and optimize these tuning instances, the configuration parameter auto-tuning module provides an automatic scope definer. This automatic scope definer automatically creates, manages, and determines the scope of each tuning instance based on detailed context information provided by the application.
[0216] For example, depending on the hardware type, some applications run better on multi-core CPUs, while others may rely more on GPU acceleration. For different workloads, the optimization strategies for an application handling read-intensive workloads may differ from those for handling write-intensive workloads. For different functions, the data processing module in an application may need to optimize computational efficiency, while the user interface module may focus more on improving response time.
[0217] Specifically, the automatic range definer can dynamically configure and adjust the parameters and settings of tuning instances based on hardware specifications, specific workload characteristics, and the specific functionalities required by the application. This allows each tuning instance to maximize its performance and be adjusted as needed to meet the specific requirements and expected goals of the application.
[0218] With the automatic range definer, applications can flexibly adapt to and utilize the provided tuning capabilities under different environments and requirements, thereby improving the overall efficiency and performance of the system.
[0219] In the specific implementation, for the first target applications with different tuning focuses, an automatic scope definer can be used to automatically create, manage, and determine the applicable scope of each tuning instance corresponding to each tuning focus based on the tuneable configuration parameters of each first target application.
[0220] In heterogeneous server scenarios (i.e., servers with different configurations), each server may have different hardware characteristics, workload requirements, or functional requirements. To effectively manage this diversity, the automatic range definer of the configuration parameter auto-tuning module categorizes and groups servers based on their specific contextual attributes and tunes them individually. This categorization and grouping process ensures that each server receives an optimized configuration to maximize its performance potential.
[0221] The specific context attributes of a server refer to configuration information, status information, or other relevant data related to a specific scenario or application during server operation. These attributes can be used to describe information such as the server's configuration, status, and performance characteristics.
[0222] In the specific implementation, for different configurations of the first server, the automatic scope delimiter can first classify and group all the second target applications according to the tunable configuration parameter information of the second target application in each first server to obtain several second target application groups; for each second target application group, a heterogeneous tuning instance is created, and the parameter configuration of the second target application group is tuned through the heterogeneous tuning instance, referring to steps S02 to S05.
[0223] For servers with identical configurations, multiple tuning processes can be performed simultaneously using a single tuning instance. This parallel exploration allows the system to optimize its performance and configuration more efficiently, thereby quickly finding the optimal parameter settings.
[0224] Therefore, in the specific implementation, for second servers with the same configuration, steps S01 to S05 can be referred to to simultaneously perform parameter configuration optimization on each second server through a single optimization instance, so that each second server obtains the optimal parameter configuration.
[0225] In other words, the automatic scope definer allows for the simultaneous management and configuration of multiple tuning instances. This means the automatic configuration parameter tuning module can automatically optimize multiple tuning instances at the same time. Furthermore, under the same server configuration, a single tuning instance can undergo multiple parallel tuning processes.
[0226] The automatic configuration parameter tuning module continuously receives feedback from the application to adjust the system configuration. This feedback includes evaluations of the system's responsiveness under high load based on various performance metrics. Specifically, the module collects and analyzes feedback data over several hours to understand the system's performance and resource utilization. Based on this data, the allocation of memory and computing resources can be dynamically adjusted to optimize the overall system efficiency and stability. This automated adjustment enables the system to maintain high efficiency in the face of constantly changing request characteristics and load patterns.
[0227] In this step, the automatic tuning module based on configuration parameters sequentially determines the configurations that need tuning, automatically manages the scope of configuration tuning, and simultaneously and rapidly adjusts numerical and classification configurations using reinforcement learning algorithms, thereby reducing the overhead of configuration tuning. Simultaneously, by combining the output results of the static optimization analysis in the preceding steps, the state and reward in the reinforcement learning algorithm can be directly calculated, thus avoiding time-consuming and labor-intensive experiments and achieving efficient configuration tuning for large language model training and inference.
[0228] Step 104: Perform dynamic optimization processing on the computing resource data based on the unloading strategy to obtain the optimal resource configuration;
[0229] This step primarily provides a dynamic optimization module based on offloading strategies. By applying these strategies, it aggregates computing and storage resources (computing resource data) from the CPU, GPU, and hard drive, and flexibly configures them within hardware resource constraints to improve throughput for large model inference on a single GPU. Simultaneously, users can dynamically adjust weights, activation values, and the proportion of KV cache on the CPU, GPU, and hard drive to further optimize the overall performance of large model inference. A flowchart illustrating the processing flow in the dynamic optimization module based on offloading strategies is shown below. Figure 4 As shown.
[0230] First, the large-scale model reasoning problem that introduces an unloading strategy is abstracted into a graph traversal problem. Under certain constraints, an effective path is found to traverse and compute each abstracted square in the graph.
[0231] Large model inference problems refer to the problems or challenges encountered when using large pre-trained models (such as deep learning models) for inference. These models often have high computational complexity and resource requirements due to their large scale.
[0232] Assuming the GPU, CPU, and hard drive constitute a three-tier storage structure for a machine, such as... Figure 5 As shown. From Figure 5 As can be seen, the storage space and data read / write speeds of GPUs, CPUs, and hard drives differ. GPUs have relatively small video memory, but their data read / write speeds are the fastest. Similarly, hard drives have the largest storage space, but their data read / write speeds are the slowest.
[0233] It is worth noting that in the assumed three-tier storage structure, only the GPU and CPU possess computing power, therefore, it is impossible to load the complete large model using only the GPU. The large model needs to be offloaded to auxiliary storage devices. Simultaneously, the computational task is completed step-by-step by partially loading the model. Therefore, to address the issue of the large resource requirements during large model inference, this embodiment of the invention introduces an offloading strategy. The large model inference problem with the introduced offloading strategy can be abstracted as a graph traversal problem. The computational graph representation of a large language model during model inference is as follows: Figure 6 As shown.
[0234] Assume that three tokens are returned for each prompt word in the batch. Figure 6 In this diagram, each square represents the batch size of the GPU at one layer. Since the primary purpose of the dynamic optimization module based on the offloading strategy is to improve throughput during large model inference, the specific number of batches to be processed is not a primary concern for the abstracted graph traversal problem. It should be noted that squares of the same color have the same model weight values.
[0235] As described above, this embodiment of the invention abstracts the large model reasoning problem that introduces an unloading strategy into a graph traversal problem. In the graph traversal problem, the definition of an effective path is crucial. The effective path defined here must traverse every abstracted square and is subject to the following four constraints:
[0236] (a) in Figure 6 In this process, when a certain square is about to be traversed, it is necessary to ensure that all squares in the same row as the current square and located to its left have already been traversed. That is, all these squares have been computed by the GPU.
[0237] (b) in Figure 6 In the graph traversal problem shown, a square represents the batch computation of the GPU in one layer of the model. During the traversal of these squares (i.e., during the GPU's computation of each abstracted square), in order to compute the squares in the graph on the same device, all the inputs for that square need to be loaded onto the same device. These inputs include activation values, weight values, and caches.
[0238] (c) in Figure 6 In this process, once a square block completes its calculation, it generates an activation value and a KVCache output. These outputs should be stored promptly. According to the principles of large model inference, the output of each token in the large language model's response depends on all tokens generated before it. For the activation value, it needs to be stored in the square block to the right of the current square block to complete the calculation. For the KV Cache, it needs to be stored in the rightmost square block in the same row as the current square block to complete the calculation.
[0239] (d) When traversing all the squares in the graph (i.e., during GPU computation), many tensors will be generated. These tensors need to be stored in the device for later use. However, the total size of the tensors stored in the device cannot exceed the device's memory capacity.
[0240] Based on the four constraints mentioned above, further verification can be performed. Figure 6 Building upon the abstract graph traversal problem, the definition and selection of effective paths are crucial. The selection of effective paths not only needs to satisfy the basic definition (i.e., traversing every square in the graph) but also the four constraints mentioned above. More importantly, the selection of effective paths should minimize the total execution time. This total execution time includes not only computational time costs but also input / output time costs.
[0241] Secondly, a multi-parameter search space is constructed. For computation scheduling, block granularity is added to the computation graph, performing zigzag block scheduling within each block. Therefore, this embodiment of the invention introduces two parameters: GPU batch size and block size. Simultaneously, to address tensor storage issues, nine parameter variables are introduced: wg, wc, wd, hg, hc, hd, cg, cc, and cd. These nine parameter variables, together with the GPU batch size and block size, constitute the multi-parameter search space. Specifically, wg, wc, and wd represent the percentage of model weights stored on the GPU, CPU, and hard disk, respectively; hg, hc, and hd represent the percentage of activation values stored on the GPU, CPU, and hard disk, respectively; and cg, cc, and cd represent the percentage of KV cache stored on the GPU, CPU, and hard disk, respectively.
[0242] In practical calculations, in addition to considering the impact of percentages, the impact of granularity also needs to be taken into account. Model granularity is reflected in the percentage of layers allocated to the GPU. Layer granularity is reflected in the percentage of tensors allocated to the GPU within a layer. Tensor granularity is reflected in the percentage of elements allocated to the GPU within a tensor. Through experiments with different granularities, and based on the experimental results while comprehensively considering factors such as overhead, different granularities are selected. In this embodiment of the invention, layer granularity is used for weights, while tensor granularity is used for activation values and KV cache.
[0243] An effective offloading strategy should consider efficient computation scheduling and reasonable storage of tensors. This process introduces several influencing parameters. An offloading strategy should include the aforementioned 11 variables. Based on a fixed batch size and block size of the GPU, a linear programming problem is solved to determine the optimal combination of the remaining influencing parameters (i.e., performing a graph traversal search calculation on each square block using efficient paths for each set of influencing parameters; after traversal, the final optimal combination of parameters can be determined). Finally, the optimizer is called to obtain an offloading strategy. Furthermore, pipelined parallelism can be employed to further improve throughput and reduce the burden on a single GPU.
[0244] Meanwhile, the dynamic optimization module also includes a group-wise quantization method. Specifically, the weights and elements in the KV cache are compressed into 4-bit integers to reduce communication costs. Before subsequent calculations, each element is decompressed and restored to an FP16 half-precision floating-point number. Although this process results in some loss of data precision, within acceptable error limits, a suitable model approximation method is more conducive to improving the throughput of large model inference.
[0245] Based on the above introduction, in the specific implementation, dynamic optimization of computing resource data based on an offloading strategy is performed to obtain the optimal resource configuration. This can be achieved by: first, abstracting the large model inference problem encountered by the large language model during model inference into a graph traversal problem; then, performing a multi-parameter search to solve the graph traversal problem to obtain the offloading strategy. The offloading strategy provides the optimal resource configuration when offloading computing resource data to an auxiliary storage device.
[0246] Furthermore, a multi-parameter search is performed on the graph traversal problem to obtain an unloading strategy. Specifically, this involves: determining the original parameter variables and introducing multiple influencing variables; constructing a multi-parameter search space based on the original parameter variables and the multiple influencing variables, and setting different search granularities for different search parameters; determining the effective path for graph traversal under preset constraints, whereby the effective path is used to traverse each square block abstracted from the computation graph, and each square block corresponds to a set of search parameters; and, based on the fixed binary pairs of the original parameter variables, performing a graph traversal-style parameter search calculation on each set of search parameters through the effective path to solve the linear programming problem, determine the optimal parameter combination of multiple influencing variables, and obtain the unloading strategy.
[0247] In this step, by setting up a dynamic optimization module based on the unloading strategy, the problems of inefficient input / output scheduling and tensor placement can be solved, making full use of CPU, GPU and hard disk resources, and greatly improving the throughput of large language model inference.
[0248] Step 105: Based on the optimal parameter configuration and the optimal resource configuration, dynamically optimize the parameter configuration and resource allocation of the large language model during model inference.
[0249] Specifically, based on the optimal parameter configuration and optimal resource configuration, the parameter configuration and resource allocation of the large language model during model inference can be dynamically optimized. This can be achieved by: dynamically adjusting the configurable parameters of the large language model during model inference based on the optimal parameter configuration to maximize system performance indicators and adapt to changes in hardware and workload; and offloading the computational resource data of the large language model during model inference to auxiliary storage devices according to the offloading strategy corresponding to the optimal resource configuration to improve the throughput during model inference.
[0250] It should be noted that, in order to enable those skilled in the art to better distinguish data of the same type but with different actual meanings, the embodiments of the present invention use "first" and "second" to distinguish and describe some technical features. "First" and "second" are only used to distinguish data and have no other special meaning. It is understood that the present invention does not impose any limitations on them.
[0251] In this embodiment of the invention, static optimization analysis of configuration parameters is first performed based on white-box modeling. A series of linear equations combined with model and graphics card configuration information provide quantitative analysis results, allowing users to modify certain parameters to optimize the inference performance of large language models. Simultaneously, reinforcement learning algorithms are used to achieve automatic parameter tuning for further optimization of some configuration parameters. Secondly, dynamic optimization is achieved based on an offloading strategy. By applying the offloading strategy, computing and storage resources from the CPU, GPU, and hard drive are aggregated and flexibly configured under hardware resource constraints. This invention not only improves the throughput of large model inference and enables automatic tuning of configuration parameters, but also helps users adjust certain configuration parameters during subsequent model inference to improve the overall performance of the large model inference process.
[0252] The technical solutions provided by the embodiments of the present invention can form a complete system-level large language model inference optimization framework. For ease of understanding, a specific example is used below to verify and illustrate the technical effects brought about by the technical solutions of the present invention.
[0253] The models used in the experiments were all mainstream open-source language models commonly used by various enterprises and universities. Their configuration parameters were all actual parameters of the large language models.
[0254] The following will use various metrics from memory analysis of a large language model as examples. Specifically, the large language model analyzed here is llama-13b, and the graphics card used is v100-sxm-32gb. The theoretical analysis results of the large model obtained by the static optimization module based on white-box modeling technology, and the optimization results obtained after these analysis results are automatically tuned by the configuration parameter tuning module based on reinforcement learning algorithm, are shown in Table 5 below:
[0255]
[0256] Table 5: Comparison of Optimized Metrics for Memory Analysis of Large Language Models
[0257] As can be seen, compared with the original indicators, all indicators have been significantly improved after static optimization analysis and automatic adjustment of configuration parameters, demonstrating good optimization effect.
[0258] The method provided in this embodiment of the invention is equally effective for large language models with different configurations than those described above. Taking the large language model OPT-30B and graphics card a-100 as an example, the experimental results of the dynamic optimization module based on the unloading strategy are shown in Tables 6 and 7 below:
[0259]
[0260] Table 6: Explanation of Experimental Parameters for the Dynamic Optimization Module Experiment
[0261]
[0262] Table 7: Experimental Results of the Dynamic Optimization Module
[0263] Table 6 shows the meaning and numerical values of the experimental parameters. Through this experiment, we can conclude that the weight percentage on the GPU is 20%, the weight percentage on the CPU is 80%, the attention cache percentage on the GPU is 100%, and the activation percentage on the CPU is 100%. Table 7 shows the inference performance of the large language model in real-world operation under these experimental parameters.
[0264] Reference Figure 7 The diagram illustrates a structural block diagram of a large language model inference optimization device provided in an embodiment of the present invention, which may specifically include:
[0265] The data acquisition module 701 is used to acquire configuration parameter data and computing resource data of the large language model during model inference in real time.
[0266] The static optimization analysis module 702 is used to perform static optimization analysis on the configuration parameter data and output the configuration analysis results.
[0267] The automatic parameter tuning module 703 is used to combine the configuration parameter data with the configuration analysis results to perform automatic tuning of configuration parameters based on reinforcement learning, so as to obtain the optimal parameter configuration.
[0268] The dynamic optimization processing module 704 is used to perform dynamic optimization processing on the computing resource data based on the unloading strategy to obtain the optimal resource configuration;
[0269] The model inference optimization module 705 is used to dynamically optimize the parameter configuration and resource allocation of the large language model during model inference based on the optimal parameter configuration and the optimal resource configuration.
[0270] In one optional embodiment, the automatic configuration parameter tuning module 703 includes:
[0271] The information extraction module is used to perform step S01: extract the tunable configuration parameter information of the application from the configuration parameter data and the configuration analysis results, and create an optimization instance at the same time;
[0272] The information filtering module is used to perform step S02: filtering the target parameter configuration that has the greatest impact on the performance of the application from the tunable configuration parameter information through the optimization instance;
[0273] The parameter configuration automatic recommendation module is used to perform step S03: based on the current operating status of the system, automatically recommend the configuration value corresponding to the target parameter configuration to the application through the optimization instance;
[0274] The optimization instance adjustment module is used to perform step S04: obtain the reward value fed back by the application based on the configuration value, and dynamically adjust and update the configuration of the optimization instance based on the reward value;
[0275] The configuration parameter tuning loop module is used to execute step S05: repeatedly execute the configuration parameter tuning process from steps S02 to S04 until the optimal parameter configuration is obtained.
[0276] In one alternative embodiment, the system manages and controls multiple servers, each server corresponding to multiple applications, and the apparatus further includes:
[0277] The optimization instance applicability determination module is used to automatically create, manage and determine the applicability of optimization instances corresponding to each optimization focus for first target applications with different optimization focuses, through an automatic scope definer, based on the tunable configuration parameter information of each first target application.
[0278] The application classification module is used to classify and group all second target applications based on the tunable configuration parameter information of the second target application in each first server with different configurations, using an automatic scope delimiter, to obtain several second target application groups.
[0279] The first parameter configuration tuning module is used to create a heterogeneous tuning instance for each of the second target application groups. Referring to steps S02 to S05, the parameter configuration of the second target application group is tuned through the heterogeneous tuning instance.
[0280] The second parameter configuration tuning module is used to simultaneously tune the parameter configuration of each second server with the same configuration, referring to steps S01 to S05, by creating a single tuning instance, so that each second server obtains the optimal parameter configuration.
[0281] In one optional embodiment, the dynamic optimization processing module 704 includes:
[0282] The model reasoning problem abstraction module is used to abstract the large model reasoning problem encountered by the large language model in the model reasoning process into a graph traversal problem.
[0283] A multi-parameter search and solution module is used to perform a multi-parameter search and solution on the graph traversal problem to obtain an unloading strategy. The unloading strategy provides the optimal resource configuration when unloading the computing resource data to an auxiliary storage device.
[0284] In one optional embodiment, the multi-parameter search and solution module includes:
[0285] A multi-parameter search space construction module is used to determine the original parameter variables and introduce multiple influencing variables. Based on the original parameter variables and the multiple influencing variables, a multi-parameter search space is constructed, and different search granularities are set for different search parameters.
[0286] The effective path determination module is used to determine the effective path when performing graph traversal under preset constraints. The effective path is used to traverse each square block abstracted in the computation graph, and each square block corresponds to a set of search parameters.
[0287] The parameter search and calculation module is used to perform graph traversal parameter search and calculation on each of the search parameter groups through the effective path, based on the fixed binary tuples of the original parameter variables, in order to solve the linear programming problem, determine the optimal parameter combination of the multiple influencing variables, and obtain the unloading strategy.
[0288] In one optional embodiment, the model inference optimization module 705 includes:
[0289] A configurable parameter dynamic adjustment module is used to dynamically adjust the configurable parameters of the large language model during model inference according to the optimal parameter configuration, so as to maximize system performance indicators and adapt to changes in hardware and workload.
[0290] The resource unloading module is used to unload the computational resource data of the large language model during model inference to an auxiliary storage device according to the unloading strategy corresponding to the optimal resource configuration, so as to improve the throughput during model inference.
[0291] In one optional embodiment, the configuration parameter data includes the system's GPU configuration information, the model configuration information and model architecture information of the large language model, and the static optimization analysis module 702 is specifically used for:
[0292] The GPU configuration information, the model configuration information, and the model architecture information are input into a multi-layered analysis model pre-built based on white-box modeling for static optimization analysis, and the configuration analysis results are output.
[0293] The static optimization analysis includes inference configuration analysis, parameter quantity analysis, computational quantity analysis, memory analysis, and latency analysis.
[0294] As the device embodiment is basically similar to the method embodiment, it is described in a relatively simple way. For relevant details, please refer to the description of the method embodiment above.
[0295] This invention also provides an electronic device, which includes a processor and a memory:
[0296] The memory is used to store program code and transfer the program code to the processor;
[0297] The processor is used to execute the large language model inference optimization method of any embodiment of the present invention according to the instructions in the program code.
[0298] This invention also provides a computer-readable storage medium for storing program code for executing the large language model inference optimization method of any embodiment of this invention.
[0299] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0300] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.
[0301] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0302] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0303] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0304] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for optimizing reasoning in a large language model, characterized in that, include: Real-time acquisition of configuration parameter data and computing resource data of large language models during model inference; Perform static optimization analysis on the configuration parameter data and output the configuration analysis results; By combining the configuration parameter data with the configuration analysis results, automatic tuning of the configuration parameters based on reinforcement learning is performed to obtain the optimal parameter configuration; The computing resource data is dynamically optimized based on an unloading strategy to obtain the optimal resource configuration; Based on the optimal parameter configuration and the optimal resource configuration, the parameter configuration and resource allocation of the large language model during model inference are dynamically optimized. The step of automatically optimizing configuration parameters based on reinforcement learning by combining the configuration parameter data with the configuration analysis results to obtain the optimal parameter configuration includes: Step S01: Extract the tunable configuration parameter information of the application from the configuration parameter data and the configuration analysis results, and create a tuning instance at the same time; Step S02: Using the tuning instance, filter the target parameter configuration that has the greatest impact on the performance of the application from the tuneable configuration parameter information; Step S03: Based on the current operating status of the system, automatically recommend the configuration values corresponding to the target parameter configuration to the application through the optimization instance; Step S04: Obtain the reward value fed back by the application based on the configuration value, and dynamically adjust and update the configuration of the optimization instance based on the reward value; Step S05: Repeat the configuration parameter tuning process from steps S02 to S04 until the optimal parameter configuration is obtained; The step of performing dynamic optimization processing on the computing resource data based on an unloading strategy to obtain the optimal resource configuration includes: The large model reasoning problem encountered by the large language model during the model reasoning process is abstracted into a graph traversal problem; A multi-parameter search is performed on the graph traversal problem to obtain an unloading strategy, which provides the optimal resource configuration when the computing resource data is unloaded to an auxiliary storage device.
2. The large language model inference optimization method according to claim 1, characterized in that, The system manages and controls multiple servers, each server corresponding to multiple applications, and the method further includes: For first target applications with different tuning focuses, an automatic scope definer is used to automatically create, manage, and determine the applicable scope of each tuning instance corresponding to each tuning focus based on the tuneable configuration parameter information of each first target application. For different configurations of the first server, an automatic scope delimiter is used to classify and group all the second target applications according to the tunable configuration parameter information of the second target application in each first server, so as to obtain several groups of second target applications. For each of the second target application groups, a heterogeneous optimization instance is created. Referring to steps S02 to S05, the parameter configuration of the second target application group is optimized through the heterogeneous optimization instance. For the second server with the same configuration, refer to steps S01 to S05, and simultaneously perform parameter configuration tuning on each of the second servers by creating a single tuning instance, so that each of the second servers obtains the optimal parameter configuration.
3. The large language model inference optimization method according to claim 1, characterized in that, The process of solving the graph traversal problem using a multi-parameter search to obtain the unloading strategy includes: The original parameter variables are determined, and multiple influencing variables are introduced. Based on the original parameter variables and the multiple influencing variables, a multi-parameter search space is constructed, and different search granularities are set for different search parameters. Under preset constraints, an effective path is determined when traversing the graph. The effective path is used to traverse each square block abstracted in the computation graph, and each square block corresponds to a set of search parameters. Based on the fixed binary tuples of the original parameter variables, a graph traversal parameter search calculation is performed on each of the search parameter groups through the effective path to solve the linear programming problem, determine the optimal parameter combination of the multiple influencing variables, and obtain the unloading strategy.
4. The large language model inference optimization method according to claim 1 or 3, characterized in that, The step of dynamically optimizing the parameter configuration and resource allocation of the large language model during model inference based on the optimal parameter configuration and the optimal resource configuration includes: Based on the optimal parameter configuration, the configurable parameters of the large language model during model inference are dynamically adjusted to maximize system performance indicators and adapt to changes in hardware and workload. According to the unloading strategy corresponding to the optimal resource configuration, the computational resource data of the large language model during model inference is unloaded to the auxiliary storage device to improve the throughput during model inference.
5. The large language model inference optimization method according to claim 1, characterized in that, The configuration parameter data includes the system's GPU configuration information, the model configuration information and model architecture information of the large language model, and the static optimization analysis of the configuration parameter data, outputting the configuration analysis results, includes: The GPU configuration information, the model configuration information, and the model architecture information are input into a multi-layered analysis model pre-built based on white-box modeling for static optimization analysis, and the configuration analysis results are output. The static optimization analysis includes inference configuration analysis, parameter quantity analysis, computational quantity analysis, memory analysis, and latency analysis.
6. A large language model reasoning optimization device, characterized in that, include: The data acquisition module is used to acquire configuration parameter data and computing resource data of the large language model in real time during model inference; The static optimization analysis module is used to perform static optimization analysis on the configuration parameter data and output the configuration analysis results. The automatic parameter tuning module is used to combine the configuration parameter data with the configuration analysis results to automatically tune the configuration parameters based on reinforcement learning, so as to obtain the optimal parameter configuration. The dynamic optimization processing module is used to perform dynamic optimization processing on the computing resource data based on the unloading strategy to obtain the optimal resource configuration; The model inference optimization module is used to dynamically optimize the parameter configuration and resource allocation of the large language model during model inference based on the optimal parameter configuration and the optimal resource configuration. The automatic configuration parameter tuning module includes: The information extraction module is used to perform step S01: extract the tunable configuration parameter information of the application from the configuration parameter data and the configuration analysis results, and create an optimization instance at the same time; The information filtering module is used to perform step S02: filtering the target parameter configuration that has the greatest impact on the performance of the application from the tunable configuration parameter information through the optimization instance; The parameter configuration automatic recommendation module is used to perform step S03: based on the current operating status of the system, automatically recommend the configuration value corresponding to the target parameter configuration to the application through the optimization instance; The optimization instance adjustment module is used to perform step S04: obtain the reward value fed back by the application based on the configuration value, and dynamically adjust and update the configuration of the optimization instance based on the reward value; The configuration parameter tuning loop module is used to execute step S05: repeatedly execute the configuration parameter tuning process from steps S02 to S04 until the optimal parameter configuration is obtained; The dynamic optimization processing module includes: The model reasoning problem abstraction module is used to abstract the large model reasoning problem encountered by the large language model in the model reasoning process into a graph traversal problem. A multi-parameter search and solution module is used to perform a multi-parameter search and solution on the graph traversal problem to obtain an unloading strategy. The unloading strategy provides the optimal resource configuration when unloading the computing resource data to an auxiliary storage device.
7. An electronic device, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the large language model inference optimization method according to any one of claims 1-5 according to the instructions in the program code.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the large language model inference optimization method according to any one of claims 1-5.