Large language model reasoning optimization method and device, electronic equipment and storage medium

Abstract

The invention discloses a large language model reasoning optimization method and device, electronic equipment and a storage medium, which are used for solving the problem of how to reduce huge resources needed during large model reasoning due to the fact that model reasoning has a large demand on calculation and storage resources due to huge parameter quantity in the prior art. The method comprises the following steps: acquiring configuration parameter data and computing resource data of a large language model during model reasoning in real time; performing static optimization analysis on the configuration parameter data, and outputting a configuration analysis result; performing automatic configuration parameter tuning based on reinforcement learning in combination with the configuration parameter data and the configuration analysis result to obtain optimal parameter configuration; performing dynamic optimization processing on the computing resource data based on an unloading strategy to obtain optimal resource configuration; and according to the optimal parameter configuration and the optimal resource configuration, dynamically optimizing parameter configuration and resource allocation of the large language model during model reasoning.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a large language model reasoning optimization method, device, electronic device and storage medium. Background Art

[0002] With the sharp increase in the parameters of large language models, the required video memory has also gradually increased, which has brought great challenges to the use of computing and storage resources. The parameters of the emerging large language models have reached billions or even trillions. When performing model inference, a large amount of video memory is needed to store the model weights and intermediate calculation results. However, it is undeniable that the increase in model parameters can bring beneficial effects to the improvement of model performance and the solution of complex problems. It can be seen that the huge computing and storage resources consumed during large model inference and training is an issue that cannot be ignored.

[0003] For large language model reasoning, the current focus is on model compression, collaborative reasoning, and offloading strategies to reduce the consumption of computing and storage resources. Model compression aims to reduce the computing and storage resources required for the model by reducing model parameters or simplifying the model structure while ensuring model performance within a certain range. Collaborative reasoning mainly completes the task of model reasoning through the cooperation of multiple devices or multiple models to reduce the burden of a single device and a single model, thereby reducing overall resource consumption. The core idea of ​​the offloading strategy method is to offload the computing tasks concentrated on a certain device to other devices, thereby reducing the device load and achieving efficient allocation of resources.

[0004] Among the above methods, the first two methods usually assume that the model fits into GPU memory, making it difficult to run models with large parameters using only a single consumer GPU. In the latest systems of the third method, it is impossible to achieve acceptable throughput on a single GPU due to inefficiencies in I / O scheduling and tensor placement. Summary of the invention

[0005] The present invention provides a large language model reasoning optimization method, device, electronic device and storage medium, which are used to solve or partially solve the problem in the existing related technology that the huge amount of parameters leads to a large demand for computing and storage resources for model reasoning, and how to reduce the huge resources required for large model reasoning.

[0006] The present invention provides a large language model reasoning optimization method, the method comprising:

[0007] Obtain configuration parameter data and computing resource data of large language models in real time during model inference;

[0008] Performing static optimization analysis on the configuration parameter data and outputting configuration analysis results;

[0009] Combining the configuration parameter data with the configuration analysis result, automatic tuning of the configuration parameters based on reinforcement learning is performed to obtain an optimal parameter configuration;

[0010] Performing dynamic optimization processing on the computing resource data based on the offloading strategy to obtain optimal resource configuration;

[0011] According to the optimal parameter configuration and the optimal resource configuration, the parameter configuration and resource allocation of the large language model during model inference are dynamically optimized.

[0012] The present invention also provides a large language model reasoning optimization device, comprising:

[0013] The data acquisition module is used to obtain the configuration parameter data and computing resource data of the large language model in real time during model inference;

[0014] A static optimization analysis module, used to perform static optimization analysis on the configuration parameter data and output configuration analysis results;

[0015] A configuration parameter automatic tuning module, used to combine the configuration parameter data with the configuration analysis result to perform automatic tuning of the configuration parameters based on reinforcement learning to obtain the optimal parameter configuration;

[0016] A dynamic optimization processing module, used to perform dynamic optimization processing on the computing resource data based on the offloading strategy to obtain the optimal resource configuration;

[0017] The model reasoning optimization module is used to dynamically optimize the parameter configuration and resource allocation of the large language model during model reasoning according to the optimal parameter configuration and the optimal resource configuration.

[0018] The present invention also provides an electronic device, the device comprising a processor and a memory:

[0019] The memory is used to store program code and transmit the program code to the processor;

[0020] The processor is used to execute the large language model reasoning optimization method as described in any one of the above items according to the instructions in the program code.

[0021] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the large language model reasoning optimization method as described in any one of the above items.

[0022] It can be seen from the above technical solutions that the present invention has the following advantages:

[0023] A large language model reasoning optimization method is provided. The configuration parameter data and computing resource data of the large language model during model reasoning are obtained in real time; static optimization analysis is performed on the configuration parameter data, and the configuration analysis results are output to facilitate the user to modify some of the parameters to optimize the reasoning performance of the large language model; the configuration parameter data and the configuration analysis results are combined to automatically tune the configuration parameters based on reinforcement learning to obtain the optimal parameter configuration to further optimize some of the configuration parameters; the computing resource data is dynamically optimized based on the unloading strategy to obtain the optimal resource configuration to achieve flexible configuration under the constraints of hardware resources; according to the optimal parameter configuration and the optimal resource configuration, the parameter configuration and resource allocation of the large language model during model reasoning are dynamically optimized. Through the technical solution of the present invention, not only the throughput during large model reasoning can be improved, but also the automatic tuning of configuration parameters can be achieved. It can also help users to adjust certain configuration parameters in subsequent model reasoning to improve the overall performance of the large model reasoning process. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A flowchart of the steps of a large language model inference optimization method;

[0026] Figure 2 It is a schematic diagram of the processing flow in a static optimization module based on white box modeling;

[0027] Figure 3 A schematic diagram of the processing flow in an automatic parameter tuning module for a large model based on a reinforcement learning algorithm;

[0028] Figure 4 A schematic diagram of a processing flow in a dynamic optimization module based on an offloading strategy;

[0029] Figure 5 A schematic diagram of a preset three-level storage structure;

[0030] Figure 6 It is a computational graph representation of a large language model during model reasoning;

[0031] Figure 7 The structural block diagram of a large language model inference optimization device. DETAILED DESCRIPTION

[0032] The embodiments of the present invention provide a large language model reasoning optimization method, device, electronic device and storage medium, which are used to solve or partially solve the problem in the existing related technology that the huge amount of parameters leads to a large demand for computing and storage resources for model reasoning, and how to reduce the huge resources required for large model reasoning.

[0033] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] As an example, for large language model reasoning, the current focus is on reducing the consumption of computing and storage resources around model compression, collaborative reasoning, and offloading strategies. Model compression and collaborative reasoning usually assume that the model fits into GPU memory, so it is difficult to run models with large parameters using only a single consumer GPU. Specifically, model compression aims to reduce memory usage and computational complexity by reducing the number or precision of model parameters, making it easier to deploy on resource-constrained devices. However, when faced with deep learning models with millions or even billions of parameters, even after compression, the size of the model may still exceed the memory capacity of a single consumer GPU. Collaborative reasoning involves distributing computing tasks among multiple devices to achieve a more efficient reasoning process. This approach usually assumes that all participating devices have enough memory and computing power to process part of the model. But in reality, consumer GPUs have limited memory and cannot independently handle the complete reasoning task of large models. Therefore, these two methods still face challenges in practical applications and need further optimization and innovation to effectively run large-scale models in resource-constrained environments.

[0035] In the latest offload-based systems, inefficiencies in input / output scheduling and tensor placement prevent acceptable throughput on a single GPU. Specifically, input / output scheduling involves how to effectively manage the transfer of data from storage devices to GPU memory. Inefficient scheduling strategies may lead to bottlenecks in data transfer, which prevents the GPU from fully utilizing its computing power, thus affecting the overall system performance. Tensor placement refers to the proper allocation and management of computing units of deep learning models in GPU memory. Inappropriate tensor placement strategies may lead to memory fragmentation or frequent memory swapping, which will significantly reduce the operating efficiency of the system. These inefficiencies work together to result in underutilization of GPU resources. As a result, the system throughput does not meet the expected standards when processing large amounts of data and complex computing tasks. Therefore, even though the latest offload-based systems can theoretically improve performance, in practice, the inefficiency of input / output scheduling and tensor placement remains a key bottleneck that requires further optimization and improvement to achieve acceptable throughput on a single GPU.

[0036] Furthermore, if only white-box modeling technology is used to build a simulator for a large language model, since the series of analysis results are obtained through pre-established linear equations, they cannot adapt to new data or environmental changes during operation. This means that the model cannot perform online learning, that is, it cannot dynamically adjust and optimize itself to adapt to new situations during system operation. It is also impossible to use effective optimization algorithms to reversely optimize configuration parameters. Among them, reverse optimization is a process of finding the optimal configuration parameters through an optimization algorithm based on an objective function and constraints. Since the linear equations of the white-box model are predetermined and cannot be adjusted at runtime, the model cannot adaptively optimize configuration parameters under new data or conditions.

[0037] Therefore, one of the core invention points of the embodiment of the present invention is: in view of the shortcomings in the prior art, in order to reduce the huge resources required for model reasoning, a large language model reasoning optimization method based on an offloading strategy is proposed. First, static optimization analysis of the configuration parameters is performed based on the white box modeling method, and quantitative analysis results are given by combining a series of linear equations with the configuration information of the model and the graphics card, so that users can modify some of the parameters to optimize the reasoning performance of the large language model. At the same time, the automatic parameter tuning is realized in combination with the reinforcement learning algorithm to further optimize some of the configuration parameters. Secondly, dynamic optimization is realized based on the offloading strategy. Through the application of the offloading strategy, the computing and storage resources from the CPU, GPU and hard disk are summarized, and flexible configuration is performed under the constraints of hardware resources. Through the technical solution of the present invention, not only the throughput during large model reasoning can be improved, but also the automatic tuning of configuration parameters can be realized. It can also help users to adjust certain configuration parameters during subsequent model reasoning to improve the overall performance of the large model reasoning process. The method proposed in the present invention can provide strong support for optimizing the performance of the large language model system, improving the reliability and efficiency of the system, and promoting the application of large language models in various fields.

[0038] The technical solution of the present invention mainly consists of three parts: a static optimization module based on white box modeling technology, a large model configuration parameter automatic tuning module based on reinforcement learning algorithm, and a dynamic optimization module based on unloading strategy. Through the processing of these three modules, a large language model reasoning acceleration framework can be formed as a whole, which can improve the throughput of large model reasoning to a certain extent, reduce the computing and storage resources spent by the system during large model reasoning, and provide strong support for the subsequent construction of large language model reasoning simulation and optimization system.

[0039] Reference Figure 1 , shows a flowchart of a large language model reasoning optimization method provided by an embodiment of the present invention, which may specifically include the following steps:

[0040] Step 101, obtaining configuration parameter data and computing resource data of a large language model during model inference in real time;

[0041] Configuration parameter data mainly includes the system's GPU configuration information, the model configuration information of the large language model, and the model architecture information. Computing resource data refers to the computing and storage resources from the CPU, GPU, and hard disk obtained through aggregation.

[0042] For the processing of configuration parameter data, in a specific implementation, the configuration parameter data such as model configuration parameters, performance parameters and GPU configuration information can be organized into a json file. Among them, the model configuration information can specifically include the model name, model architecture and model type. The model architecture includes the number of model layers, hidden layer dimensions, feedforward neural network intermediate layer dimensions, maximum sequence length, number of attention heads and vocabulary table length. GPU configuration information includes GPU name, GPU device memory size, GPU device memory bandwidth, bandwidth within GPU device node, bandwidth between GPU device nodes, GPU single precision computing performance peak size, GPU half precision computing performance peak size, GPU 8-bit integer computing performance peak size, GPU 4-bit integer computing performance peak size, and minimum message delay within GPU node. Model configuration information can be downloaded from the Hugging Face community (an open source community) The corresponding large language model can be organized by referring to the config file therein.

[0043] Step 102, performing static optimization analysis on the configuration parameter data and outputting configuration analysis results;

[0044] This step mainly performs static optimization analysis based on the static optimization module built by white box modeling technology. In the specific implementation, static optimization analysis is performed on the configuration parameter data, and the configuration analysis results are output. The GPU configuration information, model configuration information, and model architecture information are input into a multi-level analysis model (which can also be understood as a systematic linear equation) pre-built based on white box modeling for static optimization analysis, and a series of configuration analysis results are output. Among them, static optimization analysis includes reasoning configuration analysis, parameter quantity analysis, computational quantity analysis, memory analysis, and latency analysis.

[0045] Figure 2 A schematic diagram of a processing flow in a static optimization module based on white box modeling provided by an embodiment of the present invention is shown. In order to enable those skilled in the art to better understand the analysis process of step 102, the following will focus on the five parts of theoretical performance analysis results obtained through analysis, and introduce in detail the linear equations used therein and the specific analysis in the static optimization module.

[0046] (1) Analysis of the main reasoning configurations of large language models

[0047] For the main reasoning configuration analysis of the large language model, first create a reasoning configuration dictionary. The created reasoning configuration dictionary mainly consists of two parts: the reasoning configuration part and the GPU configuration part.

[0048] Specifically, the inference configuration section includes the model name, the batch size for each GPU, the length of the input sequence, the size of the thread pool, the size of the pipeline parallelism, the number of generated tokens, and whether KV Cache (Key-ValueCache, data storage technology / used to quickly access and retrieve data) is used.

[0049] It is worth noting that pooling technology refers to saving some resources in advance to avoid insufficient resources when they are used. At the same time, it improves the efficiency of resource reuse. Especially in scenarios with a large number of requests, it can significantly improve the overall performance of the system. At the same time, it also makes the resource consumption reach the optimal state. The thread pool uses pooling technology. The thread pool will start several threads, and these started threads are in a dormant state. When specific work is required, one of the threads will be started. When the task is completed, the thread will return to a dormant state. The corresponding meanings of the various parameters output in the inference configuration part are shown in Table 1 below:

[0050]

[0051] Table 1: Meaning of parameters in the inference configuration output

[0052] The GPU configuration section includes the GPU name, GPU memory size, GPU memory bandwidth, GPU node bandwidth, and GPU floating-point operations per second.

[0053] It is worth noting that GPU memory bandwidth describes the data transfer rate within the GPU. It is usually expressed in the amount of data transferred per second. HBM is a newer GPU memory chip that greatly improves performance and bandwidth. GPU node bandwidth describes the data transfer rate between the GPU and other system components (including the CPU and other GPUs). It is usually expressed in the amount of data transferred per second. GPU floating point operations per second are usually measured in TFLOPS. TFLOPS represents trillions of floating point operations per second.

[0054] Finally, the information in the inference configuration dictionary is printed out in an unsorted dictionary. That is, the order of the key-value pairs in the dictionary is not sorted, and the order in which the key-value pairs were originally inserted into the dictionary is retained and output. The meanings of the various parameters output in the GPU configuration section are shown in Table 2 below:

[0055]

[0056] Table 2: Meaning of parameters in the GPU configuration output

[0057] It is worth noting that in the Transformer model, the feedforward neural network exists not only in the encoder but also in the decoder. The feedforward neural network consists of two linear layers. The activation function of the first linear layer is ReLU, and the second layer does not use an activation function. The output dimension of the first layer is the dimension size of the middle layer of the feedforward neural network. The output dimension of the second layer is the dimension size of the hidden layer in the model. These two parameters are involved in the detailed information of each model. The model configuration information file contains several detailed information about the model. Among them, the information contained in each model is shown in Table 3 below:

[0058]

[0059] Table 3: Model information contained in the model configuration information file

[0060] The GPU configuration information file contains several detailed information about the graphics card. The information contained in each graphics card is shown in Table 4 below:

[0061]

[0062] Table 4: Graphics card information contained in the GPU configuration information file

[0063] (2) Analysis of the number of parameters of large language models

[0064] The following will introduce the relevant specific calculation formulas for large language model parameter analysis.

[0065] The number of parameters for each layer of the attention module mainly includes four linear layers, namely query, key, value projection and output matrix. The specific calculation formula for the number of parameters for each layer of the attention module is to add the number of parameters of the four linear layers. The result is also the square of the hidden layer dimension hidden_dim of the quadruple model. The calculation formula for the number of parameters params_attn for each layer of the attention module is as follows:

[0066]

[0067] The number of parameters in the linear layer of the multilayer perceptron includes the intermediate and output matrices. Specifically, the number of parameters in the linear layer of the multilayer perceptron params_mlp is eight times the square of the hidden layer dimension of the model. The calculation formula is as follows:

[0068]

[0069] For the parameter params_layernorm of each layer in the two normalization modules, if ln_ignore is True, params_layernorm is 0, otherwise its calculation formula is as follows:

[0070]

[0071] Among them, ln_ignore indicates whether to ignore the parameter quantity in the standardization module.

[0072] The parameters of each layer of the decoder module in the Transformer model mainly include the parameters of each layer of the attention module and the parameters in the linear layer of the multilayer perceptron. Since the parameters of each layer of the attention module and the parameters in the linear layer of the multilayer perceptron are mainly considered, the parameter ln_ignore is set. The default value of ln_ignore is True. If ln_ignore is True, only the parameters of each layer of the attention module plus the parameters in the linear layer of the multilayer perceptron are considered. Otherwise, the parameters of each layer in the normalization module need to be considered. In summary, when ln_ignore is False, the calculation formula for the parameter amount params_per_layer of each layer of the decoder module in the Transformer model is as follows:

[0073]

[0074] When ln_ignore is True, the calculation formula of params_per_layer is as follows:

[0075]

[0076] For the number of parameters of the embedding layer, the parameter shared_embedding indicates whether the input and output embedding layers share parameters, and the default value is True.

[0077] The calculation formula for the input embedding layer parameter num_params_input_embedding is as follows:

[0078]

[0079] When the input and output embedding layers share parameters, the output embedding layer parameter quantity is 0. Otherwise, the calculation formula for the output embedding layer parameter quantity num_params_output_embedding is as follows:

[0080]

[0081] The calculation of the total number of model parameters needs to consider all layers and embedding layers in the model. By default, the input and output embedding layers share parameters and do not consider the number of parameters of each layer in the normalization module. The specific calculation formula for the total number of model parameters params_model is as follows:

[0082]

[0083] Among them, params_embedding represents the number of parameters in the embedding layer.

[0084] (3) Computational Analysis of Large Language Models

[0085] The computational cost analysis of large language models is only related to the selected model, but not to the parallel strategy adopted. It also ignores the computational cost of the normalization module and other element-by-element operations.

[0086] The floating point calculation amount flops_attn during the forward propagation of the attention module in the Transformer layer mainly includes linear calculations such as query, key, value projection and output matrix multiplication, as well as self-attention internal operations. The specific calculation formula of flops_attn is as follows:

[0087]

[0088] Among them, flops_q represents the amount of computation required for the query operation; flops_k represents the amount of computation required for the key projection operation, flops_v represents the amount of computation required for the value projection operation, flops_output represents the amount of computation required for the output matrix multiplication operation, and flops_self_attention represents the amount of computation required for the internal operation of the self-attention.

[0089] The computational amount flops_mlp in the multilayer perceptron module mainly includes two floating-point operations of matrix multiplication. The specific calculation formula of flops_mlp is as follows:

[0090]

[0091] Among them, flops_fc1 and flops_fc2 both refer to the floating-point operations of matrix multiplication.

[0092] For the calculation amount of each layer during forward propagation, flops_fwd_per_layer, the main analysis idea is to add the floating point operation calculation amount of the attention module in the Transformer layer during forward propagation and the calculation amount in the multi-layer perceptron module. The specific calculation formula of flops_fwd_per_layer is as follows:

[0093]

[0094] Among them, flops_layernorm refers to the computational amount of the normalization module.

[0095] When analyzing the floating point calculation amount flops_model during the forward propagation of the Transformer model, the floating point calculation amount in the output tag logits layer should be considered. It is worth noting that the output tag logits layer is usually the last layer of the model used in the field of natural language processing, and its output is a vector of the same length as the vocabulary. Each value in the vector represents the probability distribution of the corresponding word in the current context. During the model training process, it can be compared with the true value to calculate the loss function. During the model inference process, the word with the highest probability can be selected as the current output of the model.

[0096] In addition, activation functions are usually not used in the output label logits layer, the purpose of which is to preserve the numerical size of the probability distribution. The specific calculation formula of flops_model is as follows:

[0097]

[0098] Among them, flops_logits_layer refers to the amount of floating-point operations in the output label logits layer.

[0099] (4) Memory Analysis of Large Language Models

[0100] The first part is the memory and maximum batch size analysis of the pre-population phase.

[0101] First, the specific calculation formula for the memory size occupied by the model weight, weight_memory_per_gpu, is as follows.

[0102]

[0103] Among them, bytes_per_param indicates the storage precision of the model weights, and the default value is FP16; embedding_dtype_bytes indicates the data type of the embedding layer, and the default value is FP16; params_embedding indicates the sum of the input embedding layer parameters and the output embedding layer parameters. According to the default value, the input and output embedding layers share parameters, and the output embedding layer parameters are 0, so the value of params_embedding is ; According to the above analysis, params_per_layer has the default value of ln_ignore being True and params_per_layer being .

[0104] Next, we analyze the memory size occupied by the model activation values ​​in the pre-filling phase. For the calculation of the memory size occupied by the activation values ​​of each layer, if the activation value recalculation flag is not FULL, the memory size occupied by the activation values ​​of each layer is 0. If the activation value recalculation flag is FULL, the specific value will be affected by the batch size.

[0105] If the batch size is specified as 1, the following formula can be used for calculation:

[0106]

[0107] Among them, prefill_activation_memory_batch_size_1 refers to the memory size occupied by the model activation value in the prefilling stage when the specified batch size is 1; num_layers_per_gpu refers to the number of layers placed on each GPU in the multi-layer neural network.

[0108] If a specific batch size is specified, the memory size prefill_activation_memory_per_gpu occupied by the activation value in this case can be calculated using the following formula:

[0109]

[0110] For the maximum batch size of the prefill phase, prefill_max_batch_size_per_gpu, the following formula can be used to calculate:

[0111]

[0112] Among them, memory_left refers to the remaining available video memory; gpu_memory_in_GB refers to the total video memory capacity on each GPU; weight_memory_per_gpu refers to the size of the video memory on each GPU used to store model weights.

[0113] The second part is the memory and maximum batch size analysis of the decoding stage.

[0114] For the decoding stage, the analysis of memory and maximum batch processing size needs to be discussed in two cases, with whether KV Cache is used as the defining criterion.

[0115] First, we analyze the memory size decode_activation_memory_per_gpu occupied by the activation value in the decoding stage. If KV Cache is used, decode_activation_memory_per_gpu can be calculated by the following formula:

[0116]

[0117] Since KV Cache is used, the seq_len value here is 1, which means the newly generated token.

[0118] If KV Cache is not used, the parameter seq_len is , that is, the context length is no longer the newly generated token, but At this time, the specific calculation formula of decode_activation_memory_per_gpu is as follows:

[0119]

[0120] The second is the analysis of the memory size kv_cache_memory_per_gpu occupied by the KV Cache in the decoding phase. When the KV Cache is used, the memory (in bytes) required to store the KV Cache of the Transformer layer during model inference needs to be calculated first. The calculated result needs to be multiplied by the number of neural network layers processed on each GPU device. The specific calculation formula for kv_cache_memory_per_gpu is as follows:

[0121]

[0122] kv_cache_dtype_bytes refers to the number of bytes occupied by the data type used to store the key-value cache.

[0123] If KV Cache is not used, the kv_cache_memory_per_gpu value is 0.

[0124] Next is the analysis of the total memory consumption decode_memory_total in the decoding phase. The specific calculation formula is as follows:

[0125]

[0126] When using KV Cache, the specific calculation formula for decode_memory_total is as follows:

[0127]

[0128] Without using KV Cache, the specific calculation formula for decode_memory_total is as follows:

[0129]

[0130] Next is the analysis of the maximum batch size decode_max_batch_size_per_gpu in the decoding stage. When using KV Cache, decode_max_batch_size_per_gpu can be calculated as follows:

[0131]

[0132] Among them, decode_activation_memory_batch_size_1 refers to the memory size occupied by model activation when processing a single batch of data; kv_cache_memory_batch_size_1 refers to the memory size occupied by the key-value cache of the model when processing a single batch of data.

[0133] When KV Cache is not used, decode_max_batch_size_per_gpu can be calculated as follows:

[0134]

[0135] (5) Latency Analysis of Large Language Models

[0136] The first part is the detailed analysis of the delay in the pre-filling phase. The specific calculation formula is as follows:

[0137]

[0138] Among them, latency_attn refers to the computational delay in the self-attention mechanism; compute_latency refers to the latency of computational processing; memory_latency refers to the latency of memory access; flops refers to the number of floating-point operations; memory refers to the amount of memory storage; latency_mlp refers to the computational delay of the feedforward neural network part in Transformer; latency_layernorm refers to the latency of computational layer normalization.

[0139] The normalization module is not a computationally intensive module, so compute_latency in the formula is 0. At the same time, the normalization module has no matrix weights, only vector weights that can be ignored, so memory_latency in the formula is also 0. In summary, latency_layernorm is 0. It is worth noting that there are two normalization layers here.

[0140] The following calculation shows the latency of a single AllReduce communication in a tensor parallel group during the forward pass of the Transformer layer. The AllReduce operation refers to a class of algorithms that aims to efficiently integrate data results from different machines and distribute the integrated results to each machine. The integration here can be a series of operations such as summation, maximum value, minimum value, etc. In addition, the communication volume of an AllReduce operation is It is worth noting that if tp_size is 1, the latency_tp_comm value is 0. If tp_size is not 1, the latency of tensor parallel communication latency_tp_comm can be calculated as follows:

[0141]

[0142] Among them, latency_per_all_reduce refers to the latency of each AllReduce operation.

[0143] The latency per layer of the decoder module in the Transformer model can be calculated as follows:

[0144]

[0145] Among them, latency_attn, latency_mlp, latency_layernorm and latency_tp_comm can be respectively substituted into the calculation results.

[0146] The delay prefill_latency_fwd_attn when calculating the self-attention mechanism in the forward pass of the model can be calculated according to the following formula:

[0147]

[0148] Among them, prefill_latency_fwd_mlp refers to the delay in the feedforward neural network part; prefill_latency_fwd_layernorm refers to the delay of the normalization operation of the computing layer; prefill_latency_fwd_tp_comm refers to the delay of data communication between different computing devices in the tensor parallel process. latency_attn can be calculated by the formula derived from the above content and the result can be entered.

[0149] prefill_latency_fwd_mlp can be calculated as follows:

[0150]

[0151] Where latency_mlp is calculated using the formula derived from the above content and the result is entered.

[0152] Since latency_layernorm is 0, prefill_latency_fwd_layernorm is also 0.

[0153] prefill_latency_fwd_tp_comm can be calculated as follows:

[0154]

[0155] Where latency_tp_comm is calculated using the formula derived from the above content and the result is entered.

[0156] The specific calculation formula of the input embedding layer forward transmission delay prefill_latency_fwd_input_embedding is as follows:

[0157]

[0158] Among them, memory_latency refers to the delay in accessing computer memory; comm_latency refers to the delay in data communication in a distributed computing system.

[0159] For the analysis of the forward pass of the output embedding layer, considering that the operations in this part are computationally intensive, it mainly calculates the unnormalized probabilities, namely logits.

[0160] The specific calculation formula of the output embedding layer forward transmission delay prefill_latency_fwd_output_embedding_loss is as follows:

[0161]

[0162] Among them, flops_efficiency refers to the ratio of the actual number of operations to the theoretical peak number of operations.

[0163] The calculation formula for the delay prefill_latency generated by outputting the first token in the prefill phase is as follows:

[0164]

[0165] Among them, latency_per_layer, prefill_latency_fwd_input_embedding and prefill_latency_fwd_output_embedding_loss are calculated according to the formulas derived from the above content and the results are entered.

[0166] The second part is a detailed analysis of the latency in the decoding phase. The specific calculation formula for latency in the decoding phase is as follows (regardless of whether KV Cache is used, the calculation is performed according to the following formula):

[0167]

[0168] For the standardization module, according to the above analysis, latency_layernorm is 0.

[0169] Referring to the above analysis, if tp_size is 1, the latency_tp_comm value is 0; if tp_size is not 1, regardless of whether KV Cache is used, it can be calculated according to the following formula:

[0170]

[0171] The computational delay decode_latency_fwd_attn of the self-attention mechanism in the decoder can be calculated as follows:

[0172]

[0173] Where latency_attn is calculated according to the formula derived from the above content. decode_latency_fwd_mlp refers to the calculation delay of the feedforward neural network in the decoder; decode_latency_fwd_layernorm refers to the calculation delay of the layer normalization operation in the decoder; decode_latency_fwd_tp_comm refers to the communication delay in the tensor parallel process in the decoder.

[0174] decode_latency_fwd_mlp can be calculated as follows:

[0175]

[0176] The latency_mlp is calculated using the formula derived from the above content and the result is entered.

[0177] Since latency_layernorm is 0, decode_latency_fwd_layernorm is also 0.

[0178] decode_latency_fwd_tp_comm can be calculated as follows:

[0179]

[0180] latency_tp_comm can be calculated using the formula derived from the above content and the result can be entered.

[0181] For the analysis of the forward pass of the input embedding layer, regardless of whether KV Cache is used, the forward pass latency decode_latency_fwd_input_embedding of the input embedding layer in the decoder can be calculated according to the following formula:

[0182]

[0183] For the analysis of the forward pass of the output embedding layer, if KV Cache is used, the calculation formula for the forward pass delay decode_latency_fwd_output_embedding_loss of the output embedding layer in the decoder is as follows:

[0184]

[0185] If KV Cache is not used, the specific calculation formula for decode_latency_fwd_output_embedding_loss is as follows:

[0186]

[0187] The following calculates the latency of the forward pass of the key-value cache in the Transformer layer. If KV Cache is not used, the latency is 0. If KV Cache is used, the value of context_len changes from Traverse to The amount of memory kv_cache_memory_per_gpu used on each GPU to cache the key and value vectors in the self-attention mechanism is calculated according to the following formula:

[0188]

[0189] Here, context_len refers to the context length considered by the model when processing the input sequence.

[0190] The latency kv_cache_latency associated with using the key-value cache is calculated using the following formula:

[0191]

[0192] Among them, hbm_memory_efficiency refers to the ratio of actually available memory to total memory.

[0193] The kv_cache_memory_per_gpu and kv_cache_latency values ​​obtained in each loop are added to the lists kv_cache_memory_list_per_gpu and kv_cache_latency_list respectively. The values ​​of kv_cache_avg_latency and kv_cache_peak_latency are the average and maximum values ​​of the list kv_cache_latency_list respectively.

[0194] Furthermore, decode_avg_latency and decode_peak_latency can be calculated according to the following formula:

[0195]

[0196]

[0197] Among them, kv_cache_memory_list_per_gpu refers to the memory list of the key-value cache on each GPU; kv_cache_latency_list refers to the key-value cache latency list under different context lengths; kv_cache_avg_latency refers to the average latency of the key-value cache operation; kv_cache_peak_latency refers to the maximum latency of the key-value cache operation; decode_avg_latency refers to the average latency of the decoding process; decode_peak_latency refers to the maximum latency of the decoding process; decode_model_latency refers to the total latency of the entire decoding stage.

[0198] Therefore, through a series of automatically generated analysis results, users can adjust certain configuration parameters during subsequent reasoning to improve the overall performance of the large language model reasoning process. At the same time, it helps users estimate the computing and storage resources required for reasoning and select more appropriate hardware resources.

[0199] In this step, a relatively complete theoretical performance analysis model for large language models is constructed based on white box modeling. In practical applications, a series of analysis results can be given in combination with the configuration information of the model and graphics card, so that the reasoning process of the large language model can be better simulated.

[0200] Step 103, combining the configuration parameter data with the configuration analysis result to perform automatic configuration parameter tuning based on reinforcement learning to obtain an optimal parameter configuration;

[0201] This step mainly uses the large model configuration parameter automatic tuning module based on the reinforcement learning algorithm to automatically create and manage tuning instances, and uses dynamic context information and feedback data to continuously optimize the configuration parameters of the application to maximize performance indicators and adapt to changes in hardware and workloads. The processing flow diagram of this process is shown in the figure below. Figure 3 shown.

[0202] Tuning Instance refers to the process of optimizing performance and adjusting parameters in a computer system, software, or application. Its purpose is to improve performance, reduce latency, increase throughput, or improve other key indicators.

[0203] Specifically, the process of automatically tuning the configuration parameters based on reinforcement learning in combination with the configuration parameter data and the configuration analysis results to obtain the optimal parameter configuration can be achieved by executing the following steps S01 to S05:

[0204] Step S01: extracting the tunable configuration parameter information of the application from the configuration parameter data and the configuration analysis results, and creating a tuning instance;

[0205] The application first creates a tuning instance by configuring the parameter automatic tuning module and provides a parameter list and its metadata during the call process. This data includes the configuration parameter data (i.e., model configuration parameters, performance parameters, GPU configuration information) and configuration analysis results in the previous steps, as well as the adjustable configuration parameters of other large models. From this data, the configuration parameters that the application can tune and the detailed information of each parameter can be extracted. The process of creating a tuning instance can be automated (completed by the application itself) or manually configured by the developer.

[0206] Step S02: Filtering the target parameter configuration that has the greatest impact on application performance from the tunable configuration parameter information through the tuning instance;

[0207] During the creation of a tuning instance, the selector module in the automatic configuration parameter tuning module automatically selects key performance parameters (configuration parameters that have a greater impact on the overall performance of the large language model) to find the configuration parameters that have the greatest impact on application performance. In this way, the automatic configuration parameter tuning module can focus on optimizing the most important parameters, thereby improving tuning efficiency and effectiveness.

[0208] Step S03: According to the current running state of the system, automatically recommend the configuration value corresponding to the target parameter configuration to the application through the tuning instance;

[0209] The application interacts with the tuning instance by using interface calls to obtain recommended configuration values. These configuration values ​​are the best settings recommended based on the current state of the system and performance characteristics. The current state refers to the current operating state of the system, and the performance characteristics refer to the characteristics and properties of the system when executing tasks.

[0210] Step S04: obtaining a reward value of the application based on the configuration value feedback, and dynamically adjusting and updating the configuration of the tuning instance based on the reward value;

[0211] Over time, the application can monitor and evaluate the effects and performance of these recommended values. Once the application has collected enough data and experience, it submits reward values ​​to the configuration parameter automatic tuning module by calling the reward setting function. These reward values ​​reflect the actual performance and performance optimization of the configuration values. The configuration parameter automatic tuning module will use these reward values ​​to dynamically adjust and update the configuration of the tuning instance.

[0212] Step S05: loop through the configuration parameter tuning process from steps S02 to S04 until the optimal parameter configuration is obtained.

[0213] The above process is a continuous optimization cycle, which enables the tuning instance to gradually improve its configuration to better meet the performance requirements and goals of the application.

[0214] When the system has sufficient servers or resources, the configuration parameter automatic tuning module provided in this step can provide support for parallel exploration. As an optional embodiment, the system can manage and control multiple servers, and each server can correspond to multiple applications.

[0215] For different hardware types, workloads, and functions, applications may need to be tuned for each case. To manage and optimize these tuning instances, the configuration parameter automatic tuning module provides an automatic scoper. The automatic scoper can automatically create, manage, and determine the scope of each tuning instance based on the detailed context information provided by the application.

[0216] For example, for different hardware types, some applications run better on multi-core CPUs, but others may rely more on GPU acceleration. For different workloads, for example, the optimization strategy of an application when processing read-intensive workloads may be different from the optimization strategy when processing write-intensive workloads. For different functions, for example, the data processing module in an application may need to optimize computing efficiency, while the user interface module may focus more on improving response time.

[0217] Specifically, the automatic scoper can dynamically configure and adjust the parameters and settings of the tuning instance based on the hardware specifications, specific workload characteristics, and the specific functions required by the application. This allows each tuning instance to maximize its performance and adjust according to actual conditions to meet the specific requirements and expected goals of the application.

[0218] With the automatic scoping tool, applications can flexibly adapt and utilize the provided tuning capabilities under different environments and requirements, thereby improving the overall efficiency and performance of the system.

[0219] In a specific implementation, for first target applications with different tuning focuses, an automatic range definer can be used to automatically create, manage and determine the applicable scope of the tuning instance corresponding to each tuning focus based on the tunable configuration parameter information of each first target application.

[0220] In the case of heterogeneous servers (i.e., servers that are configured differently from each other), each server may have different hardware characteristics, workload requirements, or functional requirements. To effectively manage this diversity, the automatic scoper of the configuration parameter automatic tuning module classifies and groups each server according to its specific contextual attributes and tunes them separately. This classification and grouping process enables each server to obtain the most optimized configuration to maximize its performance potential.

[0221] The specific context attributes of the server refer to configuration information, status information or other relevant data related to a specific scenario or application during the operation of the server. These attributes can be used to describe information such as the configuration, status, and performance characteristics of the server.

[0222] In a specific implementation, for first servers with different configurations, firstly, an automatic range definer can be used to classify and group all second target applications according to the tunable configuration parameter information of the second target application in each first server to obtain a number of second target application groups; a heterogeneous tuning instance is created for each second target application group, and with reference to steps S02 to S05, parameter configuration tuning is performed on the second target application group through the heterogeneous tuning instance.

[0223] For servers with the same configuration, multiple tuning processes can be performed simultaneously through a single tuning instance. This parallel exploration enables the system to optimize its performance and configuration in a more efficient way, quickly finding the best parameter settings.

[0224] Therefore, in a specific implementation, for second servers with the same configuration, steps S01 to S05 can be referred to, and parameter configuration tuning can be performed on each second server simultaneously through the created single tuning instance, so that each second server obtains the most optimized parameter configuration.

[0225] That is to say, multiple tuning instances can be managed and configured at the same time through the automatic scoper. This means that the configuration parameter automatic tuning module can automatically optimize multiple tuning instances at the same time. Furthermore, under the same configuration server, a single tuning instance can correspond to multiple parallel tuning processes.

[0226] The automatic configuration parameter tuning module allows you to continuously receive feedback from applications to adjust the system's configuration. This feedback includes evaluating the system's responsiveness under high load based on various performance indicators. Specifically, the automatic configuration parameter tuning module collects and analyzes multiple hours of feedback data to understand the system's performance and resource utilization. Based on this data, the allocation of memory and computing resources can be dynamically adjusted to optimize the overall efficiency and stability of the system. This automated adjustment allows the system to maintain efficient operation in the face of changing request characteristics and load patterns.

[0227] In this step, based on the configuration parameter automatic tuning module, the configuration that needs to be tuned is determined, the scope of configuration tuning is automatically managed, and the reinforcement learning algorithm is used to quickly adjust the numerical and classification configurations at the same time, thereby reducing the configuration tuning overhead. At the same time, combined with the output results of the static optimization analysis in the previous steps, the state and reward in the reinforcement learning algorithm can be directly calculated, thereby avoiding time-consuming and labor-intensive experiments and realizing efficient configuration tuning for large language model training and reasoning.

[0228] Step 104, dynamically optimizing the computing resource data based on the offloading strategy to obtain optimal resource configuration;

[0229] This step mainly provides a dynamic optimization module based on the offloading strategy. Through the application of the offloading strategy, the computing and storage resources (computing resource data) from the CPU, GPU and hard disk are aggregated and flexibly configured under the constraints of hardware resources to improve the throughput of large model reasoning on a single graphics card. At the same time, users can also dynamically adjust the weights, activation values ​​and the proportion of KV Cache on the CPU, GPU and hard disk to further optimize the overall performance of large model reasoning. The processing flow diagram in the dynamic optimization module based on the offloading strategy is shown in the figure below. Figure 4 shown.

[0230] First, the large model reasoning problem that introduces the offloading strategy is abstracted into a graph traversal problem. Under certain constraints, an effective path is found to traverse each square block abstracted in the computational graph.

[0231] Among them, the large model inference problem refers to the problems or challenges encountered when using large pre-trained models (such as deep learning models) for inference. These models often have high computational complexity and resource requirements due to their large size.

[0232] Assume that the GPU, CPU, and hard disk constitute a three-level storage structure of a machine, such as Figure 5 As shown. Figure 5 It can be seen that the storage space size and data reading and writing speed of GPU, CPU and hard disk are different. The GPU's video memory is relatively small, but its data reading and writing speed is the fastest. Similarly, the hard disk has the largest storage space, but its data reading and writing speed is the slowest.

[0233] It is worth noting that in the assumed three-level storage structure, only the GPU and CPU have computing power, so the GPU alone cannot load the complete large model. The large model needs to be unloaded to the auxiliary storage device. At the same time, the computing task is gradually completed by partially loading the model. Therefore, in order to solve the problem of large resources required for large model reasoning, the embodiment of the present invention introduces an unloading strategy. The large model reasoning problem that introduces the unloading strategy can be abstracted as a graph traversal problem. The computational graph representation of the large language model during model reasoning is as follows: Figure 6 shown.

[0234] Assume that three tokens are returned for each prompt word in the batch. Figure 6 In the figure, each square represents the batch processing amount of the GPU on one layer. Since the main purpose of the dynamic optimization module based on the offloading strategy is to improve the throughput of large model inference, the abstract graph traversal problem does not focus on how many batches need to be processed. It should be pointed out that in the squares of the same color, the weight values ​​of the model are the same.

[0235] As described above, the embodiment of the present invention abstracts the large model reasoning problem that introduces the offloading strategy into a graph traversal problem. In the graph traversal problem, the definition of a valid path is very important. The valid path defined here needs to traverse each abstracted square block and is subject to the following four conditions:

[0236] (a) In Figure 6 When a square is about to be traversed, it is necessary to ensure that all the squares in the same row and on the left of the square have been traversed. That is, these squares have been calculated by the GPU.

[0237] (b) In Figure 6 In the graph traversal problem shown in the figure, a square represents the batch computation of the GPU in one layer of the model. In the process of traversing the square (that is, in the process of the GPU calculating each abstracted square), in order to calculate the square in the figure on the same device, all the inputs of the square need to be loaded onto the same device. The inputs mentioned here include activation values, weight values, and caches.

[0238] (c) In Figure 6 In the example, when a square block completes the calculation, the square block will generate an activation value and KVCache output. These outputs should be stored in time. According to the principle of large model reasoning, the output of each token in the answer given by the large language model depends on all tokens generated before. For the activation value, it needs to be stored in the square block to the right of the square block to complete the calculation. For KV Cache, it needs to be stored in the rightmost square block in the same row as the square block to complete the calculation.

[0239] (d) When traversing all the squares in the graph (i.e. during the GPU calculation process), many tensors will be generated. These tensors need to be stored in the device for use. However, the total size of the tensors stored in the device cannot exceed the memory capacity of the device.

[0240] Based on the above four constraints, it can be further verified that Figure 6 Based on the abstract graph traversal problem, the definition and selection of valid paths are very important. The selection of valid paths not only needs to meet the basic definition (i.e. traversing every square block in the graph), but also needs to meet the four constraints mentioned above. More importantly, the selection of valid paths needs to minimize the total execution time. The total execution time includes not only the time cost of calculation, but also the time cost of input and output.

[0241] Secondly, a multi-parameter search space is constructed. For computational scheduling, the block granularity is mainly added to the computational graph, and a zigzag block scheduling is performed in each block. Therefore, the embodiment of the present invention introduces two parameters, the batch size and the block size of the GPU. At the same time, in order to consider the problem of tensor storage, 9 parameter variables are also introduced, namely wg, wc, wd, hg, hc, hd, cg, cc and cd. The 9 introduced parameter variables together with the batch size and block size of the GPU constitute a multi-parameter search space. Among them, wg, wc, and wd represent the percentages of model weights stored on the GPU, CPU, and hard disk respectively; hg, hc, and hd represent the percentages of activation values ​​stored on the GPU, CPU, and hard disk respectively; cg, cc, and cd represent the percentages of KV Cache stored on the GPU, CPU, and hard disk respectively.

[0242] In actual calculations, in addition to considering the impact of percentages, it is also necessary to consider the impact of granularity. Model granularity is reflected in the percentage of layers of the model allocated to the GPU. Layer granularity is reflected in the percentage of tensors allocated to the GPU in a layer. Tensor granularity is reflected in the percentage of elements of the tensor allocated to the GPU. By conducting experiments on different granularities, different granularities are selected based on the experimental results and comprehensive consideration of factors such as overhead. In an embodiment of the present invention, layer granularity is used for weights, while tensor granularity is used for activation values ​​and KV Cache.

[0243] An effective offloading strategy should consider how to perform efficient computational scheduling and store tensors reasonably. In this process, multiple influencing parameter variables are also introduced. An offloading strategy should include the above 11 variables. Based on the binary group consisting of the fixed GPU batch size and block size, the optimal combination of the remaining influencing parameter variables to be solved is determined by solving the linear programming problem (that is, the search parameter group composed of each group of influencing parameter variables is searched and calculated in each square block through an effective path, and the final optimal combination of parameter variables can be determined after the traversal is completed). Finally, the optimizer is called to obtain an offloading strategy. On this basis, the pipeline parallel method can also be used to further improve the throughput and reduce the burden on a single graphics card.

[0244] At the same time, the dynamic optimization module also includes a Group-wise quantization method. Specifically, the weights and elements in the KV Cache are compressed into 4-bit integers to reduce communication costs. Before subsequent calculations, each element is decompressed and restored to FP16 half-precision floating point numbers. Although this process will lose some data accuracy, within the allowable error range, a suitable model approximation method is more conducive to improving the throughput of large model reasoning.

[0245] Based on the above introduction, in the specific implementation, the computing resource data is dynamically optimized based on the offloading strategy to obtain the optimal resource configuration. First, the large model reasoning problem encountered by the large language model in the model reasoning process is abstracted into a graph traversal problem; then the graph traversal problem is solved by multi-parameter search to obtain the offloading strategy. Among them, the offloading strategy gives the optimal resource configuration when the computing resource data is offloaded to the auxiliary storage device.

[0246] Furthermore, a multi-parameter search is performed on the graph traversal problem to obtain an unloading strategy, which can be specifically as follows: the original parameter variables are determined, and multiple influencing variables are introduced at the same time, a multi-parameter search space is jointly constructed according to the original parameter variables and the multiple influencing variables, and different search granularities are set for different search parameters; under preset constraints, a valid path is determined when performing graph traversal, and the valid path is used to traverse each square block abstracted in the calculation graph, and each square block corresponds to a search parameter group; on the basis of fixing the binary group composed of the original parameter variables, a graph traversal-style parameter search calculation is performed on each search parameter group through an effective path to solve the linear programming problem, determine the optimal parameter combination of multiple influencing variables, and obtain an unloading strategy.

[0247] In this step, by setting up a dynamic optimization module based on the offloading strategy, the problems of inefficient input and output scheduling and tensor placement can be solved, and the resources of the CPU, GPU, and hard disk can be fully utilized, which greatly improves the throughput of large language model inference.

[0248] Step 105: Dynamically optimize the parameter configuration and resource allocation of the large language model during model inference according to the optimal parameter configuration and the optimal resource configuration.

[0249] Specifically, according to the optimal parameter configuration and the optimal resource configuration, the parameter configuration and resource allocation of the large language model during model inference are dynamically optimized, which can be as follows: according to the optimal parameter configuration, the configurable parameters of the large language model during model inference are dynamically adjusted to maximize system performance indicators and adapt to changes in hardware and workload; according to the unloading strategy corresponding to the optimal resource configuration, the computing resource data of the large language model during model inference is unloaded to the auxiliary storage device to improve the throughput during model inference.

[0250] It should be noted that, in order to enable those skilled in the art to better distinguish data of the same type but with different actual meanings, some technical features are distinguished by the first and the second in the embodiments of the present invention. The first and the second are only used for data distinction and have no other special meanings. It can be understood that the present invention is not limited to this.

[0251] In an embodiment of the present invention, firstly, static optimization analysis of the configuration parameters is performed based on the white box modeling method, and quantitative analysis results are given through a series of linear equations combined with the configuration information of the model and the graphics card, so that users can modify some of the parameters to optimize the reasoning performance of the large language model. At the same time, the reinforcement learning algorithm is combined to realize automatic parameter tuning to further optimize some of the configuration parameters. Secondly, dynamic optimization is realized based on the offloading strategy. Through the application of the offloading strategy, the computing and storage resources from the CPU, GPU and hard disk are summarized and flexibly configured under the constraints of hardware resources. Through the technical solution of the present invention, not only the throughput of large model reasoning can be improved, but also the automatic tuning of configuration parameters can be realized. It can also help users to adjust certain configuration parameters in subsequent model reasoning to improve the overall performance of the large model reasoning process.

[0252] Through the technical solution provided by the embodiment of the present invention, a complete system-level large language model reasoning optimization framework can be formed. For ease of understanding, the technical effect brought by the technical solution of the present invention is verified and explained through a specific example below.

[0253] The models used in the experiment are all open source mainstream large language models commonly used by various enterprises, universities, etc. Their configuration parameters are all real parameters of large language models.

[0254] The following will take the various indicators of memory analysis of a large language model as an example. Specifically, the large language model analyzed here is llama-13b, and the graphics card used is v100-sxm-32gb. The theoretical analysis results of the large model obtained by the static optimization module based on white box modeling technology, and the optimization results obtained after these analysis results are passed through the configuration parameter automatic tuning module based on the reinforcement learning algorithm are shown in Table 5 below:

[0255]

[0256] Table 5: Comparison of optimization of memory analysis indicators for large language models

[0257] It can be seen that compared with the original indicators, after static optimization analysis and automatic tuning of configuration parameters, all indicators have been significantly improved, with good optimization effects.

[0258] For large language models with different configurations from the above models, the method provided by the embodiment of the present invention is also effective. Taking the large language model OPT-30B and the graphics card a-100 as an example, the experimental results of the dynamic optimization module based on the offloading strategy are shown in Tables 6 and 7 below:

[0259]

[0260] Table 6: Experimental parameter description of dynamic optimization module experiment

[0261]

[0262] Table 7: Experimental results of dynamic optimization module

[0263] Table 6 shows the meaning and numerical values ​​of the experimental parameters. Through this experiment, it can be concluded that the weight percentage on the GPU is 20%, the weight percentage on the CPU is 80%, the attention cache percentage on the GPU is 100%, and the activation value percentage on the CPU is 100%. Table 7 shows the inference performance of the large language model in real-world operation under these experimental parameters.

[0264] Reference Figure 7 , shows a structural block diagram of a large language model reasoning optimization device provided by an embodiment of the present invention, which may specifically include:

[0265] The data acquisition module 701 is used to acquire the configuration parameter data and computing resource data of the large language model in real time during model inference;

[0266] Static optimization analysis module 702, used to perform static optimization analysis on the configuration parameter data and output configuration analysis results;

[0267] A configuration parameter automatic tuning module 703, used to perform automatic tuning of configuration parameters based on reinforcement learning in combination with the configuration parameter data and the configuration analysis result to obtain an optimal parameter configuration;

[0268] A dynamic optimization processing module 704 is used to perform dynamic optimization processing on the computing resource data based on the offloading strategy to obtain the optimal resource configuration;

[0269] The model reasoning optimization module 705 is used to dynamically optimize the parameter configuration and resource allocation of the large language model during model reasoning according to the optimal parameter configuration and the optimal resource configuration.

[0270] In an optional embodiment, the configuration parameter automatic tuning module 703 includes:

[0271] The information extraction module is used to execute step S01: extract the tunable configuration parameter information of the application from the configuration parameter data and the configuration analysis result, and create a tuning instance;

[0272] An information screening module, used to execute step S02: screening the target parameter configuration having the greatest impact on the performance of the application program from the tunable configuration parameter information through the tuning instance;

[0273] The parameter configuration automatic recommendation module is used to execute step S03: according to the current running state of the system, automatically recommend the configuration value corresponding to the target parameter configuration to the application through the tuning instance;

[0274] The tuning instance adjustment module is used to execute step S04: obtaining a reward value of the application based on the configuration value feedback, and dynamically adjusting and updating the configuration of the tuning instance based on the reward value;

[0275] The configuration parameter tuning loop module is used to execute step S05: looping the configuration parameter tuning process of steps S02 to S04 until the optimal parameter configuration is obtained.

[0276] In an optional embodiment, the system manages and controls a plurality of servers, each of the servers corresponds to a plurality of applications, and the device further comprises:

[0277] A tuning instance application scope determination module is used to automatically create, manage and determine the application scope of the tuning instance corresponding to each tuning focus according to the tunable configuration parameter information of each of the first target applications with different tuning focuses through an automatic scope definer;

[0278] an application classification module, for first servers with different configurations, classifying and grouping all second target applications according to tunable configuration parameter information of the second target application in each of the first servers through an automatic range definer, to obtain a plurality of second target application groups;

[0279] A first parameter configuration and tuning module is used to create a heterogeneous tuning instance for each second target application group, and perform parameter configuration and tuning on the second target application group through the heterogeneous tuning instance with reference to steps S02 to S05;

[0280] The second parameter configuration tuning module is used to perform parameter configuration tuning on each second server with the same configuration by creating a single tuning instance, referring to steps S01 to S05, so that each second server obtains the optimal parameter configuration.

[0281] In an optional embodiment, the dynamic optimization processing module 704 includes:

[0282] A model reasoning problem abstraction module, used for abstracting the large model reasoning problem encountered by the large language model during the model reasoning process into a graph traversal problem;

[0283] The multi-parameter search and solution module is used to perform a multi-parameter search and solution on the graph traversal problem to obtain an unloading strategy, wherein the unloading strategy provides an optimal resource configuration when unloading the computing resource data to an auxiliary storage device.

[0284] In an optional embodiment, the multi-parameter search solution module includes:

[0285] A multi-parameter search space construction module is used to determine the original parameter variables and introduce multiple influencing variables, jointly construct a multi-parameter search space according to the original parameter variables and the multiple influencing variables, and set different search granularities for different search parameters;

[0286] An effective path determination module, used to determine an effective path for graph traversal under preset constraints, wherein the effective path is used to traverse each square block abstracted from the computation graph, and each square block corresponds to a search parameter group;

[0287] The parameter search calculation module is used to perform graph traversal parameter search calculation on each search parameter group through the effective path on the basis of fixing the binary group composed of the original parameter variables, so as to solve the linear programming problem, determine the optimal parameter combination of the multiple influencing variables, and obtain the unloading strategy.

[0288] In an optional embodiment, the model reasoning optimization module 705 includes:

[0289] A configurable parameter dynamic adjustment module, used to dynamically adjust the configurable parameters of the large language model during model inference according to the optimal parameter configuration, so as to maximize system performance indicators and adapt to changes in hardware and workload;

[0290] The resource unloading module is used to unload the computing resource data of the large language model during model inference to an auxiliary storage device according to the unloading strategy corresponding to the optimal resource configuration, so as to improve the throughput during model inference.

[0291] In an optional embodiment, the configuration parameter data includes GPU configuration information of the system, model configuration information and model architecture information of the large language model, and the static optimization analysis module 702 is specifically used to:

[0292] Inputting the GPU configuration information, the model configuration information and the model architecture information into a multi-level analysis model pre-built based on a white box modeling method to perform static optimization analysis, and outputting a configuration analysis result;

[0293] The static optimization analysis includes inference configuration analysis, parameter quantity analysis, computational quantity analysis, memory analysis and latency analysis.

[0294] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the aforementioned method embodiment.

[0295] An embodiment of the present invention further provides an electronic device, the device comprising a processor and a memory:

[0296] The memory is used to store the program code and transmit the program code to the processor;

[0297] The processor is used to execute the large language model reasoning optimization method of any embodiment of the present invention according to the instructions in the program code.

[0298] An embodiment of the present invention further provides a computer-readable storage medium, which is used to store program code, and the program code is used to execute the large language model reasoning optimization method of any embodiment of the present invention.

[0299] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0300] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0301] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0302] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0303] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0304] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A large language model reasoning optimization method, characterized in that: include: Obtain configuration parameter data and computing resource data of large language models in real time during model inference; Performing static optimization analysis on the configuration parameter data and outputting configuration analysis results; Combining the configuration parameter data with the configuration analysis result, automatic tuning of the configuration parameters based on reinforcement learning is performed to obtain an optimal parameter configuration; Performing dynamic optimization processing on the computing resource data based on the offloading strategy to obtain optimal resource configuration; According to the optimal parameter configuration and the optimal resource configuration, the parameter configuration and resource allocation of the large language model during model inference are dynamically optimized.

2. The large language model reasoning optimization method according to claim 1, characterized in that: The step of automatically tuning the configuration parameters based on reinforcement learning in combination with the configuration parameter data and the configuration analysis result to obtain the optimal parameter configuration includes: Step S01: extracting tunable configuration parameter information of the application from the configuration parameter data and the configuration analysis result, and creating a tuning instance; Step S02: Filtering the target parameter configuration that has the greatest impact on the performance of the application program from the tunable configuration parameter information through the tuning instance; Step S03: According to the current running state of the system, automatically recommending the configuration value corresponding to the target parameter configuration to the application through the tuning instance; Step S04: obtaining a reward value of the application based on the configuration value feedback, and dynamically adjusting and updating the configuration of the tuning instance based on the reward value; Step S05: cyclically execute the configuration parameter tuning process of steps S02 to S04 until the optimal parameter configuration is obtained.

3. The large language model reasoning optimization method according to claim 2, characterized in that: The system manages and controls a plurality of servers, each of the servers corresponding to a plurality of applications, and the method further comprises: For first target applications with different tuning priorities, an automatic scope definer is used to automatically create, manage and determine the applicable scope of a tuning instance corresponding to each tuning priority according to tunable configuration parameter information of each first target application; For first servers with different configurations, classifying and grouping all second target applications according to tunable configuration parameter information of the second target application in each of the first servers through an automatic range definer, to obtain a plurality of second target application groups; Create a heterogeneous tuning instance for each of the second target application groups, and refer to steps S02 to S05 to perform parameter configuration tuning on the second target application group through the heterogeneous tuning instance; For second servers with the same configuration, referring to steps S01 to S05, parameter configuration tuning is performed on each second server at the same time through the created single tuning instance, so that each second server obtains the most optimized parameter configuration.

4. The large language model reasoning optimization method according to claim 2, characterized in that: The dynamically optimizing the computing resource data based on the offloading strategy to obtain the optimal resource configuration includes: Abstracting the large model reasoning problem encountered by the large language model during the model reasoning process into a graph traversal problem; A multi-parameter search is performed on the graph traversal problem to obtain an offloading strategy, wherein the offloading strategy provides an optimal resource configuration when the computing resource data is offloaded to an auxiliary storage device.

5. The large language model reasoning optimization method according to claim 4, characterized in that: The performing a multi-parameter search and solving of the graph traversal problem to obtain an unloading strategy includes: Determine the original parameter variables, introduce multiple influencing variables, jointly construct a multi-parameter search space according to the original parameter variables and the multiple influencing variables, and set different search granularities for different search parameters; Determine a valid path for graph traversal under preset constraints, where the valid path is used to traverse each square block abstracted from the computation graph, each of which corresponds to a search parameter group; On the basis of fixing the binary group consisting of the original parameter variables, a graph traversal parameter search calculation is performed on each of the search parameter groups through the effective path to solve the linear programming problem, determine the optimal parameter combination of the multiple influencing variables, and obtain the unloading strategy.

6. The large language model reasoning optimization method according to claim 4 or 5, characterized in that: The dynamically optimizing the parameter configuration and resource allocation of the large language model during model inference according to the optimal parameter configuration and the optimal resource configuration includes: According to the optimal parameter configuration, dynamically adjust the configurable parameters of the large language model during model inference to maximize system performance indicators and adapt to changes in hardware and workload; According to the unloading strategy corresponding to the optimal resource configuration, the computing resource data of the large language model during model inference is unloaded to the auxiliary storage device to improve the throughput during model inference.

7. The large language model reasoning optimization method according to claim 1, characterized in that: The configuration parameter data includes GPU configuration information of the system, model configuration information and model architecture information of the large language model, and the static optimization analysis of the configuration parameter data and outputting the configuration analysis result include: Inputting the GPU configuration information, the model configuration information and the model architecture information into a multi-level analysis model pre-built based on a white box modeling method to perform static optimization analysis, and outputting a configuration analysis result; The static optimization analysis includes reasoning configuration analysis, parameter quantity analysis, computational quantity analysis, memory analysis and latency analysis.

8. A large language model reasoning optimization device, characterized in that: include: The data acquisition module is used to obtain the configuration parameter data and computing resource data of the large language model in real time during model inference; A static optimization analysis module, used to perform static optimization analysis on the configuration parameter data and output configuration analysis results; A configuration parameter automatic tuning module, used to combine the configuration parameter data with the configuration analysis result to perform automatic tuning of the configuration parameters based on reinforcement learning to obtain the optimal parameter configuration; A dynamic optimization processing module, used to perform dynamic optimization processing on the computing resource data based on the offloading strategy to obtain the optimal resource configuration; The model reasoning optimization module is used to dynamically optimize the parameter configuration and resource allocation of the large language model during model reasoning according to the optimal parameter configuration and the optimal resource configuration.

9. An electronic device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the large language model reasoning optimization method described in any one of claims 1-7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the large language model reasoning optimization method described in any one of claims 1-7.

Images