Cost-aware large model deployment optimization method and system in cloud environment

Abstract

The invention discloses a cost-aware large model deployment optimization method and system in a cloud environment, and the method comprises the steps: initializing a memory cache region DP, and achieving the deployment optimization of a large model through the combination of pipeline parallelism and tensor parallelism based on a given allocated GPU set A and an unallocated GPU set U by using a recursive function Estimate-Cost. Parameters needing to be introduced by the recursive function Estimate-Cost comprise a memory cache region DP, the number of layers j of a large model, an allocated GPU set A and an unallocated GPU set U. According to the cross-cloud cooperation mechanism, multiple neural network layers in the large model are reasonably divided for the cloud computing environment so that heterogeneous GPU deployment can be carried out, the computing cost generated by large model reasoning can be reduced, a user can be effectively helped to select a proper heterogeneous deployment scheme, and the computing cost of large model reasoning service can be reduced.

Description

Technical Field

[0001] The present invention relates to a large model deployment collaboration technology in a cloud computing environment, and in particular to a cost-aware large model deployment optimization method and system in a cloud environment. Background Art

[0002] Cloud computing is a network-based computing model. Its core concept is to provide computing resources (including computing resources, GPU, storage and network resources, etc.) to users in the form of services through virtualization technology. Users can obtain and release these resources according to their needs. Joint Cloud Computing is based on open collaboration between multiple cloud service providers. Through the deep integration of multi-party cloud resources, it is convenient for developers to customize cloud services through "software definition", so that developers can use the resources of multiple cloud vendors to create value beyond the cloud itself, and realize the new generation of cloud computing model of "borderless services, collaboration between clouds, easy resource sharing, and convertible value". Joint Cloud Computing is one of the important ways to solve the high cost-effectiveness caused by the current high demand for computing power.

[0003] The development of large models has attracted much attention in the field of machine learning. With the improvement of computing power and the growth of data sets, the training and deployment of large models have become more feasible. Model scaling has been proven to be one of the most important factors in achieving higher accuracy, and its effectiveness has been consistently demonstrated in the practice of ultra-large models in all fields of industry, such as Vision Model BiT, NLP Model BERT, GPT3, and Deep Learning Recommendation Model DLRM. The development of large models has brought great challenges to computing power, and these models usually contain billions or even tens of billions of parameters. Training these large models and deploying these large models to provide inference services require a lot of computing resources and storage space. Therefore, such large models require a lot of computing resources to work together to provide stable services to the outside world.

[0004] In terms of model inference, large models are usually deployed as online services to respond to user requests. Large model inference services usually have high expectations for response latency and inference accuracy. Large model applications that fail to complete within the specified time or have lower accuracy than expected have little commercial value. Therefore, an effective scheduler to balance inference latency, accuracy, and cost is crucial. In actual inference service scenarios, the amount of user requests for inference services is huge. Faced with such a huge amount of user requests, it is impossible to provide services to users with low latency and high accuracy without high-performance computing power. Therefore, state-of-the-art model inference services are usually hosted in centralized data centers with homogeneous high-performance GPUs, which can be very expensive in terms of cloud service fees. The high cost of centralized homogeneous deployment in a single cloud may limit the expansion of large model inference services.

[0005] However, the computing power and billing standards of GPU computing resources provided by different cloud service providers vary, and the service coverage and data transmission fees of different cloud service providers vary worldwide. We can effectively reduce the overall cost of large model inference services by combining different types of computing resources in the cloud, flexibly selecting and dynamically adjusting resources, taking advantage of price competition, optimizing geographical location and data transmission, and improving resource utilization. Deploying large model inference on a cloud heterogeneous environment may be a feasible direction to reduce the inference cost in terms of cloud service fees. Large model service systems should strive to meet service level goals while minimizing the cost of providing services in the cloud.

[0006] Therefore, there is an urgent need to design a deployment optimization mechanism that can take into account the computing power, price, and communication transmission of each cloud service provider and deploy large models across clouds in an inter-cloud environment to achieve the best cost-effectiveness. Summary of the invention

[0007] Technical problem to be solved by the present invention: In view of the above-mentioned problems of the prior art, a cost-aware large model deployment optimization method and system in a cloud environment are provided. The present invention aims to reasonably divide multiple neural network layers in a large model for a cloud computing environment so as to perform heterogeneous GPU deployment, thereby reducing the computing cost generated by large model reasoning. The cross-cloud collaboration mechanism can effectively help users choose appropriate heterogeneous deployment solutions to reduce the computing cost of large model reasoning services.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0009] A cost-aware large model deployment optimization method in a cloud environment includes initializing a memory cache DP, and using a recursive function Estimate-Cost based on a given allocated GPU set A and an unallocated GPU set U to achieve deployment optimization of the large model by combining pipeline parallelism and tensor parallelism. The parameters that need to be input into the recursive function Estimate-Cost include the memory cache DP, the number of layers j of the large model, the allocated GPU set A and the unallocated GPU set U, and the initial value of the number of layers j of the large model is set to 1. The execution steps of the recursive function Estimate-Cost include:

[0010] S1, if the jth layer of the large model has not reached the last layer of the model and the unallocated GPU set U is not empty, jump to step S2; otherwise, output the allocated GPU set Cur_Set and current cost of each layer recorded in the memory buffer DP, end and exit;

[0011] S2, traverse the unallocated GPU set U and try to add different combinations of each type of GPU in it to the allocated GPU set Cur_Set, and calculate the computing delay and memory consumption of the large model according to the allocated GPU set Cur_Set. If the computing delay of the large model violates the user-defined delay requirement, or the memory consumption of any GPU in the allocated GPU set Cur_Set is violated, then the current cost c′ of the jth layer of the large model is recorded as positive infinity; otherwise, the current cost c′ of the jth layer of the large model is calculated according to the following formula;

[0012] c′=Cost(Cur_Set)+DP[[j-1,A]],

[0013] In the above formula, Cost is a preset cost function, DP[[j-1,A] represents the cost of the j-1th layer of the large model recorded in the memory buffer DP, and the preset cost function is composed of the computational cost and the communication cost;

[0014] S3, determine whether the current cost c′ is less than the known minimum cost recorded in the memory buffer DP, if so, store the allocated GPU set Cur_Set of the jth layer and the current cost c′ in the memory buffer DP;

[0015] S4, add 1 to the number of layers j of the large model, add the allocated GPU set A to the allocated GPU set Cur_Set as the new allocated GPU set A, subtract the allocated GPU set Cur_Set from the unallocated GPU set U as the new unallocated GPU set U, and recursively call the recursive function Estimate-Cost according to the new number of layers j of the large model, the allocated GPU set A and the unallocated GPU set U.

[0016] Optionally, when the computation delay and memory consumption of the large model are respectively calculated according to the allocated GPU set Cur_Set in step S2, the function expression of the computation delay is:

[0017]

[0018] In the above formula, represents the computational delay of the large model, Represented as n on cloud c * GPUs for computing large models. express The number of GPUs, express The inference computation time is is n on cloud c * GPUs serving layer j of the large model, express The tensor parallel time is express The pipeline parallel time is calculated according to the allocated GPU set Cur_Set in step S2. When the computing delay and memory consumption of the large model are calculated respectively, the function expression for calculating the memory consumption is:

[0019]

[0020] In the above formula, is the memory consumption, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models. Expressed as The number of GPUs, represents the GPU that provides computing services for the jth layer of the large model, l j Indicates the total number of neural network layers currently served by the GPU that currently provides computing services for the jth layer of the large model, b t Indicates the batch size of large model inference service, The sequence length of the input prompt for the large model, The sequence length of output token generation for large models.

[0021] Optionally, the calculation function expression of the inference calculation time is:

[0022]

[0023] In the above formula, max means taking the maximum value, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models. The sequence length of output tokens generated for large models, Expressed as The number of GPUs, represents the GPU that provides computing services for the jth layer of the large model. represents the memory bandwidth of the nth GPU on cloud c, l j Indicates the total number of neural network layers currently served by the GPU that currently provides computing services for the jth layer of the large model, b t Indicates the batch size of large model inference service, The sequence length of the input prompt for the large model, represents the tensor core computing capability of the nth GPU on cloud c. for The GPU in.

[0024] Optionally, the calculation function expression of the tensor parallel time is:

[0025]

[0026] In the above formula, max means taking the maximum value. express Except GPU, express and The time delay between the two, b t Indicates the batch size of large model inference service, is the sequence length of the input prompt of the large model, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models. Expressed as The number of GPUs, It represents the GPU that provides computing services for the jth layer of the large model. express and The transmission bandwidth between the two, l j Indicates the total number of neural network layers currently served by the GPU that currently provides computing services for the jth layer of the large model. The sequence length of output tokens generated for large models, and Both The GPU in.

[0027] Optionally, the calculation function expression of the pipeline parallel time is:

[0028]

[0029] In the above formula, min means taking the minimum value. express and The time delay between the two, b t represents the batch size of the large model inference service, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models. express and The transmission bandwidth between the two express Except GPU, The sequence length of output tokens generated for large models, for GPU in, for GPU in, is n on cloud c * GPUs serving layer j of the large model, is n on cloud c * A GPU serving the j+1th layer of the large model.

[0030] Optionally, the function expression of the cost function preset in step S2 is:

[0031]

[0032] In the above formula, express The cost, Represented as n on cloud c * GPUs for computing large models. express The computational cost of express communication costs.

[0033] Optionally, the calculation function expression of the calculation cost is:

[0034]

[0035] In the above formula, Represented as n on cloud c * GPUs for computing large models. express The number of GPUs, express The inference computation time is is n on cloud c *GPUs serving layer j of the large model, represents the price of the nth GPU on cloud c;

[0036] The calculation function expression of the communication cost is:

[0037]

[0038] In the above formula, express The communication data size of tensor parallel communication, is the data transmission price between the nth GPU and the n′th GPU, It represents the set of GPUs that provide computing services at layer j+1 of the large model. is the communication data size of pipeline parallel communication, and:

[0039]

[0040] In the above formula, b t Indicates the batch size of large model inference service, is the sequence length of the input prompt of the large model, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models. Expressed as The number of GPUs, represents the GPU that provides computing services for the jth layer of the large model, The sequence length of output token generation for large models.

[0041] In addition, the present invention also provides a cost-aware large model deployment optimization system in a cloud environment, including an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the cost-aware large model deployment optimization method in the cloud environment.

[0042] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the cost-aware large model deployment optimization method in the cloud environment through a processor.

[0043] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the cost-aware large model deployment optimization method in a cloud environment through a processor.

[0044] Compared with the prior art, the present invention mainly has the following advantages: the cloud service computing cost of large model reasoning deployment is high, and the GPU computing resources provided by different cloud service providers have certain differences in performance and pricing, which makes heterogeneous GPU deployment of large model reasoning services a necessary issue. The cost-aware large model deployment optimization method in the cloud environment of the present invention proposes a cost-aware large model deployment optimization mechanism in the cloud environment, and constructs a comprehensive cost model for large model heterogeneous GPU deployment. Drawing on relevant algorithms such as dynamic programming, a deployment scheme search algorithm is proposed, which adopts a method combining pipeline parallelism and tensor parallelism, and can effectively help users choose appropriate heterogeneous deployment schemes to reduce the computing cost of large model reasoning services. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagram of the basic flow of the method of the embodiment of the present invention.

[0046] Figure 2 The figure is a schematic diagram of the application process of the method according to the embodiment of the present invention. DETAILED DESCRIPTION

[0047] like Figure 1 As shown, the cost-aware large model deployment optimization method in the cloud environment of this embodiment includes initializing the memory cache area DP, and using the recursive function Estimate-Cost based on the given allocated GPU set A and the unallocated GPU set U to achieve the deployment optimization of the large model using a combination of pipeline parallelism and tensor parallelism. The parameters that need to be passed in to the recursive function Estimate-Cost include the memory cache area DP, the number of layers j of the large model, the allocated GPU set A and the unallocated GPU set U, and the initial value of the number of layers j of the large model is set to 1, that is: Estimate-Cost(DP,j=1,A,U).

[0048] The memory buffer DP is used to record the GPU sets assigned to different model layers and the minimum cost of using this set to run the inference service. DP[[j,gpu_set];c] means that the minimum cost of the GPU set assigned to the jth layer of the large model is c. Set Set the value in DP to positive infinity. The allocated GPU set A is initialized to the empty set. The unallocated GPU set U is initialized to It is a subset of the user-given GPU set G, representing all GPUs on cloud c, with a total of n c GPUs.

[0049] The recursive function Estimate-Cost is used to calculate the cost of reasoning given an allocated GPU set A and an unallocated GPU set U. The dynamic programming algorithm is used to search for the best GPU solution. The execution steps of the recursive function Estimate-Cost in this embodiment include:

[0050] S1, if the jth layer of the large model has not reached the last layer of the model, and the unassigned GPU set U is not empty, then jump to step S2; otherwise, output the GPU set Cur_Set and current cost of each layer recorded in the memory buffer DP, end and exit; the GPU set Cur_Set assigned to each layer is initially an empty set, and the key values ​​are all empty, which is used to subsequently record the current GPU set, record the GPU combination currently assigned to the jth layer and its corresponding cost;

[0051] S2, traverse the unallocated GPU set U and try to add different combinations of each type of GPU in it to the allocated GPU set Cur_Set, and calculate the computing delay and memory consumption of the large model according to the allocated GPU set Cur_Set. If the computing delay of the large model violates the user-defined delay requirement, or the memory consumption of any GPU in the allocated GPU set Cur_Set is violated, then the current cost c′ of the jth layer of the large model is recorded as positive infinity; otherwise, the current cost c′ of the jth layer of the large model is calculated according to the following formula;

[0052] c′=Cost(Cur_Set)+DP[[j-1,A]],

[0053] In the above formula, Cost is a preset cost function, DP[[j-1,A] represents the cost of the j-1th layer of the large model recorded in the memory buffer DP, and the preset cost function is composed of the computational cost and the communication cost;

[0054] S3, determine whether the current cost c′ is less than the known minimum cost recorded in the memory buffer DP, if so, store the allocated GPU set Cur_Set of the jth layer and the current cost c′ in the memory buffer DP;

[0055] S4, add 1 to the number of layers j of the large model, add the allocated GPU set A to the allocated GPU set Cur_Set as the new allocated GPU set A, and subtract the allocated GPU set Cur_Set from the unallocated GPU set U as the new unallocated GPU set U, which can be expressed as:

[0056] A=A+Cur_Set,

[0057] U=U-Cur_Set,

[0058] j=j+1,

[0059] The recursive function Estimate-Cost is recursively called according to the number of layers j of the new large model, the allocated GPU set A, and the unallocated GPU set U, that is, Estimate-Cost(DP, j, A, U). The recursion terminates when all layers of the model are traversed (that is, when j>L, L is the total number of layers of the model to be deployed) or the unallocated GPU set U is empty. Finally, the memoized buffer DP stores the GPU set allocation for each layer and its minimum cost.

[0060] In step S2 of this embodiment, when the computation delay and memory consumption of the large model are respectively calculated according to the allocated GPU set Cur_Set, the function expression of the computation delay is:

[0061]

[0062] In the above formula, represents the computational delay of the large model, Represented as n on cloud c * GPUs for computing large models. express The number of GPUs, express The inference computation time is is n on cloud c * GPUs serving layer j of the large model, express The tensor parallel time is express The pipeline parallel time;

[0063] During the inference calculation process, memory consumption comes from three aspects: storing model parameters; storing intermediate results of neural network layers (including keys and values); and activating caches. A single GPU in For example, when the computation delay and memory consumption of the large model are calculated according to the allocated GPU set Cur_Set in step S2, the function expression for calculating the memory consumption is:

[0064]

[0065] In the above formula, is the memory consumption, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models. Expressed as The number of GPUs, represents the GPU that provides computing services for the jth layer of the large model, l j Indicates the total number of neural network layers served by the GPU currently providing computing services for the jth layer of the large model (the GPU can provide services for one neural network layer or multiple layers), b t Indicates the batch size of large model inference service, The sequence length of the input prompt for the large model, The sequence length of output token generation for large models.

[0066] The calculation function expression of the inference calculation time in this embodiment is:

[0067]

[0068] In the above formula, max means taking the maximum value, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models. The sequence length of output tokens generated for large models, Expressed as The number of GPUs, represents the GPU that provides computing services for the jth layer of the large model, represents the memory bandwidth of the nth GPU on cloud c, l j Indicates the total number of neural network layers currently served by the GPU that currently provides computing services for the jth layer of the large model, b t Indicates the batch size of large model inference service, The sequence length of the input prompt for the large model, represents the tensor core computing capability of the nth GPU on cloud c. for The GPU in.

[0069] The calculation function expression of tensor parallel time in this embodiment is:

[0070]

[0071] In the above formula, max means taking the maximum value. express Except GPU, express and The time delay between the two, b t Indicates the batch size of large model inference service, is the sequence length of the input prompt of the large model, H is the hidden layer dimension of the transformer in the large model, and B typeThe number of bytes used for inference calculation accuracy for large models. Expressed as The number of GPUs, It represents the GPU that provides computing services for the jth layer of the large model. express and The transmission bandwidth between the two, l j Indicates the total number of neural network layers currently served by the GPU that currently provides computing services for the jth layer of the large model. The sequence length of output tokens generated for large models, and Both The GPU in.

[0072] The calculation function expression of the pipeline parallel time in this embodiment is:

[0073]

[0074] In the above formula, min means taking the minimum value. express and The time delay between the two, b t represents the batch size of the large model inference service, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models. express and The transmission bandwidth between the two express Except GPU, The sequence length of output tokens generated for large models, for GPU in, for GPU in, is n on cloud c * GPUs serving layer j of the large model, is n on cloud c * A GPU serving the j+1th layer of the large model.

[0075] set up is a collection of different GPUs on different clouds, including N GPUs on C clouds, for example: Indicates GPU numbered 2 on cloud numbered 1. In this embodiment, some cloud service provider GPUs and related parameters of the model inference service to be deployed are set, as explained below:

[0076] cg : GPU tensor core computing power;

[0077] M g : GPU memory limit;

[0078] m g : GPU memory bandwidth;

[0079] α g,g′ : The latency between the two GPUs g and g′;

[0080] β g,g′ : The transmission bandwidth between the two GPUs g and g′;

[0081] GPU number n on cloud number c is charged based on time.

[0082] The charging standard for data transfer between GPU number n and GPU number n′;

[0083] L: the number of neural network layers of the model to be deployed;

[0084] H: hidden layer dimension of transformer in large model;

[0085] B type : The numerical precision used when performing inference calculations, that is, the number of bytes used for inference calculation precision.

[0086] b t : Inference serving batch size;

[0087] The sequence length of the input prompt;

[0088] Output the sequence length of token generation.

[0089] In this embodiment, is a subset of the G set, representing all GPUs on cloud c, with a total of n c GPU. yes A subset of the set, representing n on cloud c * GPUs serve the jth stage of the model, i.e., the lth stage of the neural network structure of the model to be deployed. j Layer service. If the number of GPUs contained in is greater than 1, then for l j The layer provides tensor parallel services. Let σ be a mapping Indicates that this group of GPUs is l j The layer provides computing services. Assume that there is a set of reasoning tasks T, which follows the distribution Represents the distribution of incoming user reasoning requests. Deploying the large model to be deployed on multiple heterogeneous GPUs provided by cloud service providers using a combination of pipeline parallelism and tensor parallelism can be formally summarized as finding the optimal solution σ * , so that the following formula holds:

[0090]

[0091] st[delay(σ)≤delay max ]∩[C mem (σ)≤M g ]∩[∑(l j )=L],

[0092] In the above formula, Cost refers to the total cost, Cost(σ)=C comp (σ)+C comm (σ), C comp (σ) is the computational cost of using GPU group to perform inference service, C comm (σ) is the communication cost of using GPU group to perform inference service, C mem (σ) is the memory consumption of using GPUs in the GPU group to perform inference services; delay max It refers to the delay requirement of the inference service specified by the user in the json file submitted. Delay(σ) refers to the actual delay of executing the inference service corresponding to the model layer using the GPU group service in the σ mapping, which will be described in detail in subsequent chapters. g Refers to the memory limit of the current GPU. The above formula can be understood as: Under the condition of meeting the latency requirements and not causing the memory of each GPU to overflow, find the best cloud GPU combination for performing the inference task, so that the total cost composed of the computing cost and the communication cost is the lowest. The problem that the method of this embodiment needs to solve can be summarized as: Find a subset to provide computing services for each layer of the large model to execute the arriving reasoning task request, and minimize the total cost of the user, and the delay is lower than the delay requirement set by the user, and the GPU memory used does not exceed the memory limit. Therefore, the function expression of the preset cost function in step S2 of this embodiment is:

[0093]

[0094] In the above formula, express The cost, Represented as n on cloud c * GPUs for computing large models. express The computational cost of express communication costs.

[0095] For the GPU group that provides computing services for the jth stage of the model Generally speaking, the computation time is mainly composed of the following two processes: i. GPU uses high-bandwidth memory to quickly transfer model parameters to dedicated tensor cores; ii. GPU performs matrix multiplication related calculations in the Transformer module. The time to perform inference calculations is:

[0096]

[0097] In the above formula, Refers to the GPU group that provides computing services for the jth stage of the model The number of GPUs is also the dimension of tensor parallelism. The total computational cost of executing a large model inference task is the multiplication of the time-based billing standard of each GPU in the GPU group of each stage by the execution time, and then the sum of the costs incurred by all GPUs used. Therefore, the calculation function expression of the computational cost in this embodiment is:

[0098]

[0099] In the above formula, Represented as n on cloud c * GPUs for computing large models. express The number of GPUs, express The inference computation time is is n on cloud c * GPUs serving layer j of the large model, Represents the price of the nth GPU on cloud c.

[0100] The communication cost is usually charged according to the amount of data transferred. For the GPU group that provides computing services for the jth stage of the model, For example, since the combination of tensor parallelism and pipeline parallelism is used in large-model heterogeneous GPU deployment, the modeling of communication cost is divided into two parts (i. tensor parallel communication cost and ii. pipeline parallel communication cost) for discussion. In this embodiment, the calculation function expression of communication cost is:

[0101]

[0102] In the above formula, express The communication data size of tensor parallel communication, is the data transmission price between the nth GPU and the n′th GPU, It represents the set of GPUs that provide computing services at layer j+1 of the large model. The communication data size for pipelined parallel communication.

[0103] i. Modeling Tensor Parallel Communication Cost

[0104] The communication data size of tensor parallel communication is as follows:

[0105]

[0106] In the above formula, b t Indicates the batch size of large model inference service, is the sequence length of the input prompt of the large model, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models. Expressed as The number of GPUs, represents the GPU that provides computing services for the jth layer of the large model, The length of the sequence generated for the large model output token. The first item on the right is the size of the data that needs to be transmitted for parallel communication of the tensor model in the pre-filling phase, and the second item on the right is the size of the data that needs to be transmitted for parallel communication of the tensor model in the decoding phase. It can be obtained that:

[0107] The calculation function expression of tensor parallel time in this embodiment is:

[0108]

[0109] ii. Pipeline Parallel Communication Cost Modeling

[0110] Pipeline parallel communication mainly occurs in the communication cost between adjacent stages, such as stage j and stage j+1.

[0111] The communication transmission data size is shown as follows:

[0112]

[0113] In the above formula, b t Indicates the batch size of large model inference service, is the sequence length of the input prompt of the large model, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models. Expressed as The number of GPUs, represents the GPU that provides computing services for the jth layer of the large model, The length of the sequence generated for the large model output token. The first item on the right refers to the size of data that needs to be transmitted for the parallel communication of the pipeline model in the pre-filling phase of the inference calculation process, and the second item on the right refers to the size of data that needs to be transmitted for the parallel communication of the pipeline model in the decoding phase. Therefore, the calculation function expression of the pipeline parallel time in this embodiment is:

[0114]

[0115] Communication overhead is usually charged according to the amount of data transmitted, so the total communication cost of the model inference service is modeled as follows:

[0116]

[0117] For each pair of GPUs with data transmission, the data transmission cost is divided into two parts: tensor parallelism and pipeline parallelism. The data transmission cost is summed to obtain the total communication cost of the model inference service.

[0118] like Figure 2 As shown, the cost-aware large model deployment optimization method in the cloud environment of this embodiment is specifically executed by using the deployment optimizer as the execution subject. Figure 2 In this embodiment, the working steps of the deployment optimizer include:

[0119] 1. The user submits the model to be deployed, the model code, and the model inference service code, and specifies the latency requirements of the model inference service and the range of cloud computing power used. The user submits the model weight file, the model-related code in the form of a compressed package, and specifies the latency requirements of the inference service (for example, setting the inference request response time to less than 200 milliseconds) and the available cloud computing resource information (for example, specifying the GPU name, cost, computing power, etc.) in the format of a json file.

[0120] 2. The deployment optimizer performs offline analysis based on the cloud computing power range specified by the user to obtain preliminary computing resource information. The deployment optimizer reads the json file submitted by the user and reads the GPU information of multiple cloud service providers specified by the user, including computing nodes that can provide GPU computing, available GPU models, names, computing capabilities, memory limits, memory bandwidth, and billing standards. The communication consumption between GPUs is then analyzed to obtain the latency, transmission bandwidth, and data transmission billing standards between GPUs. Within the cloud computing power range given by the user, a communication matrix for each GPU of each cloud service provider is generated, which is used to record the latency, transmission bandwidth, and data transmission billing standards between each GPU, so as to facilitate the subsequent calculation of the communication cost of the inference service.

[0121] 3. The deployment optimizer analyzes the model to be deployed and its code to obtain model-related information. The deployment optimizer reads the model weight file and model-related code, analyzes the model neural network hierarchical structure, determines the number of parameters for each layer, the required computing resources, and the data transmission relationship between the neural network layers, so that the subsequent deployment optimizer can divide the model into stages based on the computing node information that can provide GPU computing obtained in the previous step. Each stage contains several neural network layers, and the sum of the number of neural network layers in each stage is the total number of network layers of the model to be deployed. In addition, the deployment optimizer also needs to obtain the number of bytes used for inference calculation accuracy and the inference service batch size.

[0122] 4. The deployment optimizer performs multi-faceted cost evaluation based on the constructed cost model (i.e., the preset cost function Cost).

[0123] 5. The deployment optimizer searches for the optimal large model deployment solution within the cloud computing range specified by the user based on the constructed cost model, latency requirements (computation latency in step S2), and memory consumption, i.e., executes the recursive function Estimate-Cost to search for the best GPU solution. At this point, the deployment optimizer can obtain the optimal large model heterogeneous GPU deployment solution and its estimated cost for each cloud service provider.

[0124] 6. The deployment optimizer compares heterogeneous GPU deployment solutions from different cloud service providers and lists the estimated costs and inference delays of the solutions for users to choose from.

[0125] 7. The user selects a deployment plan, and the deployment optimizer deploys the large model on heterogeneous GPUs according to the selected optimal deployment plan and executes the inference service.

[0126] In summary, the cloud service computing cost of large model reasoning deployment is high, and the GPU computing resources provided by different cloud service providers have certain differences in performance and pricing. Therefore, heterogeneous GPU deployment of large model reasoning services becomes a necessary problem. The cost-aware large model deployment optimization method in the cloud environment of this embodiment proposes a cost-aware large model deployment optimization mechanism in the cloud environment, including obtaining the large model to be deployed, model code, model reasoning service code submitted by the user, and the delay requirements of the specified model reasoning service and the cloud computing power range; obtaining the corresponding computing resource information according to the cloud computing power range specified by the user; analyzing the large model to be deployed and the model code to obtain model information; searching for the large model deployment solution that optimizes the cost of the cost model from the cloud computing power range specified by the user according to the delay requirements of the specified model reasoning service and the preset cost model. The computing resource information and model information are used to perform the cost evaluation required by the preset cost model; the deployment optimizer searches for the large model deployment solution with the best cost from the cloud computing power range specified by the user according to the constructed cost model and delay requirements. The deployment optimizer compares the heterogeneous GPU deployment solutions of different cloud service providers and lists the estimated cost and reasoning delay of the solutions for users to choose. The user selects a deployment plan, and the deployment optimizer deploys the large model on a heterogeneous GPU according to the selected optimal deployment plan and executes the inference service. In addition, this embodiment also constructs a comprehensive cost model for heterogeneous GPU deployment of large models, draws on relevant algorithms such as dynamic programming, and proposes a deployment plan search algorithm. The combination of pipeline parallelism and tensor parallelism can effectively help users select appropriate heterogeneous deployment plans to reduce the computing cost of large model inference services.

[0127] In addition, this embodiment also provides a cost-aware large model deployment optimization system in a cloud environment, including an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the cost-aware large model deployment optimization method in the cloud environment.

[0128] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the cost-aware large model deployment optimization method in the cloud environment through a processor.

[0129] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the cost-aware large model deployment optimization method in the cloud environment through a processor.

[0130] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0131] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A cost-aware large model deployment optimization method in a cloud environment, characterized in that: The method includes initializing a memory buffer DP, and using a recursive function Estimate-Cost based on a given allocated GPU set A and an unallocated GPU set U to implement deployment optimization of a large model by combining pipeline parallelism and tensor parallelism. The parameters that need to be passed into the recursive function Estimate-Cost include the memory buffer DP, the number of layers j of the large model, the allocated GPU set A and the unallocated GPU set U, and the initial value of the number of layers j of the large model is set to 1. The execution steps of the recursive function Estimate-Cost include: S1, if the jth layer of the large model has not reached the last layer of the model and the unallocated GPU set U is not empty, jump to step S2; otherwise, output the allocated GPU set Cur_Set and current cost of each layer recorded in the memory buffer DP, end and exit; S2, traverse the unallocated GPU set U and try to add different combinations of each type of GPU in it to the allocated GPU set Cur_Set, and calculate the computing latency and memory consumption of the large model according to the allocated GPU set Cur_Set. If the computing latency of the large model violates the user-defined latency requirement, or the memory consumption of any GPU in the allocated GPU set Cur_Set is violated, then the current cost c of the jth layer of the large model is set to ′ Recorded as positive infinity; otherwise, the current cost c of the jth layer of the large model is calculated according to the following formula ′ ; c ′ =Cost(Cur_Set)+DP[[j-1,A]], In the above formula, Cost is a preset cost function, DP[[j-1,A] represents the cost of the j-1th layer of the large model recorded in the memory buffer DP, and the preset cost function is composed of the computational cost and the communication cost; S3, determine the current cost c ′ Is it smaller than the known minimum cost recorded in the memory buffer DP? If so, the allocated GPU set Cur_Set of the jth layer and the current cost c ′ Stored in the memory buffer DP; S4, add 1 to the number of layers j of the large model, add the allocated GPU set A to the allocated GPU set Cur_Set as the new allocated GPU set A, subtract the allocated GPU set Cur_Set from the unallocated GPU set U as the new unallocated GPU set U, and recursively call the recursive function Estimate-Cost according to the new number of layers j of the large model, the allocated GPU set A and the unallocated GPU set U.

2. The cost-aware large model deployment optimization method in a cloud environment according to claim 1 is characterized in that: In step S2, when the computation delay and memory consumption of the large model are calculated according to the allocated GPU set Cur_Set, the function expression of the computation delay is: In the above formula, represents the computational delay of the large model, Represented as n on cloud c * GPUs for computing large models. express The number of GPUs, express The inference computation time is is n on cloud c * GPUs serving layer j of the large model, express The tensor parallel time is express The pipeline parallel time is calculated according to the allocated GPU set Cur_Set in step S2. When the computing delay and memory consumption of the large model are calculated respectively, the function expression for calculating the memory consumption is: In the above formula, is the memory consumption, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models. Expressed as The number of GPUs, represents the GPU that provides computing services for the jth layer of the large model, l j Indicates the total number of neural network layers currently served by the GPU that currently provides computing services for the jth layer of the large model, b t Indicates the batch size of large model inference service, The sequence length of the input prompt for the large model, The sequence length of output token generation for large models.

3. The cost-aware large model deployment optimization method in a cloud environment according to claim 2 is characterized in that: The calculation function expression of the inference calculation time is: In the above formula, max means taking the maximum value, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models. The sequence length of output tokens generated for large models, Expressed as The number of GPUs, represents the GPU that provides computing services for the jth layer of the large model. represents the memory bandwidth of the nth GPU on cloud c, l j Indicates the total number of neural network layers currently served by the GPU that currently provides computing services for the jth layer of the large model, b t Indicates the batch size of large model inference service, The sequence length of the input prompt for the large model, represents the tensor core computing capability of the nth GPU on cloud c. for The GPU in.

4. The cost-aware large model deployment optimization method in a cloud environment according to claim 2 is characterized in that: The calculation function expression of the tensor parallel time is: In the above formula, max means taking the maximum value. express Except GPU, express and The time delay between the two, b t Indicates the batch size of large model inference service. is the sequence length of the input prompt of the large model, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models. Expressed as The number of GPUs, It represents the GPU that provides computing services for the jth layer of the large model. express and The transmission bandwidth between the two, l j Indicates the total number of neural network layers currently served by the GPU that currently provides computing services for the jth layer of the large model. The sequence length of output tokens generated for large models, and Both The GPU in.

5. The cost-aware large model deployment optimization method in a cloud environment according to claim 2 is characterized in that: The calculation function expression of the pipeline parallel time is: In the above formula, min means taking the minimum value. express and The time delay between the two, b t represents the batch size of the large model inference service, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models. express and The transmission bandwidth between the two express Except GPU, The sequence length of output tokens generated for large models, for GPU in, for GPU in, is n on cloud c * GPUs serving layer j of the large model, is n on cloud c * A GPU serving the j+1th layer of the large model.

6. The cost-aware large model deployment optimization method in a cloud environment according to claim 1 is characterized in that: The function expression of the cost function preset in step S2 is: In the above formula, express The cost, Represented as n on cloud c * GPUs for computing large models. express The computational cost of express communication costs.

7. The cost-aware large model deployment optimization method in a cloud environment according to claim 6 is characterized in that: The calculation function expression of the calculation cost is: In the above formula, Represented as n on cloud c * GPUs for computing large models. express The number of GPUs, express The inference computation time is is n on cloud c * GPUs serving layer j of the large model, represents the price of the nth GPU on cloud c; The calculation function expression of the communication cost is: In the above formula, express The communication data size of tensor parallel communication, is the data transmission price between the nth GPU and the n′th GPU, It represents the set of GPUs that provide computing services at layer j+1 of the large model. is the communication data size of pipeline parallel communication, and: In the above formula, b t Indicates the batch size of large model inference service. is the sequence length of the input prompt of the large model, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models. Expressed as The number of GPUs, represents the GPU that provides computing services for the jth layer of the large model. The sequence length of output token generation for large models.

8. A cost-aware large model deployment optimization system in a cloud environment, comprising interconnected microprocessors and memories, characterized in that: The microprocessor is programmed or configured to execute the cost-aware large model deployment optimization method in a cloud environment as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute, through a processor, the cost-aware large model deployment optimization method in a cloud environment as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute, through a processor, the cost-aware large model deployment optimization method in a cloud environment as described in any one of claims 1 to 7.

Images