A cost-aware large model deployment optimization method and system in cloud environments

Abstract

The present invention discloses a cost-aware large-model deployment optimization method and system in an inter-cloud environment. The method includes initializing a memory cache DP and, based on a given set of allocated GPUs A and a set of unallocated GPUs U, utilizing a recursive function, Estimate-Cost, to optimize the deployment of the large model using a combination of pipeline parallelism and tensor parallelism. The recursive function, Estimate-Cost, requires as input parameters the memory cache DP, the number of layers j of the large model, the allocated GPU set A, and the unallocated GPU set U. The present invention aims to establish a cross-cloud collaboration mechanism for inter-cloud computing environments that rationally partitions multiple neural network layers in a large model for heterogeneous GPU deployment, thereby reducing the computational cost of large-model inference. This mechanism effectively helps users select appropriate heterogeneous deployment solutions and reduces the computational cost of large-model inference services.

Description

Technical Field

[0001] The present invention relates to a large model deployment collaboration technology in a cloud computing environment, and in particular to a cost-aware large model deployment optimization method and system in a cloud environment. Background Art

[0002] Cloud computing is a network-based computing model. Its core concept is to provide computing resources (including computing resources, GPUs, storage, and network resources) to users as services through virtualization technology. Users can access and release these resources as needed. Joint Cloud Computing is based on open collaboration between multiple cloud service providers. Through the deep integration of multi-party cloud resources, it facilitates developers to customize cloud services through a "software-defined" approach. Developers can use resources from multiple cloud vendors to create value beyond the cloud itself, realizing a new generation of cloud computing model with "borderless services, collaboration between clouds, easy resource sharing, and convertible value." Joint Cloud Computing is one of the important ways to address the current cost-effectiveness problem caused by high computing power demand.

[0003] The development of large models has attracted considerable attention in the field of machine learning. With the increase in computing power and the growth of datasets, the training and deployment of large models has become more feasible. Model scaling has been proven to be one of the most important factors in achieving higher accuracy. Its effectiveness has been consistently demonstrated in the practice of large-scale models across all industrial fields, such as the Vision Model BiT, the Natural Language Processing Model BERT, GPT3, and the Deep Learning Recommendation Model DLRM. The development of large models poses a significant computing challenge, as these models often contain billions or even tens of billions of parameters. Training these large models and deploying them to provide inference services requires significant computing resources and storage space. Therefore, such large models require a large amount of computing power resources to work together to provide stable external services.

[0004] When it comes to model inference, large models are typically deployed as online services to respond to user requests. Large model inference services typically have high expectations for response latency and inference accuracy. Large model applications that fail to complete within the specified timeframe or whose accuracy falls short of expectations have little commercial value. Therefore, an effective scheduler that balances inference latency, accuracy, and cost is crucial. In actual inference service scenarios, the volume of user requests for inference services is enormous. Without high-performance computing power, it is impossible to provide services with low latency and high accuracy. Therefore, state-of-the-art model inference services are typically hosted in centralized data centers equipped with homogeneous high-performance GPUs, which can be very expensive in terms of cloud service fees. The high cost of centralized, homogeneous deployments within a single cloud can limit the scalability of large model inference services.

[0005] However, different cloud service providers offer varying computing power and pricing for GPU computing resources. Global service coverage and data transfer fees vary across cloud service providers. By combining different types of computing resources within the cloud, flexibly selecting and dynamically adjusting resources, leveraging price competition, optimizing geographic location and data transfer, and improving resource utilization, we can effectively reduce the overall cost of large-model inference services. Deploying large-model inference across heterogeneous cloud environments may be a viable approach to reducing inference costs in terms of cloud service fees. Large-model serving systems should strive to meet service-level objectives while minimizing the cost of providing services within the cloud.

[0006] Therefore, there is an urgent need to design a deployment optimization mechanism that can consider the computing power, price, and communication transmission of each cloud service provider and deploy large models across clouds in an inter-cloud environment to achieve the best cost-effectiveness. Summary of the Invention

[0007] Technical problem to be solved by the present invention: In response to the above-mentioned problems of the prior art, a cost-aware large model deployment optimization method and system in a cloud environment are provided. The present invention aims to reasonably divide multiple neural network layers in a large model in a cloud computing environment for heterogeneous GPU deployment, thereby reducing the computing cost of large model inference. The cross-cloud collaboration mechanism can effectively help users choose appropriate heterogeneous deployment solutions to reduce the computing cost of large model inference services.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0009] A cost-aware large model deployment optimization method in an inter-cloud environment includes initializing a memory cache DP and, based on a given allocated GPU set A and an unallocated GPU set U, using a recursive function Estimate-Cost to implement large model deployment optimization using a combination of pipeline parallelism and tensor parallelism. Parameters required to be input into the recursive function Estimate-Cost include the memory cache DP, the number of layers j of the large model, the allocated GPU set A, and the unallocated GPU set U. The initial value of the number of layers j of the large model is set to 1. The execution steps of the recursive function Estimate-Cost include:

[0010] S1: If the jth layer of the large model has not yet reached the last layer of the model and the unassigned GPU set U is not empty, jump to step S2; otherwise, output the allocated GPU set Cur_Set and current cost of each layer recorded in the memory buffer DP, end and exit;

[0011] S2, traverse the unassigned GPU set U and try to add different combinations of each type of GPU in it to the assigned GPU set Cur_Set, and calculate the computing latency and memory consumption of the large model according to the assigned GPU set Cur_Set. If the computing latency of the large model violates the user-defined latency requirement, or the memory consumption of any GPU in the assigned GPU set Cur_Set is violated, then the current cost c' of the j-th layer of the large model is recorded as positive infinity; otherwise, the current cost c' of the j-th layer of the large model is calculated according to the following formula;

[0012] c′=Cost(Cur_Set)+DP[[j-1,A]],

[0013] In the above formula, Cost is a preset cost function, DP[[j-1,A] represents the cost of the j-1th layer of the large model recorded in the memory buffer DP, and the preset cost function is composed of computational cost and communication cost;

[0014] S3, determine whether the current cost c′ is less than the known minimum cost recorded in the memory buffer DP. If so, store the allocated GPU set Cur_Set of the jth layer and the current cost c′ in the memory buffer DP;

[0015] S4, add 1 to the number of layers j of the large model, add the allocated GPU set A to the allocated GPU set Cur_Set as the new allocated GPU set A, subtract the allocated GPU set Cur_Set from the unallocated GPU set U as the new unallocated GPU set U, and recursively call the recursive function Estimate-Cost according to the new number of layers j of the large model, the allocated GPU set A and the unallocated GPU set U.

[0016] Optionally, when the computational latency and memory consumption of the large model are calculated respectively according to the allocated GPU set Cur_Set in step S2, the function expression of the computational latency is:

[0017]

[0018] In the above formula, represents the computational delay of the large model, Represented as n on cloud c * GPUs that provide computing services for large models, express The number of GPUs, express The inference computation time, is n on cloud c * GPUs serving layer j of the large model, express The tensor parallel time, express The pipeline parallel time is calculated; in step S2, when the computational delay and memory consumption of the large model are calculated according to the allocated GPU set Cur_Set, the function expression for calculating the memory consumption is:

[0019]

[0020] In the above formula, is the memory consumption, H is the hidden layer dimension of transformer in the large model, B type The number of bytes used for inference calculation accuracy for large models, Expressed as The number of GPUs, represents the GPU that provides computing services for the jth layer of the large model, l j Indicates the total number of neural network layers currently served by the GPU providing computing services for the jth layer of the large model, b t Indicates the batch size of large model inference service, The sequence length of the input prompt for the large model, The sequence length of output tokens generated for large models.

[0021] Optionally, the calculation function expression of the inference calculation time is:

[0022]

[0023] In the above formula, max represents the maximum value, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models, The length of the sequence generated for the large model output tokens, Expressed as The number of GPUs, Represents the GPU that provides computing services for the jth layer of the large model, represents the memory bandwidth of the nth GPU on cloud c, l j Indicates the total number of neural network layers currently served by the GPU providing computing services for the jth layer of the large model, b t Indicates the batch size of large model inference service, The sequence length of the input prompt for the large model, Indicates the tensor core computing capability of the nth GPU on cloud c. for The GPU in.

[0024] Optionally, the calculation function expression of the tensor parallel time is:

[0025]

[0026] In the above formula, max means taking the maximum value. express Except GPU, express and The time delay between the two, b t Indicates the batch size of large model inference service, is the sequence length of the input prompt for the large model, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models, Expressed as The number of GPUs, Indicates the GPU that provides computing services for the jth layer of the large model. express and The transmission bandwidth between the two, l j Indicates the total number of neural network layers currently served by the GPU providing computing services for the jth layer of the large model. The length of the sequence generated for the large model output tokens, and Both The GPU in.

[0027] Optionally, the calculation function expression of the pipeline parallel time is:

[0028]

[0029] In the above formula, min means taking the minimum value. express and The time delay between the two, b t represents the batch size of the large model inference service, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models, express and The transmission bandwidth between the two express Except GPU, The length of the sequence generated for the large model output tokens, for GPU in, for GPU in, is n on cloud c * GPUs serving layer j of the large model, is n on cloud c * A GPU serving the j+1th layer of the large model.

[0030] Optionally, the function expression of the cost function preset in step S2 is:

[0031]

[0032] In the above formula, express the cost, Represented as n on cloud c * GPUs that provide computing services for large models, express The computational cost, express communication costs.

[0033] Optionally, the calculation function expression of the calculation cost is:

[0034]

[0035] In the above formula, Represented as n on cloud c * GPUs that provide computing services for large models, express The number of GPUs, express The inference computation time, is n on cloud c *GPUs serving layer j of the large model, Indicates the price of the nth GPU on cloud c;

[0036] The calculation function expression of the communication cost is:

[0037]

[0038] In the above formula, express The communication data size of tensor parallel communication, is the data transmission price between the nth GPU and the n′th GPU, Represents the set of GPUs that provide computing services for the j+1th layer of the large model. is the communication data size of pipeline parallel communication, and:

[0039]

[0040] In the above formula, b t Indicates the batch size of large model inference service, is the sequence length of the input prompt for the large model, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models, Expressed as The number of GPUs, Represents the GPU that provides computing services for the jth layer of the large model, The sequence length of output tokens generated for large models.

[0041] In addition, the present invention also provides a cost-aware large model deployment optimization system in an inter-cloud environment, including an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the cost-aware large model deployment optimization method in the inter-cloud environment.

[0042] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the cost-aware large model deployment optimization method in the cloud environment through a processor.

[0043] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the cost-aware large model deployment optimization method in an inter-cloud environment through a processor.

[0044] Compared with the existing technology, the present invention has the following main advantages: the cloud service computing cost of large model reasoning deployment is high, and the GPU computing resources provided by different cloud service providers have certain differences in performance and pricing. Therefore, heterogeneous GPU deployment of large model reasoning services becomes a necessary issue. The cost-aware large model deployment optimization method in the cloud environment of the present invention proposes a cost-aware large model deployment optimization mechanism in the cloud environment, and constructs a comprehensive cost model for large model heterogeneous GPU deployment. Drawing on relevant algorithms such as dynamic programming, a deployment scheme search algorithm is proposed. The method combines pipeline parallelism and tensor parallelism, which can effectively help users select appropriate heterogeneous deployment schemes to reduce the computing cost of large model reasoning services. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagram of the basic process of the method of the embodiment of the present invention.

[0046] Figure 2 Schematic diagram of the application process of the method of the embodiment of the present invention. DETAILED DESCRIPTION

[0047] like Figure 1 As shown, the cost-aware large model deployment optimization method in the cloud environment of this embodiment includes initializing the memory cache DP, and using the recursive function Estimate-Cost based on the given allocated GPU set A and the unallocated GPU set U to achieve deployment optimization of the large model using a combination of pipeline parallelism and tensor parallelism. The parameters that need to be passed in to the recursive function Estimate-Cost include the memory cache DP, the number of layers j of the large model, the allocated GPU set A and the unallocated GPU set U, and the initial value of the number of layers j of the large model is set to 1, that is: Estimate-Cost(DP,j=1,A,U).

[0048] The memory buffer DP is used to record the GPU sets assigned to different model layers and the minimum cost of using this set to run the inference service. DP[[j,gpu_set];c] means that the minimum cost of the GPU set assigned to the jth layer of the large model is c. Set during initialization Set Set the value in DP to be initialized to positive infinity. The allocated GPU set A is initialized to the empty set. The unallocated GPU set U is initialized to It is a subset of the user-given GPU set G, representing all GPUs on cloud c, with a total of n c GPUs.

[0049] The recursive function Estimate-Cost is used to calculate the cost of inference given an assigned GPU set A and an unassigned GPU set U. It uses a dynamic programming algorithm to search for the optimal GPU solution. In this embodiment, the recursive function Estimate-Cost executes the following steps:

[0050] S1: If the jth layer of the large model has not yet reached the last layer of the model and the unassigned GPU set U is not empty, jump to step S2; otherwise, output the allocated GPU set Cur_Set and current cost of each layer recorded in the memory buffer DP, end and exit; the allocated GPU set Cur_Set of each layer is initially an empty set with empty key values, which is used to subsequently record the current GPU set, the GPU combination currently assigned to the jth layer and its corresponding cost;

[0051] S2, traverse the unassigned GPU set U and try to add different combinations of each type of GPU in it to the assigned GPU set Cur_Set, and calculate the computing latency and memory consumption of the large model according to the assigned GPU set Cur_Set. If the computing latency of the large model violates the user-defined latency requirement, or the memory consumption of any GPU in the assigned GPU set Cur_Set is violated, then the current cost c' of the j-th layer of the large model is recorded as positive infinity; otherwise, the current cost c' of the j-th layer of the large model is calculated according to the following formula;

[0052] c′=Cost(Cur_Set)+DP[[j-1,A]],

[0053] In the above formula, Cost is a preset cost function, DP[[j-1,A] represents the cost of the j-1th layer of the large model recorded in the memory buffer DP, and the preset cost function is composed of computational cost and communication cost;

[0054] S3, determine whether the current cost c′ is less than the known minimum cost recorded in the memory buffer DP. If so, store the allocated GPU set Cur_Set of the jth layer and the current cost c′ in the memory buffer DP;

[0055] In S4, the number of layers j of the large model is increased by 1, the allocated GPU set A is added to the allocated GPU set Cur_Set as the new allocated GPU set A, and the unallocated GPU set U is subtracted from the allocated GPU set Cur_Set as the new unallocated GPU set U, which can be expressed as:

[0056] A=A+Cur_Set,

[0057] U=U-Cur_Set,

[0058] j=j+1,

[0059] The recursive function Estimate-Cost is recursively called based on the number of layers j of the new large model, the allocated GPU set A, and the unallocated GPU set U: Estimate-Cost(DP, j, A, U). The recursion terminates when all layers of the model have been traversed (i.e., when j > L, where L is the total number of layers in the model to be deployed) or when the unallocated GPU set U is empty. Ultimately, the memoized buffer DP stores the GPU set allocation for each layer and its minimum cost.

[0060] In step S2 of this embodiment, when the computational delay and memory consumption of the large model are respectively calculated according to the allocated GPU set Cur_Set, the function expression of the computational delay is:

[0061]

[0062] In the above formula, represents the computational delay of the large model, Represented as n on cloud c * GPUs that provide computing services for large models, express The number of GPUs, express The inference computation time, is n on cloud c * GPUs serving layer j of the large model, express The tensor parallel time, express The pipeline parallel time;

[0063] During the inference calculation process, memory consumption comes from three aspects: storing model parameters; storing intermediate results of neural network layers (including keys and values); and activating caches. A single GPU in For example, when calculating the computational latency and memory consumption of the large model according to the allocated GPU set Cur_Set in step S2, the function expression for calculating the memory consumption is:

[0064]

[0065] In the above formula, is the memory consumption, H is the hidden layer dimension of transformer in the large model, B type The number of bytes used for inference calculation accuracy for large models, Expressed as The number of GPUs, represents the GPU that provides computing services for the jth layer of the large model, l j Indicates the total number of neural network layers currently served by the GPU that currently provides computing services for the jth layer of the large model (a GPU can provide services for one neural network layer or multiple layers). t Indicates the batch size of large model inference service, The sequence length of the input prompt for the large model, The sequence length of output tokens generated for large models.

[0066] The calculation function expression of the inference calculation time in this embodiment is:

[0067]

[0068] In the above formula, max represents the maximum value, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models, The length of the sequence generated for the large model output tokens, Expressed as The number of GPUs, Represents the GPU that provides computing services for the jth layer of the large model, represents the memory bandwidth of the nth GPU on cloud c, l j Indicates the total number of neural network layers currently served by the GPU providing computing services for the jth layer of the large model, b t Indicates the batch size of large model inference service, The sequence length of the input prompt for the large model, Indicates the tensor core computing capability of the nth GPU on cloud c. for The GPU in.

[0069] The calculation function expression of tensor parallel time in this embodiment is:

[0070]

[0071] In the above formula, max means taking the maximum value. express Except GPU, express and The time delay between the two, b t Indicates the batch size of large model inference service, is the sequence length of the input prompt for the large model, H is the hidden layer dimension of the transformer in the large model, and B typeThe number of bytes used for inference calculation accuracy for large models, Expressed as The number of GPUs, Indicates the GPU that provides computing services for the jth layer of the large model. express and The transmission bandwidth between the two, l j Indicates the total number of neural network layers currently served by the GPU providing computing services for the jth layer of the large model. The length of the sequence generated for the large model output tokens, and Both The GPU in.

[0072] The calculation function expression of the pipeline parallel time in this embodiment is:

[0073]

[0074] In the above formula, min means taking the minimum value. express and The time delay between the two, b t represents the batch size of the large model inference service, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models, express and The transmission bandwidth between the two express Except GPU, The length of the sequence generated for the large model output tokens, for GPU in, for GPU in, is n on cloud c * GPUs serving layer j of the large model, is n on cloud c * A GPU serving the j+1th layer of the large model.

[0075] set up is a collection of different GPUs on different clouds, including N GPUs on a total of C clouds, for example: Indicates GPU number 2 on cloud number 1. In this embodiment, some cloud service provider GPUs and parameters related to the model inference service to be deployed are set, as explained below:

[0076] cg : GPU tensor core computing power;

[0077] M g : GPU memory limit;

[0078] m g : GPU memory bandwidth;

[0079] α g,g′ : the latency between the two GPUs g and g′;

[0080] β g,g′ : the transmission bandwidth between the two GPUs g and g′;

[0081] GPU number n on cloud number c is billed based on time.

[0082] Billing standards for data transfer between GPU n and GPU n′;

[0083] L: the number of neural network layers of the model to be deployed;

[0084] H: hidden layer dimension of transformer in large model;

[0085] B type : The numerical precision used when performing inference calculations, that is, the number of bytes used for inference calculation precision.

[0086] b t : Inference service batch size;

[0087] The sequence length of the input prompt;

[0088] Output the sequence length of token generation.

[0089] In this embodiment, It is a subset of the G set, representing all GPUs on cloud c, with a total of n c GPU. yes A subset of the set, representing n on cloud c * GPUs serve the jth stage of the model, i.e., the lth stage of the neural network structure of the model to be deployed j If If the number of GPUs contained in is greater than 1, then for l j The layer provides tensor parallel services. Let σ be a mapping Indicates that this group of GPUs is l j The layer provides computing services. Assume there is a set of reasoning tasks T, which obey the distribution Represents the distribution of incoming user inference requests. Deploying the large model to be deployed on multiple heterogeneous GPUs provided by cloud service providers using a combination of pipeline parallelism and tensor parallelism can be formally summarized as finding the optimal solution σ * , so that the following formula holds:

[0090]

[0091] st[delay(σ)≤delay max ]∩[C mem (σ)≤M g ]∩[∑(l j )=L],

[0092] In the above formula, Cost refers to the total cost, Cost(σ)=C comp (σ)+C comm (σ), C comp (σ) is the computational cost of using GPU group to perform inference service, C comm (σ) is the communication cost of using GPU group to perform inference service, C mem (σ) is the memory consumption of using GPUs in the GPU group to perform inference services; delay max It refers to the delay requirement of the inference service specified by the user in the JSON file submitted. Delay(σ) refers to the actual delay of executing the inference service corresponding to the model layer using the GPU group service in the σ mapping, which will be described in detail in subsequent chapters. g Refers to the memory limit of the current GPU. The above formula can be understood as: Under the condition of meeting the latency requirements and not causing the memory overflow of each GPU, find the best cloud GPU combination to perform the inference task, so that the total cost composed of the combination of computing cost and communication cost is the lowest. The problem that the method of this embodiment needs to solve can be summarized as: Find a subset to provide computing services for each layer of the large model to execute the incoming inference task request, while minimizing the total cost of the user, and the delay is lower than the delay requirement set by the user, and the GPU memory used does not exceed the memory limit. Therefore, the function expression of the cost function preset in step S2 of this embodiment is:

[0093]

[0094] In the above formula, express the cost, Represented as n on cloud c * GPUs that provide computing services for large models, express The computational cost, express communication costs.

[0095] For the GPU group that provides computing services for the jth stage of the model Generally speaking, the computation time is mainly composed of the following two processes: i. The GPU uses high-bandwidth memory to quickly transfer model parameters to dedicated tensor cores; ii. The GPU performs matrix multiplication related calculations in the Transformer module. The time to perform inference calculations is:

[0096]

[0097] In the above formula, Refers to the GPU group that provides computing services for the jth stage of the model The number of GPUs is also the dimension of tensor parallelism. The total computational cost of executing a large model inference task is the multiplication of the time-based billing standard of each GPU in the GPU group of each stage by the execution time, and then the sum of the costs incurred by all used GPUs. Therefore, the calculation function expression of the computational cost in this embodiment is:

[0098]

[0099] In the above formula, Represented as n on cloud c * GPUs that provide computing services for large models, express The number of GPUs, express The inference computation time, is n on cloud c * GPUs serving layer j of the large model, Indicates the price of the nth GPU on cloud c.

[0100] Communication costs are usually charged according to the amount of data transmitted. For the GPU group that provides computing services for the jth stage of the model, For example, since a combination of tensor parallelism and pipeline parallelism is used when deploying large models on heterogeneous GPUs, the modeling of communication costs is divided into two parts (i. tensor parallel communication cost and ii. pipeline parallel communication cost). In this embodiment, the calculation function expression of the communication cost is:

[0101]

[0102] In the above formula, express The communication data size of tensor parallel communication, is the data transmission price between the nth GPU and the n′th GPU, Represents the set of GPUs that provide computing services for the j+1th layer of the large model. The communication data size for pipelined parallel communication.

[0103] i. Modeling Tensor Parallel Communication Cost

[0104] The communication data size of tensor parallel communication is as follows:

[0105]

[0106] In the above formula, b t Indicates the batch size of large model inference service, is the sequence length of the input prompt for the large model, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models, Expressed as The number of GPUs, Represents the GPU that provides computing services for the jth layer of the large model, The length of the sequence generated for the large model output token. The first item on the right is the size of the data that needs to be transmitted for parallel communication of the tensor model in the pre-filling phase, and the second item on the right is the size of the data that needs to be transmitted for parallel communication of the tensor model in the decoding phase. From this, we can get:

[0107] The calculation function expression of tensor parallel time in this embodiment is:

[0108]

[0109] ii. Pipeline Parallel Communication Cost Modeling

[0110] Pipeline parallel communication mainly occurs in the communication cost between adjacent stages, such as stage j and stage j+1.

[0111] The communication transmission data size is shown in the following formula:

[0112]

[0113] In the above formula, b t Indicates the batch size of large model inference service, is the sequence length of the input prompt for the large model, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models, Expressed as The number of GPUs, Represents the GPU that provides computing services for the jth layer of the large model, The length of the sequence generated for the large model output token. The first term on the right refers to the data size required for parallel communication of the pipeline model during the pre-filling phase of the inference calculation process, and the second term on the right refers to the data size required for parallel communication of the pipeline model during the decoding phase. Therefore, the calculation function expression for the pipeline parallel time in this embodiment is:

[0114]

[0115] Communication overhead is usually charged based on the amount of data transmitted. Therefore, the total communication cost of the model inference service is modeled as follows:

[0116]

[0117] For each pair of GPUs with data transmission, the data transmission cost is divided into two parts: tensor parallelism and pipeline parallelism. The data transmission cost is summed to obtain the total communication cost of the model inference service.

[0118] like Figure 2 As shown in FIG, the cost-aware large model deployment optimization method in the cloud environment of this embodiment is specifically executed by using the deployment optimizer as the execution subject. Figure 2 In this embodiment, the working steps of the deployment optimizer include:

[0119] 1. The user submits the model to be deployed, the model code, and the model inference service code, and specifies the latency requirements for the model inference service and the range of cloud computing power used. The user submits the model weight file and the model-related code in a compressed package, and specifies the latency requirements for the inference service (for example, setting the inference request response time to less than 200 milliseconds) and available cloud computing resource information (for example, specifying the GPU name, cost, and computing power) in the format of a JSON file.

[0120] 2. The deployment optimizer performs offline analysis based on the cloud computing power range specified by the user to obtain preliminary computing resource information. The deployment optimizer reads the json file submitted by the user and reads the GPU information of multiple cloud service providers specified by the user, including computing nodes that can provide GPU computing, available GPU models, names, computing power, memory limits, memory bandwidth, and billing standards. It then analyzes the communication consumption between GPUs to obtain the latency, transmission bandwidth, and data transmission billing standards between GPUs. Within the cloud computing power range given by the user, a communication matrix for each GPU of each cloud service provider is generated to record the latency, transmission bandwidth, and data transmission billing standards between each GPU, which facilitates the subsequent calculation of the communication cost of the inference service.

[0121] 3. The deployment optimizer analyzes the model to be deployed and its code to obtain model-related information. The deployment optimizer reads the model weight file and model-related code, analyzes the model's neural network hierarchy, and determines the number of parameters for each layer, the required computing resources, and the data transmission relationship between neural network layers. This allows the deployment optimizer to subsequently divide the model into stages based on the compute node information available for GPU computing obtained in the previous step. Each stage contains several neural network layers, and the total number of neural network layers in each stage is the total number of network layers in the model to be deployed. In addition, the deployment optimizer also needs to obtain the number of bytes used for inference calculation accuracy and the inference service batch size.

[0122] 4. The deployment optimizer performs multi-faceted cost evaluation based on the constructed cost model (i.e., the preset cost function Cost).

[0123] 5. Based on the constructed cost model, latency requirements (computation latency in step S2), and memory consumption, the deployment optimizer searches for the most cost-effective large-model deployment solution within the user-specified cloud computing capacity. This involves executing the recursive function Estimate-Cost to search for the optimal GPU solution. At this point, the deployment optimizer obtains the optimal large-model heterogeneous GPU deployment solution and its estimated cost for each cloud provider.

[0124] 6. The deployment optimizer compares heterogeneous GPU deployment solutions from different cloud service providers and lists the estimated costs and inference delays of the solutions for users to choose from.

[0125] 7. The user selects a deployment plan, and the deployment optimizer deploys the large model on heterogeneous GPUs based on the selected optimal deployment plan and executes the inference service.

[0126] In summary, the high computing costs of cloud services for large model inference deployment, coupled with performance and pricing differences among GPU computing resources provided by different cloud service providers, make heterogeneous GPU deployment for large model inference services essential. This embodiment proposes a cost-aware large model deployment optimization method in a cloud-based environment. The method includes obtaining a user-submitted large model to be deployed, model code, model inference service code, as well as the specified latency requirements of the model inference service and a cloud computing power range. The method then obtains corresponding computing resource information based on the user-specified cloud computing power range. The method then analyzes the large model to be deployed and the model code to obtain model information. Based on the specified latency requirements of the model inference service and a preset cost model, the method searches for a large model deployment solution within the user-specified cloud computing power range that optimizes the cost model. The method then uses the computing resource information and model information to perform a cost assessment for the preset cost model. A deployment optimizer searches for the optimal large model deployment solution within the user-specified cloud computing power range based on the constructed cost model and latency requirements. The deployment optimizer compares heterogeneous GPU deployment solutions from different cloud service providers and lists the estimated costs and inference latency of each solution for the user to choose from. The user selects a deployment plan, and the deployment optimizer deploys the large model to heterogeneous GPUs based on the selected optimal deployment plan to execute inference services. Furthermore, this embodiment constructs a comprehensive cost model for heterogeneous GPU deployment of large models. Drawing on algorithms such as dynamic programming, a deployment plan search algorithm is proposed. This algorithm, which combines pipeline parallelism with tensor parallelism, effectively helps users select appropriate heterogeneous deployment plans, thereby reducing the computational cost of large-model inference services.

[0127] In addition, this embodiment also provides a cost-aware large model deployment optimization system in an inter-cloud environment, including an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the cost-aware large model deployment optimization method in the inter-cloud environment.

[0128] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the cost-aware large model deployment optimization method in the cloud environment through a processor.

[0129] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the cost-aware large model deployment optimization method in the cloud environment through a processor.

[0130] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0131] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A cost-aware large model deployment optimization method in a cloud environment, characterized by: The method includes initializing a memory buffer DP, and using a recursive function Estimate-Cost based on a given allocated GPU set A and an unallocated GPU set U to implement deployment optimization of a large model using a combination of pipeline parallelism and tensor parallelism. The parameters required to be input into the recursive function Estimate-Cost include the memory buffer DP, the number of layers j of the large model, the allocated GPU set A, and the unallocated GPU set U. The initial value of the number of layers j of the large model is set to 1. The execution steps of the recursive function Estimate-Cost include: S1: If the jth layer of the large model has not yet reached the last layer of the model and the unassigned GPU set U is not empty, jump to step S2; otherwise, output the allocated GPU set Cur_Set and current cost of each layer recorded in the memory buffer DP, end and exit; S2, traverse the unassigned GPU set U and try to add different combinations of each type of GPU to the assigned GPU set Cur_Set, and calculate the computational latency and memory consumption of the large model according to the assigned GPU set Cur_Set. If the computational latency of the large model violates the user-defined latency requirement, or the memory consumption of any GPU in the assigned GPU set Cur_Set is violated, then the current cost c of the jth layer of the large model is set to ′ Recorded as positive infinity; otherwise, the current cost c of the jth layer of the large model is calculated according to the following formula ′ ; c ′ =Cost(Cur_Set)+DP[[j-1,A]], In the above formula, Cost is a preset cost function, DP[[j-1,A] represents the cost of the j-1th layer of the large model recorded in the memory buffer DP, and the preset cost function is composed of computational cost and communication cost; S3, determine the current cost c ′ Is it less than the known minimum cost recorded in the memory buffer DP? If so, the allocated GPU set Cur_Set of the jth layer and the current cost c ′ Stored in the memory buffer DP; S4, add 1 to the number of layers j of the large model, add the allocated GPU set A to the allocated GPU set Cur_Set as the new allocated GPU set A, subtract the allocated GPU set Cur_Set from the unallocated GPU set U as the new unallocated GPU set U, and recursively call the recursive function Estimate-Cost according to the new number of layers j of the large model, the allocated GPU set A and the unallocated GPU set U.

2. The cost-aware large model deployment optimization method in a cloud environment according to claim 1 is characterized in that: In step S2, when the computational latency and memory consumption of the large model are calculated based on the allocated GPU set Cur_Set, the function expression of the computational latency is: In the above formula, represents the computational delay of the large model, Represented as n on cloud c * GPUs that provide computing services for large models, express The number of GPUs, express The inference computation time, is n on cloud c * GPUs serving layer j of the large model, express The tensor parallel time, express The pipeline parallel time is calculated; in step S2, when the computational delay and memory consumption of the large model are calculated according to the allocated GPU set Cur_Set, the function expression for calculating the memory consumption is: In the above formula, is the memory consumption, H is the hidden layer dimension of transformer in the large model, B type The number of bytes used for inference calculation accuracy for large models, Expressed as The number of GPUs, represents the GPU that provides computing services for the jth layer of the large model, l j Indicates the total number of neural network layers currently served by the GPU providing computing services for the jth layer of the large model, b t Indicates the batch size of large model inference service, The sequence length of the input prompt for the large model, The sequence length of output tokens generated for large models.

3. The cost-aware large model deployment optimization method in a cloud environment according to claim 2 is characterized in that: The calculation function expression of the inference calculation time is: In the above formula, max represents the maximum value, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models, The length of the sequence generated for the large model output tokens, Expressed as The number of GPUs, Represents the GPU that provides computing services for the jth layer of the large model, represents the memory bandwidth of the nth GPU on cloud c, l j Indicates the total number of neural network layers currently served by the GPU providing computing services for the jth layer of the large model, b t Indicates the batch size of large model inference service, The sequence length of the input prompt for the large model, Indicates the tensor core computing capability of the nth GPU on cloud c. for The GPU in.

4. The cost-aware large model deployment optimization method in a cloud environment according to claim 2 is characterized in that: The calculation function expression of the tensor parallel time is: In the above formula, max means taking the maximum value. express Except GPU, express and The time delay between the two, b t Indicates the batch size of large model inference service, is the sequence length of the input prompt for the large model, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models, Expressed as The number of GPUs, Indicates the GPU that provides computing services for the jth layer of the large model. express and The transmission bandwidth between the two, l j Indicates the total number of neural network layers currently served by the GPU providing computing services for the jth layer of the large model. The length of the sequence generated for the large model output tokens, and Both The GPU in.

5. The cost-aware large model deployment optimization method in a cloud environment according to claim 2 is characterized in that: The calculation function expression of the pipeline parallel time is: In the above formula, min means taking the minimum value. express and The time delay between the two, b t represents the batch size of the large model inference service, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models, express and The transmission bandwidth between the two express Except GPU, The length of the sequence generated for the large model output tokens, for GPU in, for GPU in, is n on cloud c * GPUs serving layer j of the large model, is n on cloud c * A GPU serving the j+1th layer of the large model.

6. The cost-aware large model deployment optimization method in a cloud environment according to claim 1 is characterized in that: The function expression of the cost function preset in step S2 is: In the above formula, express The cost, Represented as n on cloud c * GPUs that provide computing services for large models, express The computational cost of express communication costs.

7. The cost-aware large model deployment optimization method in a cloud environment according to claim 6 is characterized in that: The calculation function expression of the calculation cost is: In the above formula, Represented as n on cloud c * GPUs that provide computing services for large models, express The number of GPUs, express The inference computation time, is n on cloud c * GPUs serving layer j of the large model, Indicates the price of the nth GPU on cloud c; The calculation function expression of the communication cost is: In the above formula, express The communication data size of tensor parallel communication, is the data transmission price between the nth GPU and the n′th GPU, Represents the set of GPUs that provide computing services for the j+1th layer of the large model. is the communication data size of pipeline parallel communication, and: In the above formula, b t Indicates the batch size of large model inference service, is the sequence length of the input prompt for the large model, H is the hidden layer dimension of the transformer in the large model, and B type The number of bytes used for inference calculation accuracy for large models, Expressed as The number of GPUs, Represents the GPU that provides computing services for the jth layer of the large model, The sequence length of output tokens generated for large models.

8. A cost-aware large model deployment optimization system in an inter-cloud environment, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the cost-aware large model deployment optimization method in an inter-cloud environment as recited in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute, through a processor, the cost-aware large model deployment optimization method in a cloud environment as recited in any one of claims 1 to 7.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute, through a processor, the cost-aware large model deployment optimization method in a cloud environment as recited in any one of claims 1 to 7.

Images