A multi-model parallel performance evaluation and cost optimization method and system

Abstract

The application relates to the technical field of distributed reasoning resource scheduling, and provides a multi-model parallel performance evaluation and cost optimization method and system. By decomposing a request processing process into a preprocessing stage and an inference calculation stage for measurement, a performance attenuation record is established, and a target deployment strategy is determined from a pareto frontier scheme set based on business priority, so that the problems that performance attenuation is difficult to quantify and resource scheduling lacks a dynamic optimization mechanism when multiple models are co-located are solved, and the method has the advantages that performance guarantee and resource efficiency balance in a multi-model parallel scene can be realized.

Description

Technical Field

[0001] This application relates to the field of distributed inference resource scheduling technology, and more specifically, to a method and system for multi-model parallel performance evaluation and cost optimization. Background Technology

[0002] With the rapid development of artificial intelligence technology, the demand for parallel deployment of multiple deep learning models on the same physical node for inference services is increasing. In multi-model co-location scenarios, models share resources such as CPU, GPU memory, and memory bandwidth, leading to performance interference between them and significantly increasing the actual processing time of each model. In existing technologies, multi-model deployment solutions are usually based on empirical rules or static configurations, lacking refined quantification methods for performance degradation during model co-location, and unable to dynamically adjust resource allocation strategies according to real-time business load, resulting in low resource utilization or substandard service quality.

[0003] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0004] The purpose of this application is to provide a method and system for performance evaluation and cost optimization of multi-model parallelism, which has the advantages of enabling fine-grained quantification of performance degradation and dynamic resource scheduling in multi-model parallel scenarios, thereby improving resource utilization efficiency and service quality.

[0005] This application provides a method for multi-model parallel performance evaluation and cost optimization, including: Obtain request processing records for multiple models within a historical time period; based on the request processing records, calculate the benchmark performance data of each model under exclusive resource conditions; Multiple models are deployed on the same physical node in different combinations. The actual performance data of each model under the combined deployment is collected. The performance degradation ratio relative to the benchmark performance data is calculated, and a performance degradation record is established. Predict the request arrival rate of each model in the next time period; based on the performance degradation record and the predicted request arrival rate, generate multiple candidate deployment strategies with the goal of meeting the preset processing time requirements of each model and minimizing the total resource cost, and select a Pareto frontier solution set. Based on the current business priority, a target deployment strategy is determined from the Pareto frontier scheme set. The target deployment strategy includes the target resource quota for each model in the next time period and the model deployment combination. According to the target deployment strategy, adjust the resource allocation and deployment location of each model; and during the adjusted operation, monitor the actual processing time of each model. When the deviation between the actual processing time and the preset processing time requirement exceeds the preset threshold, execute a self-repair action.

[0006] Furthermore, obtaining request processing records for multiple models within a historical time period includes: Record the queuing time for each request at the model processing front end, the time spent in the preprocessing stage, and the time spent in the inference calculation stage; The queuing waiting time, preprocessing stage time, and inference calculation stage time are associated with the corresponding requests and stored. The request processing record also includes contention intensity indicators for each resource type, including at least one of shared memory bandwidth contention coefficient, shared cache contention coefficient, and I / O bandwidth contention coefficient.

[0007] Furthermore, the step of statistically analyzing the benchmark performance data of each model under exclusive resource conditions based on request processing records includes: Based on the length of the input data and the length of the output data associated with each request, the requests are divided into three types: short input long output, long input short output, and balanced requests. Based on the time consumed in the preprocessing stage and the time consumed in the inference calculation stage associated with each request, the average time consumed in each stage when each model processes the above three types of requests is calculated respectively. The baseline performance data is generated based on the average time taken for each stage.

[0008] Furthermore, the establishment of performance degradation records includes: The request processing process is divided into a preprocessing stage and an inference computation stage. The actual stage time of each model when processing different types of requests under combined deployment is collected. Calculate the percentage change in actual stage time relative to the corresponding stage time in the baseline performance data; When the change in time consumption during the preprocessing stage of any model in the same model combination is greater than the change in time consumption during the inference computation stage, the source of the time increase under that combination is marked as an increase in time consumption during the preprocessing stage; when the change in time consumption during the inference computation stage is greater than the change in time consumption during the preprocessing stage, the source of the time increase under that combination is marked as an increase in time consumption during the inference computation stage; the source of the time increase and the change in time consumption during each stage are associated and stored in the performance degradation record.

[0009] Furthermore, several candidate deployment strategies are generated, including: Based on the length of the input data and the length of the output data, requests are divided into three types: short input long output, long input short output, and balanced requests. Predict the distribution ratio of request types for each model in the next time period, and determine the type with the most requests for each model in the next time period; From the performance degradation records, find the sources of increased time consumption and the percentage change in time consumption at each stage for each model combination under this type; The model combination with the smallest change in stage time under this type is determined as the candidate deployment combination; Based on the request arrival rate predicted by each model and the phase time change ratio under the candidate deployment combination, the resource quota required to meet the preset processing time requirements is calculated, and the candidate deployment strategy is obtained.

[0010] Furthermore, the step of determining the target deployment strategy from the Pareto frontier solution set based on current business priorities includes: When the current business priority is performance, select the candidate deployment strategy from the Pareto front solution set that has the closest processing time to the preset processing time requirement and the relatively high total resource cost as the target deployment strategy. When the current business priority is cost priority, the candidate deployment strategy with the lowest total resource cost and processing time not exceeding the preset processing time requirement is selected from the Pareto frontier solution set as the target deployment strategy. When the current business priority is balanced, the candidate deployment strategy with the best weighted comprehensive score of processing time and total resource cost is selected from the Pareto frontier solution set as the target deployment strategy.

[0011] Furthermore, adjusting the resource allocation and deployment location of each model according to the target deployment strategy includes: When the target resource quota of any model is greater than the current resource quota, if the remaining resources of the physical node where the model is located meet the target resource quota, the model is directly expanded; if the remaining resources are insufficient, the physical node where the model replica has been deployed and the proportion of the model's phase time change on the physical node where the replica is located is less than the preset change threshold is selected first, the newly arrived requests are routed to the replica and the instances to be expanded are simultaneously scheduled to the physical node. When the target resource quota of any model is less than the current resource quota, the excess resources of that model on the current physical node are released, and the released resources are allocated to other models on the same physical node whose target resource quota is greater than the current resource quota and whose stage time change ratio is less than the preset change threshold.

[0012] Furthermore, the self-repair action includes: Record the front-end queuing time of each model on the current physical node. The front-end queuing time is the time interval between the arrival of the request and the start of preprocessing. When the front-end queuing time of any model exceeds the preset queuing threshold, obtain the list of co-located models on the physical node where that model is located; From the performance degradation records, query the percentage change in stage consumption time corresponding to the combination of this model and each model in the co-located model list; If a co-located model exists and the change ratio of its stage time exceeds a preset change threshold, then the co-located model will be migrated from the current physical node to a physical node where the change ratio of its stage time is less than the preset change threshold, while keeping the current resource quota of the model unchanged.

[0013] Furthermore, the total resource cost is calculated based on at least one of the following technical indicators: The product of CPU core count and corresponding unit price coefficient, GPU memory capacity and corresponding unit price coefficient, network bandwidth and corresponding unit price coefficient, and memory capacity and corresponding unit price coefficient are all calculated. Each unit price coefficient is dynamically obtained from the real-time pricing interface of the current cloud computing platform.

[0014] Secondly, this application also proposes a multi-model parallel performance evaluation and cost optimization system, including: The acquisition and statistics module is used to acquire request processing records of multiple models within a historical time period; and to collect benchmark performance data of each model under exclusive resource conditions based on the request processing records. The deployment and data acquisition module is used to deploy multiple models in different combinations on the same physical node, collect the actual performance data of each model under the combined deployment, calculate the performance degradation ratio relative to the benchmark performance data, and establish a performance degradation record. The prediction and generation module is used to predict the request arrival rate of each model in the next time period; based on the performance degradation record and the predicted request arrival rate, with the goal of meeting the preset processing time requirements of each model and minimizing the total resource cost, multiple candidate deployment strategies are generated, and a Pareto frontier solution set is selected. The determination module is used to determine the target deployment strategy from the Pareto frontier scheme set based on the current business priority. The target deployment strategy includes the target resource quota for each model in the next time period and the model deployment combination. The adjustment and optimization module is used to adjust the resource allocation and deployment location of each model according to the target deployment strategy; and during the adjusted operation, it monitors the actual processing time of each model, and performs a self-repair action when the deviation between the actual processing time and the preset processing time requirement exceeds a preset threshold.

[0015] As can be seen from the above, the multi-model parallel performance evaluation and cost optimization method and system provided in this application decomposes the request processing process into a preprocessing stage and an inference calculation stage for separate measurement, establishes a performance degradation record consisting of the proportion of stage time change and the source of time increase, and generates candidate deployment strategies based on request type distribution prediction and Pareto front screening, and dynamically determines the target deployment strategy in combination with business priority. This solves the problems of difficulty in quantifying performance degradation and lack of dynamic optimization mechanism for resource scheduling when multiple models are co-located. It has the advantages of achieving a balance between performance guarantee and resource efficiency in multi-model parallel scenarios, and improving the flexibility of deployment decision and operational stability. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the steps of the multi-model parallel performance evaluation and cost optimization method disclosed in the embodiments of the present invention; Figure 2 This is a flowchart illustrating the steps for establishing performance degradation records as disclosed in an embodiment of the present invention; Figure 3 This is a flowchart illustrating the steps for generating multiple candidate deployment strategies as disclosed in an embodiment of the present invention. Figure 4 This is a flowchart illustrating the steps for performing self-repair actions as disclosed in an embodiment of the present invention; Figure 5 This is a schematic diagram of the multi-model parallel performance evaluation and cost optimization system structure disclosed in the embodiments of the present invention. Detailed Implementation

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which these embodiments belong; the terminology used herein and in the specification of the application is for the purpose of describing particular embodiments only and is not intended to limit these embodiments; the terms "comprising" and "having," and any variations thereof, in the specification of these embodiments and the foregoing drawings, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification of these embodiments and the foregoing drawings are used to distinguish different objects, not to describe a particular order.

[0019] The implementation details of the technical solution in this embodiment are described in detail below: This application proposes a multi-model parallel performance evaluation and cost optimization method, such as... Figure 1 As shown, the method includes: S101, obtain request processing records of multiple models within a historical time period; based on the request processing records, calculate the benchmark performance data of each model under exclusive resource conditions.

[0020] Furthermore, in S101, obtaining request processing records of multiple models within a historical time period includes: Record the queuing time for each request at the model processing front end, the time spent in the preprocessing stage, and the time spent in the inference calculation stage; The queuing waiting time, preprocessing stage time, and inference calculation stage time are associated with the corresponding requests and stored. The request processing record also includes contention intensity indicators for each resource type, including at least one of shared memory bandwidth contention coefficient, shared cache contention coefficient, and I / O bandwidth contention coefficient.

[0021] Specifically, in this embodiment, request processing records for Model A and Model B over the past 7 days are extracted from the log database of the distributed inference platform. Model A is an image recognition model, and Model B is a text generation model. Each request processing record includes the arrival time of the request, the processing completion time, the length of the input data, the length of the output data, the queuing time at the model processing front end, the time spent in the preprocessing stage, the time spent in the inference computation stage, and the contention intensity index for each resource type.

[0022] The queuing time is defined as the time interval between the request arriving at the model service entry point and the request starting preprocessing. The preprocessing stage time is defined as the time required for the request data to go from entering the model to completing format conversion and normalization. The inference computation stage time is defined as the time required for the model to complete forward computation.

[0023] Contention intensity metrics are calculated as follows: Shared memory bandwidth contention coefficient is the ratio of the memory bandwidth currently used by the model to the total memory bandwidth of the physical node. Shared cache contention coefficient is the ratio of the cache capacity currently used by the model to the total cache capacity of the physical node. I / O bandwidth contention coefficient is the ratio of the I / O bandwidth currently used by the model to the total I / O bandwidth of the physical node. For example, if model A uses 80GB / s of memory bandwidth on a certain physical node, and the total memory bandwidth of that node is 200GB / s, then the shared memory bandwidth contention coefficient is 0.4.

[0024] The queuing time, preprocessing time, and inference time are associated with the corresponding requests and stored to form a structured request processing record table. This record table uses the request identifier as the primary key, and the duration fields and contention intensity indicators as attribute fields, facilitating subsequent aggregation and statistics by model and request type.

[0025] Further, in S101, the step of statistically analyzing the baseline performance data of each model under exclusive resource conditions based on the request processing records includes: classifying requests into three types—short input long output, long input short output, and balanced—based on the length of the input data and the length of the output data associated with each request; calculating the average time consumed in each stage of processing the above three types of requests based on the time consumed in the preprocessing stage and the time consumed in the inference calculation stage associated with each request; and generating the baseline performance data based on the average time consumed in each stage.

[0026] Specifically, requests are categorized into three types based on the length of the input and output data. The "Short Input Long Output" type is defined as requests with an input data length less than 100KB and an output data length greater than 500KB. The "Long Input Short Output" type is defined as requests with an input data length greater than 500KB and an output data length less than 100KB. The "Balanced" type is defined as requests that do not belong to either of the above two categories.

[0027] For Model A handling short-input, long-output requests, extract n records of this type of request from the historical data, recording the preprocessing and inference computation times under exclusive resource conditions. Calculate the average preprocessing time, Tpre, which is the arithmetic mean of the preprocessing times from the n records. Similarly, calculate the average inference computation time, Tinf.

[0028] Queuing time is not included in the baseline performance data. This is because baseline performance data is used to characterize the model's inherent capabilities under exclusive resource conditions, while queuing time is affected by front-end concurrency and scheduling strategies, and is unrelated to the model's own reasoning ability.

[0029] The final generated benchmark performance data is represented in binary form, i.e., the benchmark performance data of Model A for handling short input long output requests is [Tpre, Tinf]. Similarly, benchmark performance data for Model A for handling long input short output requests and balanced requests, as well as benchmark performance data for Model B for handling all three types of requests, are generated, for a total of 6 sets of benchmark performance data, which serve as reference benchmarks for subsequent calculation of performance degradation ratios.

[0030] S102, deploy multiple models in different combinations on the same physical node, collect the actual performance data of each model under the combined deployment, calculate the performance degradation ratio relative to the benchmark performance data, and establish a performance degradation record.

[0031] Specifically, in this embodiment, Model A and Model B can be deployed on the same physical node in different combinations. The physical node is configured with an 8-core CPU, 32GB of memory, and a single inference accelerator card. With Model A exclusively using this physical node, its benchmark performance data for processing short-input long-output requests shows an average preprocessing time (TpreA) of 5 milliseconds and an average inference computation time (TinfA) of 30 milliseconds. With Model B exclusively using this physical node, its benchmark performance data for processing long-input short-output requests shows an average preprocessing time (TpreB) of 8 milliseconds and an average inference computation time (TinfB) of 50 milliseconds.

[0032] Furthermore, in S102, the establishment of performance degradation records, such as... Figure 2 As shown, it includes the following steps: S2001 divides the request processing process into a preprocessing stage and an inference calculation stage, and collects the actual stage time of each model when processing different types of requests under combined deployment; Specifically, Model A and Model B are deployed simultaneously on the aforementioned physical nodes, maintaining a concurrent request count of 100 for both models. Short-input, long-output test requests are sent to Model A, and long-input, short-output test requests are sent to Model B. The actual time consumed by Model A in processing these requests at each stage is collected through instrumentation, denoted as the preprocessing stage time (TpreAco) and the inference calculation stage time (TinfAco). Similarly, the actual time consumed by Model B at each stage under co-location conditions is collected, denoted as TpreBco and TinfBco. For example, in one test cycle, Model A's TpreAco is measured to be 7 milliseconds and TinfAco to be 45 milliseconds; Model B's TpreBco is measured to be 12 milliseconds and TinfBco to be 80 milliseconds.

[0033] S2002, calculate the percentage change in actual stage time relative to the corresponding stage time in the baseline performance data; Specifically, the formula for calculating the percentage change in stage duration is as follows: R = (Tactual - Tbase) / Tbase × 100% In the formula, R is the percentage change in stage duration; Tactual is the actual stage duration, which is the duration of a certain stage under co-location conditions measured in S2001; Tbase is the baseline stage duration, which is the average duration of the corresponding stage under exclusive resource conditions obtained in S101.

[0034] For the co-location combination of Model A and Model B, calculate the percentage change in stage time RpreA of the preprocessing stage of Model A: RpreA = (TpreAco - TpreA) / TpreA × 100% In the formula, TpreAco is the actual time consumed in the preprocessing stage of model A under co-location conditions, and TpreA is the baseline time consumed in the preprocessing stage of model A under exclusive resource conditions. Substituting the measured data, TpreAco is 7 milliseconds and TpreA is 5 milliseconds, then RpreA equals the difference between 7 and 5 divided by 5, and then multiplied by 100%, resulting in 40%.

[0035] The percentage change in time consumed during the inference calculation phase of computation model A, RinfA: RinfA = (TinfAco - TinfA) / TinfA × 100% In the formula, TinfAco is the actual time consumed by model A during the inference computation phase under co-location conditions, and TinfA is the baseline time consumed by model A during the inference computation phase under exclusive resource conditions. Substituting the measured data, TinfAco is 45 milliseconds and TinfA is 30 milliseconds. Therefore, RinfA equals the difference between 45 and 30, divided by 30, and then multiplied by 100%, resulting in 50%.

[0036] Similarly, calculate the percentage change in stage time RpreB during the preprocessing stage of model B: RpreB = (TpreBco - TpreB) / TpreB × 100% In the formula, TpreBco is the actual time consumed in the preprocessing stage of model B under co-location conditions, and TpreB is the baseline time consumed in the preprocessing stage of model B under exclusive resource conditions. Substituting the measured data, TpreBco is 12 milliseconds and TpreB is 8 milliseconds, then RpreB equals the difference between 12 and 8 divided by 8, and then multiplied by 100%, resulting in 50%.

[0037] The percentage change in time consumed during the inference calculation phase of computation model B, RinfB: RinfB = (TinfBco - TinfB) / TinfB × 100% In the formula, TinfBco is the actual time consumed by Model B during the inference computation phase under co-location conditions, and TinfB is the baseline time consumed by Model B during the inference computation phase under exclusive resource conditions. Substituting the measured data, TinfBco is 80 milliseconds and TinfB is 50 milliseconds. Therefore, TinfB equals the difference between 80 and 50 divided by 50, then multiplied by 100%, resulting in 60%.

[0038] S2003: When the change ratio of the preprocessing stage time consumption of any model in the same model combination is greater than the change ratio of the inference calculation stage time consumption, the source of the time increase under the combination is marked as the preprocessing stage time increase; when the change ratio of the inference calculation stage time consumption is greater than the change ratio of the preprocessing stage time consumption, the source of the time increase under the combination is marked as the inference calculation stage time increase; the source of the time increase and the stage time change ratio are associated and stored in the performance degradation record.

[0039] Specifically, for the co-location of Model A and Model B, the change in time during the preprocessing stage of Model A, RpreA, is 40%, and the change in time during the inference computation stage, RinfA, is 50%. Since RinfA is greater than RpreA, meaning the change in time during the inference computation stage is greater than the change in time during the preprocessing stage, the source of the increased time consumption of Model A under this combination is marked as an increase in time consumption during the inference computation stage.

[0040] The time change percentage (RpreB) for the preprocessing stage of Model B is 50%, and the time change percentage (RinfB) for the inference calculation stage is 60%. Since RinfB is greater than RpreB, the increase in time consumption for Model B under this combination is also marked as an increase in time consumption during the inference calculation stage.

[0041] The calculation results are stored in a structured format in a performance degradation record. This record includes fields for combination identifier, model identifier, request type, percentage change in time consumption for each stage, and source of time increase. For example, in the performance degradation record corresponding to the co-location of Model A and Model B, the entries for Model A are: combination identifier AB, model identifier A, request type short input long output, RpreA 40%, RinfA 50%, source of time increase: increased time consumption in the inference calculation stage. The entries for Model B are: combination identifier AB, model identifier B, request type long input short output, RpreB 50%, RinfB 60%, source of time increase: increased time consumption in the inference calculation stage.

[0042] This application's solution, by breaking down the request processing into a preprocessing stage and an inference computation stage for separate measurement, can precisely pinpoint the root cause of performance degradation during model co-location. Calculating the percentage change in stage latency transforms performance degradation from a qualitative description into a quantitative indicator, while identifying the source of increased latency attributes the degradation to a specific processing stage. This fine-grained analysis method avoids the shortcomings of merely observing end-to-end latency changes without identifying bottlenecks, providing a reliable data foundation for subsequent targeted selection of co-location combinations and resource allocation. For example, when the source of increased latency for a combination is identified as increased latency in the inference computation stage, it indicates resource competition at the computational unit level among the models in that combination. Subsequently, when generating candidate deployment strategies, at least one model in this combination can be migrated to a physical node with more abundant computing resources, or the inference computation resource quota for that model can be increased to alleviate competition at the computational level.

[0043] S103, predict the request arrival rate of each model in the next time period; based on the performance degradation record and the predicted request arrival rate, generate multiple candidate deployment strategies with the goal of meeting the preset processing time requirements of each model and minimizing the total resource cost, and select a Pareto frontier solution set. Furthermore, in S103, multiple candidate deployment strategies are generated, such as Figure 3 As shown, it includes the following steps: S3001 classifies requests into three types based on the length of input data and the length of output data: short input long output, long input short output, and balanced request. Specifically, input data length thresholds and output data length thresholds are set. When the requested input length is less than the short input threshold and the output length is greater than the long output threshold, it is classified into the short input long output class; when the input length is greater than the long input threshold and the output length is less than the short output threshold, it is classified into the long input short output class; and cases that do not meet either of the above two conditions are classified into the balanced class.

[0044] S3002, predict the distribution ratio of request types for each model in the next period, and determine the type with the most requests for each model in the next period; Specifically, for each model, the number of requests of each type arrives within a historical time period is statistically analyzed, and a moving average method is used to extrapolate the distribution ratio for the next time period. The request type with the highest percentage is determined as the type with the most requests in the next time period for that model. For example, Model A has the highest percentage of short input long output requests, while Model B has the highest percentage of long input short output requests.

[0045] S3003, from the performance degradation record, find the source of the increase in time consumption and the percentage change in time consumption in each stage for each model combination under this type; Specifically, for model A, find the percentage change in stage latency for each co-configuration combination containing model A when processing short-input, long-output requests in the performance degradation record. For model B, find the percentage change in stage latency for each co-configuration combination containing model B when processing long-input, short-output requests.

[0046] S3004. The model combination with the smallest change in stage consumption under this type is determined as the candidate deployment combination. Specifically, for Model A, the proportion of stage time change for each co-location combination is compared, and the combination with the smallest proportion is determined as the candidate deployment combination. For Model B, the same comparison is made, and the co-location combination with the smallest proportion of stage time change is determined as the candidate deployment combination.

[0047] S3005: Based on the request arrival rate predicted by each model and the phase time change ratio under the candidate deployment combination, calculate the resource quota required to meet the preset processing time requirements and obtain the candidate deployment strategy.

[0048] Specifically, the preset processing time requirement is set to 1.5 times the processing time of a single request under exclusive resource conditions for each model. The resource quota is calculated as N equals R divided by C and rounded up. Here, N is the required number of instances, R is the predicted request arrival rate of the model, and C is the processing capacity of a single instance of the model under the candidate deployment combination, obtained by dividing 1000 milliseconds by the actual processing time of a single request under that combination. Converting the calculated number of instances into the number of GPU memory and CPU cores yields the resource quota for that model. The resource quota of each model is then associated with its candidate deployment combination to form a candidate deployment strategy.

[0049] After generating multiple candidate deployment strategies, a Pareto front set is selected with the goal of minimizing the total resource cost while meeting the preset processing time requirements of each model. Specifically, the total resource cost of each candidate deployment strategy is calculated as the sum of the products of the number of CPU cores, GPU memory capacity, network bandwidth, and memory capacity, and their corresponding unit price coefficients. Candidates that are strictly dominated by other strategies are eliminated; that is, if strategy A is superior to strategy B in both total resource cost and processing time, strategy B is eliminated. The remaining candidate strategies constitute the Pareto front set, where any strategy in this set, when reducing the total resource cost, will inevitably cause the processing time of at least one model to exceed the preset requirement, or when shortening the processing time, will inevitably increase the total resource cost.

[0050] For example, if Model A predicts a request arrival rate of 200 requests per second, and the actual processing time for a single request is measured to be 45 milliseconds under the candidate deployment combination AC, then the processing capacity of a single instance is approximately 1000 divided by 45, which equals 22 requests per second. The required number of instances is 200 divided by 22 and rounded up, resulting in 10 instances. Each instance is configured with 2GB of video memory and 1 CPU core, so the resource quota for Model A is 20GB of video memory and 10 CPU cores. Similarly, after calculating the resource quota for Model B, the quotas of both models are associated with their respective candidate deployment combinations to form a complete candidate deployment strategy.

[0051] Furthermore, the total resource cost is calculated based on at least one of the following technical indicators: The product of CPU core count and corresponding unit price coefficient, GPU memory capacity and corresponding unit price coefficient, network bandwidth and corresponding unit price coefficient, and memory capacity and corresponding unit price coefficient are all calculated. Each unit price coefficient is dynamically obtained from the real-time pricing interface of the current cloud computing platform.

[0052] Specifically, when calculating the resource costs of candidate deployment strategies, the total number of CPU cores required by the strategy is calculated and multiplied by the CPU core unit price coefficient returned by the real-time pricing interface of the current cloud computing platform to obtain the CPU cost. The total GPU memory capacity required by the strategy is calculated and multiplied by the memory unit price coefficient returned by the real-time pricing interface of the current cloud computing platform to obtain the memory cost. The total network bandwidth required by the strategy is calculated and multiplied by the bandwidth unit price coefficient returned by the real-time pricing interface of the current cloud computing platform to obtain the bandwidth cost. The total memory capacity required by the strategy is calculated and multiplied by the memory unit price coefficient returned by the real-time pricing interface of the current cloud computing platform to obtain the memory cost.

[0053] The total resource cost is calculated as follows: Ctotal equals Ccpu plus Cgpu plus Cnet plus Cmem. Where Ctotal is the total resource cost, Ccpu is the product of the number of CPU cores and their unit price coefficient, Cgpu is the product of the GPU memory capacity and its unit price coefficient, Cnet is the product of the network bandwidth and its unit price coefficient, and Cmem is the product of the memory capacity and its unit price coefficient. Each unit price coefficient is dynamically obtained from the current cloud computing platform's real-time pricing interface and is updated in real time according to market price fluctuations.

[0054] After generating multiple candidate deployment strategies, the total resource cost of each strategy is calculated according to the formula above. Based on the actual processing time of each model under each strategy, a Pareto front set of solutions is selected. Any strategy in this set, when reducing the total resource cost, will inevitably cause the processing time of at least one model to exceed the preset requirement, or when shortening the processing time, will inevitably lead to an increase in the total resource cost.

[0055] This application's solution breaks down total resource costs into four technical indicators: CPU core count, GPU memory capacity, network bandwidth, and memory capacity. It also introduces a real-time pricing interface from a cloud computing platform to dynamically obtain unit price coefficients, transforming cost calculation from an abstract concept into quantifiable technical resource management. This quantitative method based on real-time market prices ensures the objectivity and timeliness of cost comparisons between different candidate deployment strategies, thus providing a reliable data foundation for selecting Pareto frontier solutions.

[0056] S104. Determine the target deployment strategy from the Pareto frontier scheme set according to the current business priority. The target deployment strategy includes the target resource quota for each model in the next time period and the model deployment combination.

[0057] Furthermore, in S104, determining the target deployment strategy from the Pareto front solution set based on the current business priority includes: when the current business priority is performance priority, selecting the candidate deployment strategy from the Pareto front solution set whose processing time is closest to the preset processing time requirement and whose total resource cost is relatively high as the target deployment strategy; when the current business priority is cost priority, selecting the candidate deployment strategy from the Pareto front solution set whose total resource cost is the lowest and whose processing time does not exceed the preset processing time requirement as the target deployment strategy; when the current business priority is balanced mode, selecting the candidate deployment strategy from the Pareto front solution set whose weighted comprehensive score of processing time and total resource cost is optimal as the target deployment strategy.

[0058] Specifically, when the current business priority is performance, the candidate deployment strategy with the closest processing time to the preset processing time requirement and the relatively high total resource cost is selected from the Pareto frontier solution set as the target deployment strategy.

[0059] Specifically, after generating the Pareto front solution set, several candidate deployment strategies are obtained. In performance-first mode, the actual total processing time of each strategy is compared with the preset processing time requirement, and the total resource cost of each strategy is also compared. The strategy with the processing time closest to the preset requirement and a cost at a relatively high level among the frontier solutions is selected as the target deployment strategy to ensure service quality.

[0060] When the current business priority is cost priority, the candidate deployment strategy with the lowest total resource cost and processing time not exceeding the preset processing time requirement is selected from the Pareto frontier solution set as the target deployment strategy.

[0061] Specifically, in the cost-first mode, the strategies in the Pareto frontier set are traversed, and the one with the lowest total resource cost is selected. Simultaneously, it is verified that the processing time of each model under this strategy does not exceed the preset processing time requirement. If the constraints are met, this strategy is determined as the target deployment strategy to control resource expenditure.

[0062] When the current business priority is balanced, the candidate deployment strategy with the best weighted comprehensive score of processing time and total resource cost is selected from the Pareto frontier solution set as the target deployment strategy.

[0063] Specifically, the formula for calculating the weighted composite score S is as follows: S = α × Tref / Tactual + β × Cref / Cactual In the formula, S is the weighted comprehensive score; α is the processing time weight coefficient; β is the total resource cost weight coefficient, and the sum of α and β equals 1, with α being greater than β, to reflect the priority guarantee of processing efficiency in the balanced mode; Tref is the sum of the preset processing time requirements, that is, the sum of the preset processing time requirements of each model; Tactual is the sum of the actual processing time of the candidate deployment strategy, that is, the sum of the processing time of each model under the strategy; Cref is the cost benchmark, taking the total resource cost of the performance-first strategy; Cactual is the actual total resource cost of the candidate deployment strategy.

[0064] In equilibrium mode, each strategy in the Pareto front set is traversed, and its weighted composite score is calculated for each strategy. The strategy with the highest score is determined as the target deployment strategy to achieve a balance between processing time and total resource cost.

[0065] This application's solution introduces business priorities into the Pareto front solution set for secondary screening, enabling deployment decisions to flexibly adapt to the needs of different business scenarios. In performance-first mode, the solution with the shortest processing time and highest cost ensures service quality; in cost-first mode, the solution with the lowest cost and meeting performance baselines controls expenditures; and in balanced mode, a weighted scoring system balances efficiency and economy. This dynamic selection mechanism based on business priorities avoids resource misallocation caused by a single optimization objective and improves decision-making flexibility in scenarios involving parallel deployment of multiple models.

[0066] S105, according to the target deployment strategy, adjust the resource allocation and deployment location of each model; and during the adjusted operation, monitor the actual processing time of each model, and when the deviation between the actual processing time and the preset processing time requirement exceeds the preset threshold, execute a self-repair action.

[0067] Furthermore, in S105, adjusting the resource allocation and deployment location of each model according to the target deployment strategy includes: When the target resource quota of any model is greater than the current resource quota, if the remaining resources of the physical node where the model is located meet the target resource quota, the model is directly expanded; if the remaining resources are insufficient, the physical node where the model replica has been deployed and the proportion of the model's phase time change on the physical node where the replica is located is less than the preset change threshold is selected first, the newly arrived requests are routed to the replica and the instances to be expanded are simultaneously scheduled to the physical node. When the target resource quota of any model is less than the current resource quota, the excess resources of that model on the current physical node are released, and the released resources are allocated to other models on the same physical node whose target resource quota is greater than the current resource quota and whose stage time change ratio is less than the preset change threshold.

[0068] Specifically, in this embodiment, when the target resource quota for model A is greater than the current resource quota, the remaining resources of the physical node where model A is located are checked. If the remaining resources meet the target resource quota, the number of instances of model A and the corresponding allocation of video memory and CPU cores are directly increased on that physical node to complete the expansion. If the remaining resources are insufficient, other physical nodes that have deployed replicas of model A are traversed, and nodes where the proportion of time change for the co-located model to model A on the replica node is less than a preset change threshold are selected. Newly arriving requests are routed to the replica, and instances to be expanded are simultaneously scheduled to the physical node to achieve cross-node expansion.

[0069] When the target resource quota of model B is less than the current resource quota, release the excess instances of model B on the current physical node, along with their corresponding GPU memory and CPU cores. Allocate the released resources to other models on the same physical node whose target resource quota is greater than the current resource quota and whose stage time change ratio is less than a preset change threshold, such as model C, to improve the overall resource utilization of that node.

[0070] Furthermore, in S105, during the adjusted operation, the actual processing time of each model is monitored. When the deviation between the actual processing time and the preset processing time requirement exceeds the preset threshold, a self-repair action is executed.

[0071] The deviation ratio is calculated as follows: δ equals Tactual minus Trequire, divided by Trequire, and then multiplied by 100%. Here, δ represents the deviation ratio, Tactual represents the actual processing time, and Trequire represents the preset processing time requirement. When δ exceeds the preset threshold, a self-healing process is triggered.

[0072] Furthermore, in S105, the self-repair action is performed, such as... Figure 4 As shown, it includes: S4001, record the front-end queuing time of each model on the current physical node, where the front-end queuing time is the time interval between the arrival of the request and the start of preprocessing; Specifically, during the operation of Model A, the arrival time and the start time of preprocessing are recorded for each request, and the difference between the two is the front-end queuing time. The front-end queuing time per unit time is continuously counted to form a queuing time sequence.

[0073] S4002, When the front-end queuing time of any model exceeds the preset queuing threshold, obtain the list of co-located models on the physical node where the model is located; Specifically, when the queuing time of model A exceeds the preset queuing threshold, it indicates that the current deployment status may be affecting processing efficiency. The deployment metadata of the physical node where model A is currently located is read to obtain a list of co-located models on that node, excluding model A, including model B and model C.

[0074] S4003, from the performance degradation record, query the percentage change in stage consumption time corresponding to the combination of this model and each model in the co-located model list; Specifically, in the performance degradation records, the percentage change in stage time for model A in combinations AB and AC is searched. The percentage change in stage time for each combination is obtained to determine if there are any co-located models that cause a significant decrease in the performance of model A.

[0075] S4004, If a co-located model exists that causes the phase time change ratio of the model to be greater than the preset change threshold, then the co-located model is moved out of the current physical node and moved to a physical node where the phase time change ratio is less than the preset change threshold, while keeping the current resource quota of the model unchanged.

[0076] Specifically, if the percentage change in stage time of model A in combination AB is greater than a preset threshold, while the percentage change in stage time of model A in combination AC is less than the preset threshold, it indicates that model B has significantly interfered with the performance of model A. In this case, model B will be migrated from its current physical node to a physical node where the percentage change in stage time is less than the preset threshold. The resource quota for model A will remain unchanged, and performance will be restored only by replacing the co-location partner.

[0077] Secondly, this embodiment also proposes a multi-model parallel performance evaluation and cost optimization system, such as... Figure 5 ,include: The acquisition and statistics module 501 is used to acquire request processing records of multiple models within a historical time period; based on the request processing records, the baseline performance data of each model under exclusive resource conditions is statistically analyzed. The deployment and acquisition module 502 is used to deploy multiple models in different combinations on the same physical node, collect the actual performance data of each model under the combined deployment, calculate the performance degradation ratio relative to the benchmark performance data, and establish a performance degradation record. The prediction and generation module 503 is used to predict the request arrival rate of each model in the next time period; based on the performance degradation record and the predicted request arrival rate, with the goal of meeting the preset processing time requirements of each model and minimizing the total resource cost, it generates multiple candidate deployment strategies and selects a Pareto frontier solution set. The determination module 504 is used to determine the target deployment strategy from the Pareto frontier scheme set according to the current business priority. The target deployment strategy includes the target resource quota for each model in the next time period and the model deployment combination. The adjustment and optimization module 505 is used to adjust the resource allocation and deployment location of each model according to the target deployment strategy; and during the adjusted operation, it monitors the actual processing time of each model, and executes a self-repair action when the deviation between the actual processing time and the preset processing time requirement exceeds a preset threshold. This system can be used to perform the multi-model parallel performance evaluation and cost optimization method described in the first aspect, which will not be elaborated further here.

[0078] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for multi-model parallel performance evaluation and cost optimization, characterized in that, include: Retrieve request processing records for multiple models within a historical time period; Based on the request processing records, the baseline performance data of each model under exclusive resource conditions are statistically analyzed; Multiple models are deployed on the same physical node in different combinations. The actual performance data of each model under the combined deployment is collected. The performance degradation ratio relative to the benchmark performance data is calculated, and a performance degradation record is established. Predict the request arrival rate for each model in the next time period; Based on the performance degradation records and the predicted request arrival rate, with the goal of meeting the preset processing time requirements of each model and minimizing the total resource cost, multiple candidate deployment strategies are generated, and a Pareto frontier solution set is selected. Based on the current business priority, a target deployment strategy is determined from the Pareto frontier scheme set. The target deployment strategy includes the target resource quota for each model in the next time period and the model deployment combination. Adjust the resource allocation and deployment location of each model according to the target deployment strategy; In addition, during the adjusted operation, the actual processing time of each model is monitored, and when the deviation between the actual processing time and the preset processing time requirement exceeds the preset threshold, a self-repair action is executed.

2. The multi-model parallel performance evaluation and cost optimization method according to claim 1, characterized in that, The process of obtaining request processing records for multiple models within a historical time period includes: Record the queuing time for each request at the model processing front end, the time spent in the preprocessing stage, and the time spent in the inference calculation stage; The queuing waiting time, preprocessing stage time, and inference calculation stage time are associated with the corresponding requests and stored. The request processing record also includes contention intensity indicators for each resource type, including at least one of shared memory bandwidth contention coefficient, shared cache contention coefficient, and I / O bandwidth contention coefficient.

3. The multi-model parallel performance evaluation and cost optimization method according to claim 2, characterized in that, The step of statistically analyzing the baseline performance data of each model under exclusive resource conditions based on request processing records includes: Based on the length of the input data and the length of the output data associated with each request, the requests are divided into three types: short input long output, long input short output, and balanced requests. Based on the time consumed in the preprocessing stage and the time consumed in the inference calculation stage associated with each request, the average time consumed in each stage when each model processes the above three types of requests is calculated respectively. The baseline performance data is generated based on the average time taken for each stage.

4. The multi-model parallel performance evaluation and cost optimization method according to claim 3, characterized in that, The establishment of performance degradation records includes: The request processing process is divided into a preprocessing stage and an inference computation stage. The actual stage time of each model when processing different types of requests under combined deployment is collected. Calculate the percentage change in actual stage time relative to the corresponding stage time in the baseline performance data; When the change in time consumption during the preprocessing stage of any model in the same model combination is greater than the change in time consumption during the inference computation stage, the source of the time increase under that combination is marked as an increase in time consumption during the preprocessing stage; when the change in time consumption during the inference computation stage is greater than the change in time consumption during the preprocessing stage, the source of the time increase under that combination is marked as an increase in time consumption during the inference computation stage; the source of the time increase and the change in time consumption during each stage are associated and stored in the performance degradation record.

5. The multi-model parallel performance evaluation and cost optimization method according to claim 4, characterized in that, Multiple candidate deployment strategies are generated, including: Based on the length of the input data and the length of the output data, requests are divided into three types: short input long output, long input short output, and balanced requests. Predict the distribution ratio of request types for each model in the next time period, and determine the type with the most requests for each model in the next time period; From the performance degradation records, find the sources of increased time consumption and the percentage change in time consumption at each stage for each model combination under this type; The model combination with the smallest change in stage time under this type is determined as the candidate deployment combination; Based on the request arrival rate predicted by each model and the phase time change ratio under the candidate deployment combination, the resource quota required to meet the preset processing time requirements is calculated, and the candidate deployment strategy is obtained.

6. The multi-model parallel performance evaluation and cost optimization method according to claim 5, characterized in that, The process of determining the target deployment strategy from the Pareto frontier solution set based on current business priorities includes: When the current business priority is performance, select the candidate deployment strategy from the Pareto front solution set that has the closest processing time to the preset processing time requirement and the relatively high total resource cost as the target deployment strategy. When the current business priority is cost priority, the candidate deployment strategy with the lowest total resource cost and processing time not exceeding the preset processing time requirement is selected from the Pareto frontier solution set as the target deployment strategy. When the current business priority is balanced, the candidate deployment strategy with the best weighted comprehensive score of processing time and total resource cost is selected from the Pareto frontier solution set as the target deployment strategy.

7. The multi-model parallel performance evaluation and cost optimization method according to claim 6, characterized in that, The adjustment of resource allocation and deployment location for each model according to the target deployment strategy includes: When the target resource quota of any model is greater than the current resource quota, if the remaining resources of the physical node where the model is located meet the target resource quota, the model is directly expanded; if the remaining resources are insufficient, the physical node where the model replica has been deployed and the proportion of the model's phase time change on the physical node where the replica is located is less than the preset change threshold is selected first, the newly arrived requests are routed to the replica and the instances to be expanded are simultaneously scheduled to the physical node. When the target resource quota of any model is less than the current resource quota, the excess resources of that model on the current physical node are released, and the released resources are allocated to other models on the same physical node whose target resource quota is greater than the current resource quota and whose stage time change ratio is less than the preset change threshold.

8. The multi-model parallel performance evaluation and cost optimization method according to claim 7, characterized in that, The self-repair action includes: Record the front-end queuing time of each model on the current physical node. The front-end queuing time is the time interval between the arrival of the request and the start of preprocessing. When the front-end queuing time of any model exceeds the preset queuing threshold, obtain the list of co-located models on the physical node where that model is located; From the performance degradation records, query the percentage change in stage consumption time corresponding to the combination of this model and each model in the co-located model list; If a co-located model exists and the change ratio of its stage time exceeds a preset change threshold, then the co-located model will be migrated from the current physical node to a physical node where the change ratio of its stage time is less than the preset change threshold, while keeping the current resource quota of the model unchanged.

9. The multi-model parallel performance evaluation and cost optimization method according to claim 1, characterized in that, The total resource cost is calculated based on at least one of the following technical indicators: The product of CPU core count and corresponding unit price coefficient, GPU memory capacity and corresponding unit price coefficient, network bandwidth and corresponding unit price coefficient, and memory capacity and corresponding unit price coefficient are all calculated. Each unit price coefficient is dynamically obtained from the real-time pricing interface of the current cloud computing platform.

10. A multi-model parallel performance evaluation and cost optimization system, characterized in that, include: The Acquisition and Statistics module is used to acquire request processing records for multiple models within a historical time period. Based on the request processing records, the baseline performance data of each model under exclusive resource conditions are statistically analyzed; The deployment and data acquisition module is used to deploy multiple models in different combinations on the same physical node, collect the actual performance data of each model under the combined deployment, calculate the performance degradation ratio relative to the benchmark performance data, and establish a performance degradation record. The prediction and generation module is used to predict the request arrival rate of each model in the next time period; Based on the performance degradation records and the predicted request arrival rate, with the goal of meeting the preset processing time requirements of each model and minimizing the total resource cost, multiple candidate deployment strategies are generated, and a Pareto frontier solution set is selected. The determination module is used to determine the target deployment strategy from the Pareto frontier scheme set based on the current business priority. The target deployment strategy includes the target resource quota for each model in the next time period and the model deployment combination. The adjustment and optimization module is used to adjust the resource allocation and deployment location of each model according to the target deployment strategy. In addition, during the adjusted operation, the actual processing time of each model is monitored, and when the deviation between the actual processing time and the preset processing time requirement exceeds the preset threshold, a self-repair action is executed.

Images