Large model concurrent resource management method and device, equipment and storage medium
By acquiring business and model resource information, and combining it with performance baseline data to calculate the resources required for large models, and dynamically adjusting resource allocation, the problem that traditional resource management models cannot adapt to large models is solved, achieving accurate resource matching and improved business performance stability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 太保科技有限公司
- Filing Date
- 2026-01-04
- Publication Date
- 2026-04-14
AI Technical Summary
Traditional AI model resource management methods are difficult to adapt to the non-linear performance impact of large models, resulting in low resource utilization and increased response time, which affects business stability, especially in the financial and insurance industry.
By acquiring business information and model resource information when large models concurrently request resources, and combining this with performance baseline data, the amount of computing resources required by the business is calculated, and resource usage is monitored in real time to dynamically adjust resource allocation to match business needs.
It achieves precise matching of resource allocation, avoids resource waste and increased response time, and improves business performance stability and resource utilization.
Smart Images
Figure CN121858293A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data management technology, and in particular to a method, apparatus, device and storage medium for managing concurrent resources of large models. Background Technology
[0002] With the rapid development of artificial intelligence technology, large-scale models are increasingly widely used in fields such as natural language processing and intelligent decision-making. Especially in the fintech sector, where business scenarios are complex, such as finance and insurance, large-scale models, with their powerful semantic understanding and reasoning capabilities, provide efficient support for core businesses such as intelligent customer service, policy review, and risk control. In the practical application of large-scale models, interface concurrency resource management is a crucial aspect of ensuring stable business operation. Traditional AI models, due to their short response times and relatively stable performance, typically employ the following resource management model: first, determine the maximum concurrency the model can support through performance stress testing; then, adjust the number of authorized resources based on the actual peak concurrency of the business. When model resources are insufficient, simply horizontally scaling up the number of model instances can meet the needs of more business operations.
[0003] However, large-scale models differ significantly from traditional AI models. Their performance is affected non-linearly by various factors such as the number of input tokens, the number of output tokens, and concurrency. The resource management model of traditional AI models is difficult to adapt to the application needs of large-scale models, gradually revealing many shortcomings: On the one hand, adjusting the number of authorizations solely based on the model interface concurrency limit and the actual peak concurrency of the business cannot accurately match the differentiated performance requirements of large-scale models in different business scenarios, resulting in resource utilization failing to meet expectations and a large amount of computing resources remaining idle; on the other hand, after adjusting the number of concurrent authorizations, the model interface return time is prone to increase, seriously affecting the normal use of the business. This is particularly prominent in the financial and insurance industry, which has extremely high requirements for response timeliness and stability. Summary of the Invention
[0004] To address the aforementioned problems, this application provides a method, apparatus, device, and storage medium for managing large-scale concurrent resources, including the following:
[0005] Firstly, this application provides a method for managing concurrent resources in a large-scale model, the method comprising:
[0006] Obtain business application information and target model resource information when requesting concurrent resources for large models; the target model resource information includes the number of idle cards for the model, the number of computing cards occupied by a single model instance, and performance baseline data. The performance baseline data is obtained by pre-collecting performance data of a single instance large model service under different input and output parameters and concurrent scenarios.
[0007] The amount of computing resources required for the service is calculated based on the service application information, the number of computing cards occupied by the single model instance, and the performance baseline data.
[0008] Compare the number of idle cards in the target model resource information with the amount of computing resources required by the service;
[0009] If the number of idle cards in the model is greater than the amount of computing resources required by the service, then the corresponding concurrent resources are allocated to the service, and the number of idle cards in the model is updated.
[0010] Optionally, the business application information includes:
[0011] Input token count, output token count, first token response time threshold, and concurrent requests required by business needs.
[0012] Optionally, the pre-collection of performance data for the single-instance large model service under different input / output parameters and concurrency scenarios includes:
[0013] Control the single-instance large model service to run test tasks with different concurrency levels under multiple preset combinations of input token counts and output token counts;
[0014] Collect data on the first token response time, complete response latency, token generation speed, and QPS metrics for each test task.
[0015] The maximum concurrency that meets the preset first token response time threshold under each combination of input and output parameters is determined as the concurrency support number for the corresponding scenario.
[0016] Optionally, calculating the amount of computing resources required for the service based on the service application information, the number of computing cards occupied by a single model instance, and the performance baseline data includes:
[0017] From the performance baseline data, match the concurrent support number corresponding to the number of input tokens, number of output tokens, and first token response time threshold in the business application information;
[0018] The required computing resources BRN for the business can be calculated using the formula BRN = (Number of concurrent business requests / Number of concurrent requests supported) × Number of computing cards required by a single model instance.
[0019] Optionally, the method further includes:
[0020] Real-time monitoring of computing card occupancy and idle card count changes at the model level, as well as actual concurrent usage and response time data at the business level, generates resource usage monitoring data;
[0021] When the resource usage monitoring data reaches a preset alarm threshold, an alarm notification is triggered.
[0022] Optionally, the method further includes:
[0023] Based on the resource usage monitoring data, determine whether resource scaling is necessary.
[0024] If expansion is required, obtain the business supplementary application information corresponding to the expansion, recalculate the amount of computing resources required for expansion by combining the target model resource information and the performance baseline data, compare the updated number of idle cards in the model with the amount of computing resources required for expansion, and if the number of idle cards in the model meets the amount of computing resources required for expansion, allocate the corresponding expansion resources and update the number of idle cards in the model.
[0025] If scaling down is required, calculate the number of concurrent connections that can be scaled down and the corresponding number of computing cards that can be released, and update the number of idle cards in the model.
[0026] Optionally, the reduction in volume includes:
[0027] Get the allocated concurrent users for the business and the actual concurrent users used by the business within the historical preset time period;
[0028] The scalable concurrency is calculated using the formula: scalable concurrency = allocated concurrency minus twice the actual concurrency used by the business. The scalable concurrency must not be less than zero.
[0029] From the performance baseline data, match the concurrent support number corresponding to the number of input tokens, number of output tokens, and first token response time threshold in the business application information;
[0030] The number of computing cards that can be released, RLN, is calculated using the formula RLN = (scalable concurrent users / concurrent users supported) × number of computing cards occupied by a single model instance.
[0031] Update the number of idle cards in the model to the current number of idle cards in the model plus RLN, and update the number of concurrent services allocated to the service to the number of concurrent services allocated to the service minus the number of concurrent services that can be scalable.
[0032] Secondly, this application provides a large-scale concurrent resource management device, which includes:
[0033] The acquisition unit is used to acquire business application information and target model resource information when applying for concurrent resources for large models. The target model resource information includes the number of idle cards in the model, the number of computing cards occupied by a single model instance, and performance baseline data. The performance baseline data is obtained by pre-collecting performance data of a single instance large model service under different input and output parameters and concurrent scenarios.
[0034] The computing unit is used to calculate the amount of computing resources required by the service based on the service application information, the number of computing cards occupied by the single model instance, and the performance baseline data.
[0035] The data comparison unit is used to compare the number of idle cards in the target model resource information with the amount of computing resources required by the service.
[0036] The data adjustment unit is used to allocate corresponding concurrent resources to the service and update the number of idle cards in the model if the number of idle cards in the model is greater than the number of computing resources required by the service.
[0037] Optionally, the business application information includes:
[0038] Input token count, output token count, first token response time threshold, and concurrent requests required by business needs.
[0039] Optionally, the pre-collection of performance data for the single-instance large model service under different input / output parameters and concurrency scenarios includes:
[0040] Control the single-instance large model service to run test tasks with different concurrency levels under multiple preset combinations of input token counts and output token counts;
[0041] Collect data on the first token response time, complete response latency, token generation speed, and QPS metrics for each test task.
[0042] The maximum concurrency that meets the preset first token response time threshold under each combination of input and output parameters is determined as the concurrency support number for the corresponding scenario.
[0043] Optionally, the computing unit is specifically used to match the number of concurrent connections from the performance baseline data with the number of input tokens, the number of output tokens, and the first token response time threshold in the business application information;
[0044] The required computing resources BRN for the business can be calculated using the formula BRN = (Number of concurrent business requests / Number of concurrent requests supported) × Number of computing cards required by a single model instance.
[0045] Optionally, the device further includes a monitoring unit for real-time monitoring of changes in computing card occupancy and idle card count at the model dimension, as well as actual concurrent usage and response time data at the business dimension, and generating resource usage monitoring data.
[0046] When the resource usage monitoring data reaches a preset alarm threshold, an alarm notification is triggered.
[0047] Optionally, the monitoring unit is further configured to determine whether resource scaling is required based on the resource usage monitoring data;
[0048] If expansion is required, obtain the business supplementary application information corresponding to the expansion, recalculate the amount of computing resources required for expansion by combining the target model resource information and the performance baseline data, compare the updated number of idle cards in the model with the amount of computing resources required for expansion, and if the number of idle cards in the model meets the amount of computing resources required for expansion, allocate the corresponding expansion resources and update the number of idle cards in the model.
[0049] If scaling down is required, calculate the number of concurrent connections that can be scaled down and the corresponding number of computing cards that can be released, and update the number of idle cards in the model.
[0050] Optionally, scaling down the monitoring unit includes:
[0051] Get the allocated concurrent users for the business and the actual concurrent users used by the business within the historical preset time period;
[0052] The scalable concurrency is calculated using the formula: scalable concurrency = allocated concurrency minus twice the actual concurrency used by the business. The scalable concurrency must not be less than zero.
[0053] From the performance baseline data, match the concurrent support number corresponding to the number of input tokens, number of output tokens, and first token response time threshold in the business application information;
[0054] The number of computing cards that can be released, RLN, is calculated using the formula RLN = (scalable concurrent users / concurrent users supported) × number of computing cards occupied by a single model instance.
[0055] Update the number of idle cards in the model to the current number of idle cards in the model plus RLN, and update the number of concurrent services allocated to the service to the number of concurrent services allocated to the service minus the number of concurrent services that can be scalable.
[0056] Thirdly, this application provides an apparatus comprising a memory and a processor, the memory for storing instructions or code, and the processor for executing the instructions or code to cause the apparatus to perform the large-scale concurrent resource management method described in any implementation of the first aspect.
[0057] Fourthly, this application provides a computer-readable storage medium storing code, wherein when the code is executed, a device running the code implements the large-scale concurrent resource management method described in any of the implementations of the first aspect.
[0058] This application provides a method for managing concurrent resources of large models. When executing the method, firstly, business application information and target model resource information are obtained when requesting concurrent resources for large models. The target model resource information includes the number of idle computing cards for the model, the number of computing cards occupied by a single model instance, and performance baseline data. The performance baseline data is obtained by pre-collecting performance data of a single-instance large model service under different input / output parameters and concurrent scenarios. Then, based on the business application information, the number of computing cards occupied by a single model instance, and the performance baseline data, the required computing resources for the business are calculated. Next, the number of idle computing cards in the target model resource information is compared with the required computing resources for the business. Finally, if the number of idle computing cards is greater than the required computing resources for the business, corresponding concurrent resources are allocated to the business, and the number of idle computing cards is updated. In this way, by accurately calculating computing resource requirements based on personalized needs such as the number of business input / output tokens and the response time of the first token, combined with performance baseline data, resource allocation can precisely match actual business needs, avoiding resource waste caused by coarse authorization based on fixed concurrency limits. Simultaneously, allocation is based on a quantitative comparison of the number of idle cards in the model and computing power requirements, ensuring that the allocated resources meet the business's preset response time requirements. This avoids the problem of increased interface return time after adjusting concurrency authorization, achieving a balance between resource utilization and business performance stability. Thus, a more adaptable resource management solution can be provided for industries with complex business scenarios such as finance and insurance, enabling dynamic optimization and allocation of resources. Attached Figure Description
[0059] To more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0060] Figure 1 A flowchart illustrating a large-scale concurrent resource management method provided in this application embodiment;
[0061] Figure 2 A model performance data table provided for embodiments of this application;
[0062] Figure 3 This is a schematic diagram of the structure of a large-scale concurrent resource management device provided in an embodiment of this application. Detailed Implementation
[0063] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0064] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.
[0065] Figure 1 A flowchart illustrating a large-scale concurrent resource management method provided in an embodiment of this application. (Combined with...) Figure 1 As shown, the large-scale concurrent resource management method provided in this application embodiment may include:
[0066] S101. Obtain business application information and target model resource information when applying for concurrent resources for a large model.
[0067] Business Request Information (BP) is crucial information submitted by the business party when requesting concurrent resources for a large model. It includes the number of input tokens, the number of output tokens, the first token response time threshold, and the required concurrent requests. A token is the basic unit for the large model to process text data; one token can be understood as a word or a character segment. The number of input tokens is the total number of tokens corresponding to the input text in the business party's request. The number of output tokens is the total number of tokens the business party expects the large model to return in the results. The first token response time threshold is the maximum time required for the large model to return the first token. The required concurrent requests are the number of requests the business party needs to initiate simultaneously.
[0068] Target Model Resource Information (LLMP) is the current resource status data of a large model, including the number of idle computing cards, the number of computing cards required for a single model instance, and performance baseline data. The number of idle computing cards refers to the number of unused computing cards for the current large model. The number of computing cards required for a single model instance is the number of computing cards needed to run a large model instance. The performance baseline data is obtained by pre-collecting performance data of a single-instance large model service under different input / output parameters and concurrency scenarios. Specifically, this includes performance data such as the model's first token response time, token generation speed, and QPS (queries per second) obtained from stress testing of the single-instance large model service across different input, output, and concurrency dimensions. This data is crucial for determining the resources required and allocated for subsequent computing operations.
[0069] The specific process for pre-collecting performance data of a single-instance large model service under different input / output parameters and concurrency scenarios is as follows: First, deploy the single-instance large model service to ensure that no other programs call the service during load testing to avoid interfering with the test results; then, control the single-instance large model service to run test tasks with different concurrency levels under preset combinations of multiple input token counts and output token counts. In this embodiment, the preset input / output combinations are input-output 2K-128 and 2K-2K (where 2K represents 2000 tokens), and the preset concurrency levels include 1, 8, 16, 24, 32, 40, 48, and 56. For tests 64, 72, 80, 88, and 100, streaming requests were used to stress test the model's interface performance. During the stress test, data such as the first token latency (i.e., the first token response time), the complete response latency (i.e., the time from receiving a request to returning a complete result), the token generation speed (the number of tokens generated per unit time), throughput, and QPS were collected for each test task. Finally, the maximum concurrency that meets the preset first token response time threshold under each combination of input and output parameters was determined as the concurrency support for the corresponding scenario. These collected and determined data together constitute the model's performance baseline data.
[0070] Figure 2This application provides a model performance data table, which includes performance data for two scenarios: input-output 2K-2K and input-output 2K-128. The meanings of each field in the table are as follows: "params" represents the parameter item, "input_tokens" represents the number of input tokens, "output_tokens" represents the number of output tokens, "max" represents the maximum value of the corresponding indicator, "min" represents the minimum value of the corresponding indicator, "mean" represents the average value of the corresponding indicator, "p99" represents the 99th percentile of the corresponding indicator (i.e., 99% of the test data does not exceed this value), "p90" represents the 90th percentile of the corresponding indicator (i.e., 90% of the test data does not exceed this value), and "pool_num" represents the concurrency. "react_p90" (in the 2K-2K scenario) and "react_p90 first token" (in the 2K-128 scenario) both represent the 90th percentile of the first token response time. "e2e_p90" (in the 2K-2K scenario) and "e2e_p90 complete response" (in the 2K-128 scenario) both represent the 90th percentile of the complete response latency. "avg_speed" (in the 2K-2K scenario) and "avg_gen_speed" (in the 2K-128 scenario) both represent the average token generation speed. "avg_throughput_output" (in the 2K-2K scenario) and "avg_throughput" (in the 2K-128 scenario) both represent the average throughput.
[0071] by Figure 2 Taking the 2K-128 input-output scenario as an example, if the preset threshold for the first token response time is 10, then from the table we can see that: when the concurrency (pool_num) is 32, the 90th percentile of the first token response time (react_p90 first token) is 8.9309, which meets the threshold; when the concurrency is 40, the 90th percentile of the first token response time is 10.1928, which exceeds the threshold; therefore, in the 2K-128 input-output scenario, the maximum concurrency that meets the first token response time threshold of 10 is 32, that is, the concurrency supported by this scenario is 32.
[0072] S102. Calculate the amount of computing resources required for the service based on the service application information, the number of computing cards occupied by the single model instance, and the performance baseline data.
[0073] Business application information comprises all relevant data submitted by the business party when requesting resources. The required computing power resources for the business are calculated based on the business application information using specific rules. The core of calculating the required computing power resources (BRN) for a business is to first match the concurrent support number corresponding to the business requirements, and then calculate the required number of computing cards using a formula. The specific steps are as follows:
[0074] First, from the performance baseline data, find the baseline data that is completely consistent with the number of input tokens and output tokens in the Business Request Information (BP), and closest to the first token response time threshold (BP [First Token Response Time]). The concurrency corresponding to this baseline data is the number of concurrent connections supported by a single instance model to meet the current business's first token response time requirement (LLMP [Model Performance Baseline]). For example, combined with... Figure 2 Performance baseline data for a 2K-128 input-output scenario, where the number of input tokens in the business application information is 2000, the number of output tokens is 128, and the first token response time threshold is 10, from Figure 2 The data shows that when the concurrency (pool_num) is 32, the 90th percentile of the first token response time (react_p90 first token) is 8.9309, which meets the threshold. When the concurrency is 40, the 90th percentile of the first token response time is 10.1928, which exceeds the threshold. Therefore, the corresponding number of concurrent connections supported (LLMP [model performance baseline]) in this scenario is 32. Then, using the formula "Number of computing resources required by the business = (Number of concurrent connections required by the business / Number of concurrent connections supported) × Number of computing cards occupied by a single model instance," the number of computing resources required by the business is calculated. This value is identified as the number of computing resources required by the business.
[0075] The above process can be applied to the allocation of new business resources. Specifically, first, based on the first token response time in the business application information, the closest baseline data is found in the model performance baseline data. Here, the baseline data is the number of concurrent connections supported by a single instance model to meet the first token response time of the business. Second, based on the business requirement concurrency in the business application information, combined with the number of computing power cards occupied by the single instance model, the number of computing power cards required by the business is calculated using the above formula. Finally, it is determined whether the current resource reserve of the model meets the business requirements. If the number of idle cards in the model is greater than the number of computing power resources required by the business, it indicates that the resources are sufficient. If the number of idle cards in the model is less than or equal to the number of computing power resources required by the business, it indicates that the resources are insufficient.
[0076] S103. Compare the number of idle cards in the target model resource information with the amount of computing resources required by the service.
[0077] After calculating the required computing resources (BRN) for the business, it is necessary to compare the number of idle cards in the target model's resource information with the BRN to determine whether the current large model's idle resources can meet the business's resource requirements. This step is a crucial judgment in resource allocation; through direct numerical comparison, the resource supply and demand relationship can be quickly determined.
[0078] S104. If the number of idle cards in the model is greater than the amount of computing resources required by the service, then allocate corresponding concurrent resources to the service and update the number of idle cards in the model.
[0079] After comparison in S103, if the number of idle GPUs in the model is greater than the amount of computing resources (BRN) required by the service, it indicates that the current large model has sufficient idle resources to support the concurrent requests of the service. In this case, the service's concurrent resource request is approved, and the corresponding concurrent resources are allocated to the service. Simultaneously, to ensure the accuracy of subsequent resource statistics, the number of idle GPUs in the model needs to be updated. The update method is to subtract the amount of computing resources (BRN) required by the service from the original number of idle GPUs in the model. The updated number of idle GPUs in the model will be used as the basis for judging resource requests from other services in the future. If the number of idle GPUs in the model is less than or equal to BRN, it indicates that the current large model resources are insufficient to meet the needs of the service, and concurrent resources will not be allocated.
[0080] Taking the large model QWen2.5-72B as an example, combined with the above calculation logic and resource allocation rules, the specific execution process is as follows: The business application information is 2000 input tokens, 100 output tokens, a first token response time threshold of 2.5 seconds, and a business requirement of 5 concurrent requests; the target model resource information is 16 idle cards, 8 computing cards occupied by a single model instance, and pre-collected performance baseline data. First, from the performance baseline data, a baseline data group that perfectly matches the 2000 input tokens and 100 output tokens is selected. The maximum concurrency in this group of data that meets the first token response time threshold of 2.5 seconds is found, and the corresponding concurrent support number for this business is determined to be 8. Then, the amount of computing resources required for the business is calculated according to the formula. The amount of computing resources required for the business is equal to the business requirement of concurrent requests divided by the number of concurrent requests supported, and then multiplied by the number of computing cards occupied by a single model instance. Substituting the values, the amount of computing resources required for the business is (5 / 8) × 8 = 5. Next, the number of idle cards in the target model resource information (16) is compared with the calculated amount of computing resources required by the business (5). The comparison result shows that 16 is greater than 5, indicating that the idle model resources can meet the business needs, and the resource application is approved. Finally, the number of idle cards is updated by subtracting the amount of computing resources required by the business from the number of idle cards. After the update, the number of idle cards is 16 minus 5, which equals 11.
[0081] In one implementation of this application, the method further includes: real-time monitoring of changes in computing card occupancy and idle card count at the model dimension, as well as actual concurrent usage and response time data at the business dimension, to generate resource usage monitoring data.
[0082] Specific monitoring methods include concurrent monitoring logs and resource monitoring dashboards: Concurrency monitoring logs refer to the real-time recording of the overall concurrency of the model and the actual concurrency of each business unit through Redis (a type of caching database) when the model interface accepts a request. After the request is processed, the concurrency of the corresponding model dimension and business dimension at the time of the request processing is also recorded to ensure the integrity of the monitoring data; the resource monitoring dashboard is a dedicated real-time monitoring interface that intuitively displays the computing power card usage and idle status of the model dimension and the concurrent resource usage of the business dimension, making it convenient for managers to grasp the resource dynamics in real time.
[0083] An alarm is triggered when resource usage monitoring data reaches a preset alarm threshold. The preset alarm threshold can be set according to the actual business scenario and resource configuration. For example, the alarm threshold could be set as 80% of the model's supported concurrency limit or 80% of the business's allocated concurrency limit. When monitored data reaches this threshold, the alarm mechanism is automatically triggered, sending a reminder message to relevant administrators via SMS or email. This allows administrators to be promptly informed of resource constraints and determine whether model-level expansion or business-level resource reallocation is necessary.
[0084] In one implementation of this application, the method further includes:
[0085] Determining whether resource scaling is necessary based on resource usage monitoring data: When the monitoring shows that the actual concurrent resources used by a business exceed the amount of resources allocated to that business, it indicates that the current resource allocation for that business is insufficient and needs to be scaled up; when the monitoring shows that the concurrent resource utilization rate of the allocated business model is less than 50%, it indicates that the resources for that business are idle and need to be scaled down.
[0086] If expansion is required, obtain the corresponding supplementary application information for the business expansion. This information should include key details such as the number of input tokens, output tokens, first token response time threshold, and the number of concurrent requests required for the business after expansion. Then, combining the updated target model resource information and performance baseline data, recalculate the required computing resources for expansion according to the logic of allocating new business resources. The specific calculation process is consistent with S102: first, match the corresponding concurrent support number from the performance baseline data, then calculate the required computing resources for expansion using the formula BRN = (concurrent requests required for expanded business / concurrent support number) × number of computing cards occupied by a single model instance. Next, compare the updated number of idle model cards with the required computing resources for expansion. If the number of idle model cards is sufficient to meet the required computing resources, allocate the corresponding expansion resources for the business and update the number of idle model cards by subtracting the required computing resources. If the number of idle model cards is insufficient to meet the expansion requirements, expansion cannot be performed, and processing must wait until sufficient idle resources become available.
[0087] If scaling down is required, calculate the maximum concurrent connections that can be scaled down and the corresponding number of computing cards that can be released, and update the number of idle cards in the model. The specific scaling down process is as follows:
[0088] Get the current allocated concurrent users for the service and the actual concurrent users used by the service within a preset historical period. The preset historical period can be set according to the usage pattern of the service, such as 1 day, 3 days, etc., to ensure that the actual concurrent users obtained are of reference value.
[0089] The formula for scalable concurrency is: Scalable concurrency = Allocated concurrency - 2 × Actual concurrency used by the business. It should be noted that the calculated scalable concurrency must not be less than zero. If the result is negative, take zero as the scalable concurrency to avoid excessive scaling down affecting normal business use.
[0090] From the performance baseline data, match the number of concurrent connections corresponding to the number of input tokens, the number of output tokens, and the first token response time threshold in the business application information. This number of concurrent connections is consistent with the number of concurrent connections used during the expansion calculation.
[0091] The number of releasable computing cards RLN is calculated using the formula RLN = (scalable concurrent users / concurrent users supported) × number of computing cards occupied by a single model instance, where RLN is the total number of computing cards that can be released after the service is scaled down.
[0092] Update relevant data: Update the number of idle cards in the model to the current number of idle cards in the model plus RLN, and at the same time update the allocated concurrency of the service to the allocated concurrency of the service minus the scalable concurrency, complete the resource scaling-down operation, and release idle resources for use by other services.
[0093] In one implementation of this application, the scaling down includes:
[0094] Get the allocated concurrent users for the business and the actual concurrent users used by the business within the historical preset time period;
[0095] The scalable concurrency is calculated using the formula: scalable concurrency = allocated concurrency minus twice the actual concurrency used by the business. The scalable concurrency must not be less than zero.
[0096] From the performance baseline data, match the concurrent support number corresponding to the number of input tokens, number of output tokens, and first token response time threshold in the business application information;
[0097] The number of computing cards that can be released, RLN, is calculated using the formula RLN = (scalable concurrent users / concurrent users supported) × number of computing cards occupied by a single model instance.
[0098] Update the number of idle cards in the model to the current number of idle cards in the model plus RLN, and update the number of concurrent services allocated to the service to the number of concurrent services allocated to the service minus the number of concurrent services that can be scalable.
[0099] The above are some specific implementations of a large-scale concurrent resource management method provided in the embodiments of this application. Based on this, this application also provides a corresponding device. The device provided in the embodiments of this application will be described below from the perspective of functional modularization.
[0100] Figure 3 This is a schematic diagram of a large-scale concurrent resource management device provided in an embodiment of this application. (Combined with...) Figure 3 As shown in the embodiment of this application, the large-scale concurrent resource management device 300 includes:
[0101] The acquisition unit 310 is used to acquire business application information and target model resource information when applying for concurrent resources of large models; the target model resource information includes the number of idle cards of the model, the number of computing power cards occupied by a single model instance and performance baseline data, the performance baseline data is obtained by pre-collecting the performance data of a single instance large model service under different input and output parameters and concurrent scenarios;
[0102] The calculation unit 320 is used to calculate the amount of computing resources required by the service based on the service application information, the number of computing cards occupied by the single model instance, and the performance baseline data.
[0103] Data comparison unit 330 is used to compare the number of idle cards in the target model resource information with the amount of computing resources required by the service;
[0104] The data adjustment unit 340 is used to allocate corresponding concurrent resources to the service and update the number of idle cards in the model if the number of idle cards in the model is greater than the number of computing resources required by the service.
[0105] In one implementation of this application, the business application information includes:
[0106] Input token count, output token count, first token response time threshold, and concurrent requests required by business needs.
[0107] In one implementation of this application, the pre-collection of performance data for a single-instance large model service under different input / output parameters and concurrency scenarios includes:
[0108] Control the single-instance large model service to run test tasks with different concurrency levels under multiple preset combinations of input token counts and output token counts;
[0109] Collect data on the first token response time, complete response latency, token generation speed, and QPS metrics for each test task.
[0110] The maximum concurrency that meets the preset first token response time threshold under each combination of input and output parameters is determined as the concurrency support number for the corresponding scenario.
[0111] In one implementation of this application, the computing unit is specifically used to match the number of concurrent connections corresponding to the number of input tokens, the number of output tokens, and the first token response time threshold in the business application information from the performance baseline data;
[0112] The required computing resources BRN for the business can be calculated using the formula BRN = (Number of concurrent business requests / Number of concurrent requests supported) × Number of computing cards required by a single model instance.
[0113] In one implementation of this application, the device further includes a monitoring unit, which is used to monitor in real time the changes in computing card occupancy rate and number of idle cards in the model dimension, as well as the actual concurrent usage and response time data in the business dimension, and generate resource usage monitoring data.
[0114] When the resource usage monitoring data reaches a preset alarm threshold, an alarm notification is triggered.
[0115] In one implementation of this application, the monitoring unit is further configured to determine whether resource scaling is required based on the resource usage monitoring data.
[0116] If expansion is required, obtain the business supplementary application information corresponding to the expansion, recalculate the amount of computing resources required for expansion by combining the target model resource information and the performance baseline data, compare the updated number of idle cards in the model with the amount of computing resources required for expansion, and if the number of idle cards in the model meets the amount of computing resources required for expansion, allocate the corresponding expansion resources and update the number of idle cards in the model.
[0117] If scaling down is required, calculate the number of concurrent connections that can be scaled down and the corresponding number of computing cards that can be released, and update the number of idle cards in the model.
[0118] In one implementation of this application embodiment, the scaling down of the monitoring unit includes:
[0119] Get the allocated concurrent users for the business and the actual concurrent users used by the business within the historical preset time period;
[0120] The scalable concurrency is calculated using the formula: scalable concurrency = allocated concurrency minus twice the actual concurrency used by the business. The scalable concurrency must not be less than zero.
[0121] From the performance baseline data, match the concurrent support number corresponding to the number of input tokens, number of output tokens, and first token response time threshold in the business application information;
[0122] The number of computing cards that can be released, RLN, is calculated using the formula RLN = (scalable concurrent users / concurrent users supported) × number of computing cards occupied by a single model instance.
[0123] Update the number of idle cards in the model to the current number of idle cards in the model plus RLN, and update the number of concurrent services allocated to the service to the number of concurrent services allocated to the service minus the number of concurrent services that can be scalable.
[0124] This application also provides corresponding devices and computer storage media for implementing the solutions provided in this application.
[0125] The device includes a memory and a processor. The memory stores instructions or code, and the processor executes the instructions or code to cause the device to perform the method described in any embodiment of this application.
[0126] The computer storage medium stores code, and when the code is run, the device running the code implements the method described in any embodiment of this application.
[0127] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus a general-purpose hardware platform. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or some parts of the embodiments of this application.
[0128] It is understood that in the specific embodiments of this application, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved need to obtain user permission or consent when the above embodiments of this application are applied to specific products or technologies, and the collection, use and processing of related data need to comply with the relevant laws, regulations and standards of relevant countries and regions.
[0129] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0130] It should also be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and apparatus embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0131] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for managing concurrent resources in a large model, characterized in that, The method includes: Obtain business application information and target model resource information when requesting concurrent resources for large models; the target model resource information includes the number of idle cards for the model, the number of computing cards occupied by a single model instance, and performance baseline data. The performance baseline data is obtained by pre-collecting performance data of a single instance large model service under different input and output parameters and concurrent scenarios. The amount of computing resources required for the service is calculated based on the service application information, the number of computing cards occupied by the single model instance, and the performance baseline data. Compare the number of idle cards in the target model resource information with the amount of computing resources required by the service; If the number of idle cards in the model is greater than the amount of computing resources required by the service, then the corresponding concurrent resources are allocated to the service, and the number of idle cards in the model is updated.
2. The method according to claim 1, characterized in that, The business application information includes: Input token count, output token count, first token response time threshold, and concurrent requests required by business needs.
3. The method according to claim 1, characterized in that, The pre-collected performance data of the single-instance large model service under different input / output parameters and concurrency scenarios includes: Control the single-instance large model service to run test tasks with different concurrency levels under multiple preset combinations of input token counts and output token counts; Collect data on the first token response time, complete response latency, token generation speed, and QPS metrics for each test task. The maximum concurrency that meets the preset first token response time threshold under each combination of input and output parameters is determined as the concurrency support number for the corresponding scenario.
4. The method according to claim 1, characterized in that, The calculation of the computing resources required for the service based on the service application information, the number of computing cards occupied by a single model instance, and the performance baseline data includes: From the performance baseline data, match the concurrent support number corresponding to the number of input tokens, number of output tokens, and first token response time threshold in the business application information; The required computing resources BRN for the business can be calculated using the formula BRN = (Number of concurrent business requests / Number of concurrent requests supported) × Number of computing cards required by a single model instance.
5. The method according to claim 1, characterized in that, The method further includes: Real-time monitoring of changes in computing card occupancy and idle card count at the model level, as well as actual concurrent usage and response time data at the business level, generates resource usage monitoring data; When the resource usage monitoring data reaches a preset alarm threshold, an alarm notification is triggered.
6. The method according to claim 5, characterized in that, The method further includes: Based on the resource usage monitoring data, determine whether resource scaling is necessary. If expansion is required, obtain the business supplementary application information corresponding to the expansion, recalculate the amount of computing resources required for expansion by combining the target model resource information and the performance baseline data, compare the updated number of idle cards in the model with the amount of computing resources required for expansion, and if the number of idle cards in the model meets the amount of computing resources required for expansion, allocate the corresponding expansion resources and update the number of idle cards in the model. If scaling down is required, calculate the number of concurrent connections that can be scaled down and the corresponding number of computing cards that can be released, and update the number of idle cards in the model.
7. The method according to claim 6, characterized in that, The reduction includes: Get the allocated concurrent users for the business and the actual concurrent users used by the business within the historical preset time period; The scalable concurrency is calculated using the formula: scalable concurrency = allocated concurrency minus twice the actual concurrency used by the business. The scalable concurrency must not be less than zero. From the performance baseline data, match the concurrent support number corresponding to the number of input tokens, number of output tokens, and first token response time threshold in the business application information; The number of computing cards that can be released, RLN, is calculated using the formula RLN = (scalable concurrent users / concurrent users supported) × number of computing cards occupied by a single model instance. Update the number of idle cards in the model to the current number of idle cards in the model plus RLN, and update the number of concurrent services allocated to the service to the number of concurrent services allocated to the service minus the number of concurrent services that can be scalable.
8. A large-scale concurrent resource management device, characterized in that, The device includes: The acquisition unit is used to acquire business application information and target model resource information when applying for concurrent resources for large models. The target model resource information includes the number of idle cards in the model, the number of computing cards occupied by a single model instance, and performance baseline data. The performance baseline data is obtained by pre-collecting performance data of a single instance large model service under different input and output parameters and concurrent scenarios. The computing unit is used to calculate the amount of computing resources required by the service based on the service application information, the number of computing cards occupied by the single model instance, and the performance baseline data. The data comparison unit is used to compare the number of idle cards in the target model resource information with the amount of computing resources required by the service. The data adjustment unit is used to allocate corresponding concurrent resources to the service and update the number of idle cards in the model if the number of idle cards in the model is greater than the number of computing resources required by the service.
9. A computing device, characterized in that, The computing device includes: a memory and a processor; The memory is used to store computer programs; The processor is configured to implement the steps of the method as described in any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 7.