A Method and System for Detecting Concurrency Critical Points in Large Language Models Based on Performance Degradation Slope

By calculating the performance degradation slope and correcting it with hardware feedback information, the concurrency level detection step size is dynamically adjusted, which solves the efficiency and security problems of concurrency critical point detection in large language model inference services, and ensures the reliability of resource utilization and service stability.

CN123070189APending Publication Date: 2026-11-10四川华鲲振宇智能科技有限责任公司 +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611584577.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-14
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing methods for detecting concurrent critical points in large language model inference services suffer from nonlinear characteristics such as unsuitable step size, performance degradation, and excessively long detection times. Furthermore, they lack awareness of hardware resource reserves, leading to resource waste and service anomalies.

Method used

By collecting performance metrics under multiple consecutive concurrency levels, the performance degradation slope is calculated and a slope sequence is constructed. The prediction results are corrected by combining hardware feedback information, and the detection process is controlled by a finite state machine to dynamically adjust the step size to determine the maximum safe concurrency level.

Benefits of technology

This approach enables the rapid and accurate determination of the maximum security concurrency level for large language model inference services while ensuring detection security, thereby avoiding resource waste and service anomalies and improving detection efficiency and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123070189A_ABST
    Figure CN123070189A_ABST
Patent Text Reader

Abstract

This invention discloses a method and system for detecting the concurrency critical point of a large language model based on performance degradation slope, belonging to the field of artificial intelligence technology. The key technical points are: collecting performance indicators of a large language model inference service under multiple consecutive concurrency levels; calculating the performance degradation slope based on the ratio of the performance indicator difference between adjacent concurrency levels to the concurrency level difference; arranging these into a performance degradation slope sequence in ascending order of concurrency levels; predicting the critical concurrency level or the next concurrency level based on this sequence; obtaining hardware feedback information such as memory pressure parameters and computing power pressure parameters to correct the prediction results; controlling the advancement, reversal, or termination of the detection process through a finite state machine, and obtaining the maximum safe concurrency level through binary search when exceeding the limit. This invention can balance detection efficiency and security, accurately determine the maximum safe concurrency level that satisfies the service level agreement, and avoid memory overflow or service interruption.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a method and system for detecting concurrent critical points in large language models based on performance degradation slope. Background Technology

[0002] In recent years, large language model inference services have been widely used in the field of natural language processing, with their deployment scale and the number of concurrent requests continuing to grow. The number of concurrent requests directly determines the throughput and resource consumption of the inference service. Too low a concurrency level cannot fully utilize computing resources, while too high a concurrency level may lead to excessive response latency, memory overflow, or even service crashes. Therefore, accurately detecting the maximum concurrency level that a large language model inference service can safely handle is crucial for ensuring service level agreements and resource scheduling.

[0003] In existing technologies, the detection of concurrency critical points typically employs a fixed-step, incremental approach. Starting from an initial concurrency level, the number of concurrent requests is increased in fixed increments each round, performance metrics at that level are collected, and the concurrency level preceding the first time the performance metrics exceed the service level agreement (SLA) limit is taken as the maximum safe concurrency level. However, this method has significant shortcomings. Firstly, a fixed step size struggles to adapt to the non-linear characteristics of performance degradation. An excessively large step size can easily overshoot the critical point, leading to instantaneous service overload or even memory overflow; an excessively small step size results in too many detection rounds and excessive time consumption. Secondly, this method relies solely on post-hoc assessment of performance metrics, failing to consider underlying hardware resource reserves such as memory and computing power pressure. It cannot predict risks before performance metrics exceed limits, nor can it dynamically adjust the detection step size based on remaining resources. Furthermore, the entire detection process lacks a state management mechanism; the advancement, rollback, and termination of the detection process cannot adaptively switch based on the current service state, easily leading to problems such as inability to recover after exceeding limits or inaccurate safety boundary positioning.

[0004] Therefore, researching and designing a method and system for detecting concurrent critical points in large language models based on performance degradation slope that can overcome the above-mentioned defects is an urgent problem to be solved. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a method and system for detecting the concurrency critical point of large language models based on performance degradation slope. This method collects performance indicators at multiple consecutive concurrency levels, calculates the performance degradation slope between adjacent concurrency levels, constructs a performance degradation slope sequence, and corrects the prediction results using hardware feedback information. Furthermore, it utilizes a finite state machine to control the advancement, rollback, or termination of the detection process. Under the premise of meeting service level agreement requirements, this method efficiently and accurately determines the maximum safe concurrency level for large language model inference services, thereby avoiding the risk of out-of-bounds access and resource waste caused by fixed-step detection, and providing a reliable basis for computing power scheduling and capacity planning.

[0006] The above-mentioned technical objective of the present invention is achieved through the following technical solution: Firstly, a method for detecting concurrent critical points in large language models based on performance degradation slope is provided, including the following steps: Under multiple consecutive concurrency levels, requests are sent to the large language model inference service according to the number of requests corresponding to each concurrency level, and the performance indicators of the large language model inference service under each concurrency level are collected. Calculate the performance degradation slope between adjacent concurrency levels based on the performance index, and arrange the performance degradation slopes into a performance degradation slope sequence in order of concurrency level from low to high. Predict the critical concurrency level or the next concurrency level of the large language model inference service based on the performance degradation slope sequence; Obtain hardware feedback information of the large language model inference service under the multiple consecutive concurrency levels; The prediction results are corrected based on the hardware feedback information. Based on the revised prediction results, the concurrency level detection process is controlled by a finite state machine to advance, regress, or terminate.

[0007] Furthermore, the performance metrics include at least one of the following: first word time, inter-word latency, video memory usage, key-value cache usage, and number of words output per second. Each performance metric under each concurrency level is recorded as the average of multiple requests within a statistical window.

[0008] Furthermore, the performance degradation slope is calculated based on the ratio of the performance index difference between two adjacent concurrency levels to the concurrency level difference; Each performance degradation slope in the performance degradation slope sequence corresponds to a current concurrency level, which is the relatively higher concurrency level among two adjacent concurrency levels involved in the calculation.

[0009] Furthermore, predicting the critical concurrency level or the next concurrency level of the large language model inference service based on the performance degradation slope sequence includes: The difference between the last two performance degradation slopes in the performance degradation slope sequence is performed to obtain the slope change. If the slope change exceeds a preset slope change threshold, the large language model inference service is determined to be close to the critical concurrency level, and the current concurrency level is used as the predicted critical concurrency level. If the slope change does not exceed the preset slope change threshold, the next concurrency level is predicted based on the average of the M performance degradation slopes at the end of the performance degradation slope sequence, where M is a positive integer greater than or equal to 2.

[0010] Furthermore, the hardware feedback information includes video memory pressure parameters; The video memory pressure parameters are calculated based on the current video memory occupancy rate, the current key-value cache occupancy rate, the key-value cache growth rate, and the key-value cache expansion amount corresponding to the estimated maximum context length. The video memory pressure parameters are monotonically increasing with the current video memory occupancy rate, the current occupancy rate of the key-value cache, the growth rate of the key-value cache, and the expansion amount of the key-value cache.

[0011] Furthermore, the hardware feedback information also includes computing power pressure parameters; The computing power pressure parameters are calculated based on computing power utilization, number of requests processed per second, and average queuing latency. The computing power pressure parameters are monotonically increasing with the computing power utilization rate, the number of requests processed per second, and the average queuing delay.

[0012] Furthermore, the prediction result includes the next concurrency level, and the difference between the next concurrency level and the highest concurrency level among the currently collected performance metric concurrency levels is used as the concurrency level increment. When correcting the prediction results based on the hardware feedback information, the memory pressure parameter and the computing power pressure parameter are weighted and summed to obtain the comprehensive resource pressure parameter. The dynamic damping coefficient is calculated based on the comprehensive resource pressure parameters, and the dynamic damping coefficient has a monotonically increasing relationship with the comprehensive resource pressure parameters. The concurrent level increment is reduced using the dynamic damping coefficient, and the reduced value of the concurrent level increment is not less than 1.

[0013] Furthermore, the finite state machine includes a safety pressurization state, a risk correction state, an out-of-bounds convergence state, and a detection termination state; Under the aforementioned safe pressurization state, the concurrency level is increased according to the modified concurrency level increment; Under the risk correction state, the concurrency level increment is reduced, and the step of predicting the next concurrency level based on the performance degradation slope sequence is re-executed; When the performance index of the large language model inference service exceeds the preset range at the concurrency level during the detection process, the concurrency level at which the detection process is located is taken as the out-of-bounds concurrency level; the highest concurrency level among the detected concurrency levels whose performance index does not exceed the preset range is taken as the nearest safe concurrency level; and the out-of-bounds convergence state is entered. In the out-of-bounds convergence state, local convergence is performed between the out-of-bounds concurrency level and the nearest safe concurrency level; When the termination condition is met, the probe is entered into the termination state, and the maximum security concurrency level is output.

[0014] Furthermore, the step of performing local convergence between the out-of-bounds concurrency level and the nearest safe concurrency level includes: The average of the out-of-bounds concurrency level and the nearest safe concurrency level is used as the intermediate concurrency level; Requests are sent to the large language model inference service according to the number of requests corresponding to the intermediate concurrency level, and the corresponding performance metrics are collected. If the performance metrics at the intermediate concurrency level meet the service level agreement requirements, the intermediate concurrency level is updated to the nearest safe concurrency level; otherwise, the intermediate concurrency level is updated to the out-of-bounds concurrency level. Repeat the process until the difference between the out-of-bounds concurrency level and the nearest safe concurrency level is less than the preset convergence precision, and then determine the nearest safe concurrency level as the maximum safe concurrency level.

[0015] Secondly, a concurrent critical point detection system for large language models based on performance degradation slope is provided, including: The request collection module is used to send requests to the large language model inference service according to the number of requests corresponding to each concurrency level under multiple consecutive concurrency levels, and to collect the performance indicators of the large language model inference service under each concurrency level. The slope calculation and arrangement module is used to calculate the performance degradation slope between adjacent concurrency levels based on the performance index, and arrange the performance degradation slope into a performance degradation slope sequence in order of concurrency level from low to high. The concurrency level prediction module is used to predict the critical concurrency level or the next concurrency level of the large language model inference service based on the performance degradation slope sequence. The hardware feedback acquisition module is used to acquire hardware feedback information of the large language model inference service under the multiple consecutive concurrency levels. The prediction result correction module is used to correct the prediction result based on the hardware feedback information; The detection process control module is used to control the advancement, reversal, or termination of the concurrent level detection process through a finite state machine based on the corrected prediction results.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention provides a method for detecting the critical concurrency level of a large language model based on performance degradation slope. This method collects performance metrics of the large language model inference service at multiple consecutive concurrency levels, calculates the performance degradation slope between adjacent concurrency levels and arranges them into a performance degradation slope sequence. Based on this sequence, it predicts the critical concurrency level or the next concurrency level. Simultaneously, it obtains hardware feedback information to correct the prediction results and uses a finite state machine to control the advancement, rollback, or termination of the detection process. Performance metrics reflect the current load status from a service quality perspective; the performance degradation slope sequence reveals the performance degradation pattern with increasing concurrency from a trend perspective; hardware feedback information perceives the real-time pressure on memory and computing power from a resource perspective; and the finite state machine uniformly schedules the detection behavior from a process perspective. The prediction module provides the direction of advancement, the correction module constrains the step size, and the state machine determines when to advance, rollback, or terminate. These three components form a closed-loop linkage mechanism of feedforward prediction, feedback correction, and state control. This method rapidly approaches the critical concurrency level while ensuring the safety and controllability of the detection process, solving the problems of existing fixed-step detection methods that struggle to balance detection efficiency and safety, and are prone to exceeding the critical point, leading to service anomalies.

[0017] 2. This invention establishes a performance benchmark under low load through baseline sampling during the data acquisition phase. Within a statistical window, the average of multiple requests is used as the performance metric record value for each concurrency level. Simultaneously, it collects performance metrics across multiple dimensions, including first-word time, inter-word latency, memory usage, key-value cache usage, and the number of words output per second. Baseline sampling eliminates cold-start disruptions caused by system cold starts and cache warm-up. Statistical window averaging eliminates random errors introduced by network jitter or system fluctuations in a single request. The multi-dimensional performance metrics comprehensively reflect the service status from different perspectives, such as latency, memory usage, and throughput. This invention improves the stability and integrity of input data from the source, providing a reliable data foundation for subsequent slope calculation and trend prediction, and effectively reducing the risk of misjudgment of critical points due to data noise.

[0018] 3. This invention calculates the performance degradation slope by the ratio of the performance index difference between adjacent concurrency levels to the concurrency level difference. The slopes are arranged in ascending order of concurrency level to form a performance degradation slope sequence. The slope change is obtained by calculating the difference between the last two slopes in the sequence. Whether the slope change exceeds a preset threshold determines whether the large language model inference service is approaching a critical concurrency level. If not approaching the critical level, the next concurrency level is predicted based on the average of the last multiple slopes. If approaching the critical level, the current concurrency level is directly used as the predicted critical concurrency level. The performance degradation slope sequence transforms the original performance index into trend features, amplifying the magnitude of performance degradation. The slope change further characterizes the acceleration of the degradation trend, forming a progressive judgment logic from first-order to second-order trends. Since performance degradation near the critical point usually exhibits accelerated characteristics, this judgment method can capture critical signals earlier, reducing unnecessary fine-grained probing while ensuring prediction accuracy and improving detection efficiency.

[0019] 4. This invention comprehensively characterizes the stress level of underlying hardware resources through memory pressure parameters and computing power pressure parameters. The memory pressure parameter integrates the current memory occupancy rate, the current key-value cache occupancy rate, the key-value cache growth rate, and the key-value cache expansion amount corresponding to the estimated maximum context length. It reflects both the current memory occupancy level and the memory expansion pressure caused by continued context growth. The computing power pressure parameter integrates computing power utilization, requests processed per second, and average queuing latency. It reflects both the saturation level of computing resources and the request processing capacity and queuing status. The memory pressure parameter shows a monotonically increasing relationship with various memory-related indicators, and the computing power pressure parameter shows a monotonically increasing relationship with various computing power-related indicators. Therefore, when any underlying resource approaches a bottleneck, the corresponding pressure parameter will increase in advance, sending a resource stress signal to the prediction module before performance indicators deteriorate significantly or exceed service level agreement limits. This mechanism quantifies the hardware resource status into a feedback signal that can participate in calculations, providing advance notice for subsequent prediction result correction and effectively avoiding service interruptions caused by memory overflow or computing power overload.

[0020] 5. This invention obtains a comprehensive resource pressure parameter by weighted summation of memory pressure parameters and computing power pressure parameters. A dynamic damping coefficient is then calculated based on this comprehensive resource pressure parameter, which monotonically increases with the comprehensive resource pressure parameter. This dynamic damping coefficient is then used to reduce the concurrency level increment, ensuring that the reduced increment is not less than 1. The comprehensive resource pressure parameter unifies the two heterogeneous resources, memory and computing power, into a one-dimensional pressure index, allowing resource status to be controlled by a single threshold and a single coefficient. The dynamic damping coefficient increases as the comprehensive resource pressure parameter rises, creating an exponential compression effect on the concurrency level increment. This causes the probe step size to decrease rapidly when resources are scarce and maintain a large value when resources are abundant, achieving adaptive adjustment of the probe step size according to hardware resource availability. Simultaneously, the reduced increment is limited to not less than 1, ensuring that the probe process does not stall due to excessive conservatism. This mechanism links the predicted step size with the actual resource availability, enabling rapid advancement in low-load areas and smooth deceleration in high-load areas, fundamentally preventing excessively large single-step increases in concurrency levels that could lead to out-of-bounds errors.

[0021] 6. This invention uses a finite state machine to divide the detection process into a safe pressure state, a risk correction state, an out-of-bounds convergence state, and a detection termination state. In the safe pressure state, the concurrency level is increased according to the corrected concurrency level increment to achieve rapid advancement in low-load areas. In the risk correction state, the concurrency level increment is decreased and prediction is re-executed to achieve defensive deceleration when approaching resource boundaries. When performance indicators exceed a preset range, the current concurrency level is designated as the out-of-bounds concurrency level, and the highest concurrency level among the detected concurrency levels whose performance indicators do not exceed the preset range is designated as the nearest safe concurrency level, entering the out-of-bounds convergence state. In the out-of-bounds convergence state, the average of the out-of-bounds concurrency level and the nearest safe concurrency level is repeatedly calculated as an intermediate concurrency level. The nearest safe concurrency level or the out-of-bounds concurrency level is updated based on whether the performance indicators under the intermediate concurrency level meet the service level agreement requirements until the difference between the two is less than the preset convergence accuracy. Finally, the nearest safe concurrency level is determined as the maximum safe concurrency level. Finite state machines integrate detection, prediction, correction, and boundary crossing handling into a standardized state transition process, enabling the system to autonomously select appropriate detection strategies at different load stages. After a boundary crossing, the binary search gradually narrows the safety boundary interval with logarithmic number of detections, avoiding the large request overhead caused by exhaustive detection. The final maximum safety concurrency level output simultaneously meets the service level agreement requirements and resource security constraints, providing a reliable capacity planning basis for the deployment of large-scale language model inference services. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart from Embodiment 1 of the present invention; Figure 2 This is a system block diagram in Embodiment 2 of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0024] Example 1: A method for detecting concurrent critical points in large language models based on performance degradation slope, such as... Figure 1 As shown, it includes the following steps: S1: Under multiple consecutive concurrency levels, send requests to the large language model inference service according to the number of requests corresponding to each concurrency level, and collect the performance indicators of the large language model inference service under each concurrency level. S2: Calculate the performance degradation slope between adjacent concurrency levels based on performance metrics, and arrange the performance degradation slopes into a performance degradation slope sequence according to the concurrency level from low to high. S3: Predict the critical concurrency level or the next concurrency level of the large language model inference service based on the performance degradation slope sequence; S4: Obtain hardware feedback information for the large language model inference service under multiple consecutive concurrency levels; S5: Correct the prediction results based on hardware feedback information; S6: Based on the corrected prediction results, the concurrent level detection process is controlled by a finite state machine to advance, roll back, or terminate.

[0025] In step S1, firstly, multiple consecutive concurrency levels are determined. A concurrency level is a level representing the number of requests sent to the large language model inference service. The value is equal to the number of requests sent simultaneously to the large language model inference service at that concurrency level, where `concurrency_level` is a positive integer representing the sequence number of the concurrency level. The concurrency levels are arranged in ascending order, for example: ,in The total number of concurrent levels detected, which is a positive integer greater than or equal to 2.

[0026] In the initial probing phase, baseline sampling is performed first. During baseline sampling, at least two lower concurrency levels are selected from multiple consecutive concurrency levels. Requests are sent continuously and performance metrics are collected to establish a performance benchmark under low load. Baseline sampling is used to eliminate interference factors such as system cold starts and cache warm-up, and to eliminate the impact of cold start gaps on subsequent slope calculations.

[0027] After baseline sampling is completed, requests are sent to the large language model inference service according to the number of requests corresponding to each concurrency level. Specifically, for each concurrency level... The number of messages sent to the large language model inference service is The request enables the large language model inference service to perform inference tasks at this concurrency level. When sending the request, the parameters, model configuration, and runtime environment are kept consistent, and only the number of requests is changed to ensure that changes in performance metrics when the concurrency level changes are caused only by concurrent load.

[0028] At each concurrency level, performance metrics for the large language model inference service are collected. These metrics include at least one of the following: first-term time, inter-term latency, GPU memory utilization, key-value cache utilization, and number of terms output per second. First-term time is the time interval from the request being sent to the generation of the first term, reflecting the saturation of computing resources; inter-term latency is the time interval between the generation of two adjacent terms, reflecting the changing trend of cache access and data transfer pressure; GPU memory utilization is the percentage of GPU memory used by the computing device; key-value cache utilization is the percentage of GPU memory used by the key-value cache relative to the total GPU memory capacity; and number of terms output per second is the number of terms generated per unit time, reflecting the inference throughput.

[0029] At each concurrency level, each performance metric is recorded as the average of multiple requests within a statistical window. The statistical window is a pre-defined time period or number of requests. Within each concurrency level, multiple requests are sent within the statistical window, and the performance metric value for each request is recorded. Then, the arithmetic mean of the multiple recorded values ​​for the same performance metric is calculated, and this arithmetic mean is used as the recorded value for that performance metric at that concurrency level. Using the average of multiple requests within a statistical window eliminates random errors introduced by network jitter or system fluctuations in a single request, making the collected performance metrics more stable and reliable.

[0030] In step S2, based on the performance index records obtained in step S1 for each concurrency level, for any two adjacent concurrency levels... and , .set up and Concurrency levels and The recorded values ​​of the same performance metric under the same concurrency level. The formula for calculating the performance degradation slope between two adjacent concurrency levels is: ; in, For concurrency level and The performance degradation slope between concurrency levels is used to represent the performance degradation slope from the concurrency level. Increase to The change in performance metrics caused by a unit of concurrent increment. The unit of performance degradation slope is determined by the performance metric. When the performance metric is the first-token time or the latency between tokens, the unit is milliseconds per concurrent; when the performance metric is the video memory utilization rate or the key-value cache utilization rate, the unit is a percentage per concurrent; when the performance metric is the number of tokens output per second, the performance degradation slope is usually negative, used to reflect the rate of decline in throughput.

[0031] Then, the calculated performance degradation slopes are arranged sequentially according to the concurrency level from low to high, resulting in a performance degradation slope sequence. This sequence can be represented as follows: ,in Minimum concurrency level With the second lowest concurrency level The slope of performance degradation between them For the two highest concurrency levels and The slope of performance degradation between them.

[0032] Each performance degradation slope in the performance degradation slope sequence corresponds to a current concurrency level. For the performance degradation slope... The corresponding current concurrency level is the higher of the two adjacent concurrency levels used to calculate the performance degradation slope, i.e. The current concurrency level represents the load position of the large language model inference service under the performance degradation state reflected by this performance degradation slope. This current concurrency level is used as a reference point when subsequently predicting critical concurrency levels. For example, based on the concurrency level... and When the performance degradation slope is calculated, the current concurrency level corresponding to that slope is a relatively high concurrency level. .

[0033] The performance degradation slope sequence reflects the continuous degradation trend of performance indicators as concurrency levels increase. When the slope value in the performance degradation slope sequence continuously increases, it indicates that the performance degradation rate is accelerating, and the large language model inference service is approaching the critical concurrency level; when the slope value growth rate slows down or fluctuates, it indicates that the performance degradation trend may be leveling off. This sequence provides a data foundation for subsequent prediction of the critical concurrency level or the next concurrency level.

[0034] In some preferred embodiments, after constructing the performance degradation slope sequence, the slope change between adjacent performance degradation slopes is calculated to characterize the accelerating trend of performance degradation slope change. The difference operation is performed on two adjacent performance degradation slopes in the performance degradation slope sequence. For the first... individual performance degradation slope The formula for calculating the corresponding change in slope is: ; in, The first in the performance degradation slope sequence A performance degradation slope, For concurrency level The corresponding slope change. The slope change reflects the second-order trend of performance degradation, i.e., whether the performance degradation is gradually slowing down or accelerating. When the load approaches the critical point, the performance degradation slope increases rapidly, leading to a rapid increase in the slope change. Therefore, the slope change can be used to predict the arrival of a steep change zone before the absolute value of the performance index crosses the limit. When hour, Take 0.

[0035] In step S3, the performance degradation slope sequence has been obtained in step S2. and the corresponding change in slope Among them, the slope change at the end of the sequence. It is achieved by arranging the last two performance degradation slopes in the performance degradation slope sequence. and The result obtained by performing a difference operation, i.e. . It indicates the degree of change in the rate of performance degradation between the two latest concurrency levels, and is used to determine whether the performance degradation is showing an accelerating trend.

[0036] First, set a preset slope change threshold. Preset slope change threshold A pre-defined positive number is used to determine whether the performance degradation slope shows an accelerating trend. Its value is determined based on the performance metric type and testing environment of the large language model inference service. The change in the final slope... Compared with the preset slope change threshold Compare them.

[0037] like Greater than If the performance degradation slope is too high, the large language model inference service is considered to be approaching the critical concurrency level. The corresponding current concurrency level As a predicted value for the critical concurrency level. Current concurrency level. To participate in the calculation of performance degradation slope The higher of two adjacent concurrency levels is the highest concurrency level detected so far. When the slope change increases significantly, it indicates that the performance degradation rate is accelerating rapidly. Continuing to increase the concurrency level will quickly cause the performance indicators to exceed the service level agreement limits. Therefore, the current concurrency level is output as the prediction result of the critical concurrency level, and subsequent steps will be used to correct it based on hardware feedback.

[0038] like The slope change threshold was not exceeded. If so, then the prediction for the next concurrency level is executed. Specifically, the last one in the performance degradation slope sequence is selected. The performance degradation slope, among which It is a positive integer greater than or equal to 2, in this embodiment Take 3. Calculate this. The average of the slopes of performance degradation is used as the overall performance degradation trend measure, denoted as . Overall performance degradation trend The calculation formula is: ; in, The last one in the performance degradation slope sequence Individual performance degradation slope. Overall performance degradation trend. This average rate of performance degradation near the current concurrency level is used as the basis for calculating the subsequent prediction step size. Using the average of multiple performance degradation slopes at the end of the curve smooths out the impact of single-point slope fluctuations, making the prediction results more stable.

[0039] Furthermore, obtain the current performance metric value. The current performance metric value is the highest detected concurrency level. The recorded value of this performance indicator is denoted as: Simultaneously, obtain the Service Level Agreement (SLA) limit value corresponding to this performance metric, and denote it as... The Service Level Agreement (SLA) limit is a pre-defined upper limit for performance metrics, such as a first-to-element time not exceeding 2000 milliseconds. Calculate the absolute remaining margin between the current performance metric and the SLA limit: ; in, This represents the remaining available margin before the performance metric reaches the service level agreement (SLA) limit at the current concurrency level, expressed in units identical to the performance metric value. It should be noted that this calculation involves absolute remaining space. The performance indicators used must be consistent with the calculation of the overall performance degradation trend. The same performance index is used throughout to ensure consistency in the physical meaning of the prediction step size. For example, if Based on the performance degradation slope corresponding to the first word time, then For the first meta-time in the concurrency level The recorded values ​​below, The service level agreement limit value for the first meta term.

[0040] Based on this, calculate the basic prediction step size: ; in, The base prediction step size represents the number of additional concurrent tiers allowed after exhausting the remaining service level protocol margin, given the current rate of performance degradation; the unit is concurrency. Absolute remaining space. The unit is milliseconds, and the overall performance degradation trend is... The unit is milliseconds per concurrency; dividing the two gives the base prediction step size. The unit is the number of concurrent users.

[0041] Finally, set the current concurrency level. With the basic prediction step size Summing these values ​​yields the predicted concurrency level for the next concurrency level: ; in, This is the predicted next concurrency level. This prediction will serve as input for subsequent steps, and will be refined based on hardware feedback.

[0042] In step S4, during the execution of each concurrency level task, computing device resource data such as video memory usage, key-value cache usage, key-value cache growth rate, and computing power utilization have been collected synchronously. Performance indicators such as video memory utilization rate and key-value cache utilization rate have also been collected. Based on this collected data, this step calculates the video memory pressure parameters and computing power pressure parameters for each concurrency level.

[0043] The video memory pressure parameter is used to characterize the current video memory resource strain of the large language model inference service. Let the current video memory utilization rate be... The value is the memory usage rate under the latest concurrency level collected in step S1; the current key-value cache usage rate is... The value is the key-value cache occupancy rate under the latest concurrency level collected in step S1; the key-value cache growth rate is... The value is the key-value cache growth rate collected in step S1, representing the change in key-value cache occupancy per unit time; the estimated key-value cache expansion amount corresponding to the maximum context length is... This parameter is calculated based on the difference between the estimated maximum context length and the current context length, as well as the key-value cache usage for each lexical unit. It characterizes the additional GPU memory required by the key-value cache if the context length continues to grow to the limit supported by the model. The formula for calculating the GPU memory pressure parameter is: ; in, These are parameters related to video memory stress. , , , These are weighted coefficients for the current memory usage, current key cache usage, key cache growth rate, and key cache expansion amount corresponding to the estimated maximum context length, respectively. Each weighted coefficient is a preset positive number. Since all weighted coefficients are positive, the memory pressure parameter increases when any one of these factors increases. Therefore, the memory pressure parameter exhibits a monotonically increasing relationship with the current memory usage, current key cache usage, key cache growth rate, and key cache expansion amount corresponding to the estimated maximum context length.

[0044] The computational stress parameter is used to characterize the current computational resource strain on the large language model inference service. Let the computational utilization rate be... The value is the computing power utilization rate of the computing equipment collected in step S1, representing the percentage of computing equipment computing power resources being utilized; the number of requests processed per second is... , representing the number of requests completed per unit of time; average queuing latency is This represents the average time, in milliseconds, for a request to wait in the queue for processing. The formula for calculating the computing power pressure parameter is: ; in, For computing power pressure parameters, , , These are the weighting coefficients for computing power utilization, requests per second, and average queuing latency, respectively, and each weighting coefficient is a preset positive number. Since all weighting coefficients are positive, when any one of the computing power utilization, requests per second, or average queuing latency increases, the computing power pressure parameter also increases. Therefore, the computing power pressure parameter has a monotonically increasing relationship with computing power utilization, requests per second, and average queuing latency, respectively.

[0045] Through the above calculations, the memory pressure parameters and computing power pressure parameters under each concurrency level are obtained, and the two together constitute the hardware feedback information.

[0046] In step S5, the output prediction result is corrected based on the acquired hardware feedback information to avoid excessive concurrency level increments in pure software performance predictions when approaching physical resource boundaries, which could lead to memory overflow or resource overload. When the slope change does not exceed a preset slope change threshold, the next concurrency level prediction value is output. And at the same time, the highest concurrency level among the currently collected performance metrics was obtained. .

[0047] This step first calculates the concurrency level increment, which is the predicted value for the next concurrency level. Compared to the highest concurrency level among the currently collected performance metrics. The difference is used as the concurrency level increment: ; in, The concurrency level increment represents the number of additional concurrency levels predicted for the next round of probing, based on performance degradation trends and without considering hardware resource pressure. This increment can be large, enabling rapid transitions when resources are abundant, but carries the risk of exceeding physical boundaries when resources are scarce.

[0048] Then, the overall resource pressure parameter is calculated based on the memory pressure parameter and computing power pressure parameter obtained in step S4. The memory pressure parameter and computing power pressure parameter are then weighted and summed to obtain the overall resource pressure parameter: ; in, This is a comprehensive resource pressure parameter used to comprehensively reflect the overall pressure level of video memory and computing resources currently experienced by the large language model inference service; This is a memory stress parameter, taken from the highest concurrency level among the currently collected performance metrics. The corresponding memory pressure parameters; This is a computing power pressure parameter, taken from the concurrency level. Corresponding computing power pressure parameters; These are the weighting coefficients for the video memory stress parameters. These are the weighting coefficients for the computing power pressure parameters. and All are preset positive numbers, and satisfy the following conditions: The weighting coefficients are determined based on the impact of memory and computing resources on performance bottlenecks in the actual deployment environment. When memory resources are more scarce, the weighting coefficient is increased. The value of ; when computing resources are more strained, increase The value of .

[0049] Furthermore, a dynamic damping coefficient is calculated based on the comprehensive resource pressure parameter. The dynamic damping coefficient is used to reduce the increase in concurrency level, and its value increases monotonically with the comprehensive resource pressure parameter. That is, the larger the comprehensive resource pressure parameter, the larger the dynamic damping coefficient, and the greater the reduction in the increase in concurrency level.

[0050] Specifically, setting warning water level thresholds. , This is a pre-defined constant, ranging from 0 to 1; in this embodiment, it is set to 0.75. When considering the comprehensive resource pressure parameter... The water level did not exceed the warning threshold. When the current hardware resources still have sufficient margin, the dynamic damping coefficient is set to the base value of 1, meaning no additional reduction processing is performed. When the comprehensive resource pressure parameter... Exceeding the warning water level threshold This indicates that the large language model inference service is approaching the limit of hardware resources. At this point, it is necessary to increase the dynamic damping coefficient to compress the increase in concurrency levels. The formula for calculating the dynamic damping coefficient is: ; in, This refers to the dynamic damping coefficient; It is a natural constant; This is a pressure growth rate parameter used to control how quickly the dynamic damping coefficient increases with the overall resource pressure parameter. The value is a preset positive number, ranging from 0.5 to 2.0; in this embodiment, it is 1.0. Since the exponential function is a monotonically increasing function, the dynamic damping coefficient... With comprehensive resource pressure parameters It exhibits a monotonically increasing relationship. When the comprehensive resource pressure parameter increases from 0.75 to 0.90, the dynamic damping coefficient increases from 1 to... The result is approximately 1.16. When the value is further increased to 0.95, the dynamic damping coefficient increases to... The result is approximately 1.22, achieving a gradual compression of the concurrency level increment.

[0051] Then, the concurrency level increment is reduced using a dynamic damping coefficient. The formula for calculating the reduced concurrency level increment is: ; in, For the reduced concurrency level increment, To ensure that the resulting concurrency level increment is an integer, a floor function is used. Additionally, the reduced concurrency level increment must be at least 1. ; in, To maximize the operation, this ensures that each probe increases by at least one concurrency level, preventing the probe process from stalling due to excessive reduction.

[0052] The reduced concurrency level increment is added to the highest concurrency level among the currently collected performance metrics to obtain the corrected next concurrency level: ; in, The corrected concurrency level will be used as the actual concurrency level for the next probe, and the output will be processed by the state machine.

[0053] By employing the above method, this step retains a large concurrency level increment when the overall resource pressure parameter is low, and gradually compresses the concurrency level increment when the overall resource pressure parameter increases. This achieves a shift from passively waiting for performance to exceed limits to proactively implementing defensive slowdowns based on hardware resource pressure, effectively avoiding service interruptions caused by video memory overflow.

[0054] In step S6, the finite state machine includes a safety pressurization state, a risk correction state, an out-of-bounds convergence state, and a probe termination state. The corrected next concurrency level is output in step S5. It serves as the input for the finite state machine to perform the detection process.

[0055] Under safe pressure conditions, the concurrency level is increased according to the revised concurrency level increment. Specifically, the current concurrency level is updated to the revised next concurrency level. The system sends requests to the large language model inference service according to the request volume corresponding to the concurrency level, and collects performance metrics and hardware feedback information under that concurrency level. If the performance metrics under that concurrency level meet the service level agreement limits, and the overall resource pressure parameters... The water level did not exceed the warning threshold. If the concurrency level remains high, the system will continue to operate under safe pressure and return to step S1 to perform a new round of concurrent detection and indicator collection at that concurrency level. This is based on the comprehensive resource pressure parameters. Exceeding the warning water level threshold Then it enters a risk correction state. The safety pressure state can retain a large concurrency level increment when the overall resource pressure is low, thereby achieving rapid leap in resource-rich areas.

[0056] In the risk correction state, the concurrency level increment is reduced, and the step of predicting the next concurrency level based on the performance degradation slope sequence is re-executed. Specifically, the upper limit of the concurrency level increment is reduced to 1 concurrency level, or the dynamic damping coefficient is forcibly adjusted to the minimum value to limit the excessive increase in concurrency levels. Then, steps S2 and S3 are re-executed to predict the new next concurrency level based on the latest performance degradation slope sequence, and steps S4 and S5 are executed to obtain the corrected next concurrency level. In this state, even if the performance indicators still meet the service level agreement limits, the probing is carried out in an absolutely safe manner to prevent the concurrency level from exceeding the physical boundary, which could lead to memory overflow or service interruption. If, after re-execution of the prediction, the overall resource pressure parameter returns to below the warning water level threshold and the performance indicators still meet the service level agreement limits, then the state returns to the safe pressure increase state; if the performance indicators exceed the preset range, then the state of out-of-bounds convergence is entered.

[0057] When the performance metrics of the large language model inference service exceed the preset range at the concurrency level during the probing process, it indicates that the current concurrency level has exceeded the safety boundary. At this point, the concurrency level at which the probing process is located is designated as the out-of-bounds concurrency level, denoted as... Simultaneously, the highest concurrency level among the detected concurrency levels whose performance indicators do not exceed the preset range is taken as the most recent safe concurrency level, denoted as... It then enters an out-of-bounds convergence state. The preset range is the upper limit of the performance indicators specified in the Service Level Agreement, such as the first word / token time not exceeding 2000 milliseconds and the inter-token latency not exceeding 500 milliseconds. Since the detection process is executed in ascending order of concurrency level, the highest concurrency level among the detected concurrency levels that does not exceed the preset range is the highest load position that can be confirmed for safe operation. Using this position as the lower bound of local convergence can ensure that the convergence result meets the requirements of the Service Level Agreement.

[0058] In the out-of-bounds convergence state, in the out-of-bounds concurrency level With the latest security concurrency level Local convergence is achieved between these levels. First, the average of the out-of-bounds concurrency level and the nearest safe concurrency level is taken as the intermediate concurrency level. Since the concurrency level is a positive integer, the formula for calculating the intermediate concurrency level is: ; in, The intermediate concurrency level is then determined. Requests are sent to the large language model inference service according to the number of requests corresponding to that intermediate concurrency level, and performance metrics are collected for that level. If the performance metrics at that intermediate concurrency level meet the service level agreement (SLA) requirements, the intermediate concurrency level is updated to the most recent safe concurrency level; if the performance metrics at that intermediate concurrency level do not meet the SLA requirements, i.e., the performance metrics exceed a preset range, the intermediate concurrency level is updated to the out-of-bounds concurrency level. Through these update rules, the range between the out-of-bounds concurrency level and the most recent safe concurrency level continuously narrows, with the most recent safe concurrency level always representing a safe operating position that meets the SLA requirements, and the out-of-bounds concurrency level always representing an out-of-bounds position that does not meet the SLA requirements.

[0059] Repeat the steps of using the average of the out-of-bounds concurrency level and the most recent safe concurrency level as the intermediate concurrency level, sending requests according to the number of requests corresponding to the intermediate concurrency level and collecting performance metrics, and updating the most recent safe concurrency level or the out-of-bounds concurrency level according to the service level agreement requirements, until the difference between the out-of-bounds concurrency level and the most recent safe concurrency level meets the termination condition. The termination condition is: ; in, The preset convergence accuracy is expressed in units of concurrency levels, and in this embodiment, the value is set to 1. When the difference between the out-of-bounds concurrency level and the nearest safe concurrency level is not greater than the preset convergence accuracy, it indicates that the safety boundary has been located between adjacent concurrency levels and cannot be further subdivided. At this point, the termination condition is met, and the nearest safe concurrency level is determined as the maximum safe concurrency level.

[0060] When the termination condition is met, the finite state machine enters the detection termination state and outputs the maximum safe concurrency level. This maximum safe concurrency level is the concurrency critical point for the large language model inference service under the condition of meeting the service level agreement requirements, and can be directly used for computing power scheduling and resource planning. At this point, the entire concurrency critical point detection process ends.

[0061] This embodiment uses a large language model inference service with 7B parameters as the experimental object, deployed on a server equipped with an NVIDIA A100 GPU and 80GB of video memory. The performance metric selected is the first-to-first-to-meta time, with a Service Level Agreement (SLA) limit of 500ms. A preset slope change threshold is also used. Parameters for calculating the overall performance degradation trend, set to 15ms per concurrent connection. Take 3, warning water level threshold. Take 0.75 as the pressure growth rate parameter. Set the value to 10, and preset the convergence accuracy. Take 1.

[0062] At the start of the probing process, the finite state machine is in a safe prediction and pressure-increasing state. Request-sending collection nodes send requests to the large language model inference service in ascending order of concurrency level, and collect the first-term meta-time at each concurrency level. First, baseline sampling is performed at concurrency levels 4 and 8, with first-term meta-times measured at 120ms and 135ms, respectively. Following step S2, the performance degradation slope between adjacent concurrency levels is calculated in the continuous degradation slope and slope change calculation node; the performance degradation slope for the concurrency level interval from 4 to 8 is 3.75ms per concurrency. Subsequently, concurrency levels 12, 16, 20, and 24 are probed sequentially, with first-term meta-times measured at 150ms, 180ms, 250ms, and 320ms, respectively. The corresponding performance degradation slopes are 3.75ms per concurrency, 7.5ms per concurrency, 17.5ms per concurrency, and 17.5ms per concurrency, respectively.

[0063] As the data above shows, in the concurrency level range of 4 to 16, the performance degradation slope curve is gentle, and the large language model inference service is in a low-load safe zone. After the concurrency level exceeds 16, the performance degradation slope jumps from 7.5 ms per concurrent connection to 17.5 ms per concurrent connection, corresponding to a slope change of 10 ms per concurrent connection, indicating that the system has entered the critical transition zone. This slope change does not exceed the preset slope change threshold of 15 ms per concurrent connection, therefore, the prediction for the next concurrency level is executed.

[0064] Following step S3, the last three performance degradation slopes in the performance degradation slope sequence are selected from the trend prediction model node: 7.5ms per concurrent, 17.5ms per concurrent, and 17.5ms per concurrent. The overall performance degradation trend is then calculated. The time per concurrent user is 14.17 ms. The highest detected first-word meta-time at concurrency level 24 is 320 ms, and the service level agreement limit is 500 ms. The absolute remaining space calculated in the basic prediction step size calculation node is 180 ms. Dividing the absolute remaining space of 180 ms by the overall performance degradation trend of 14.17 ms per concurrent user yields the basic prediction step size. It is 12.7, rounded down to 12.

[0065] Following step S4, hardware feedback information at concurrency level 24 is obtained in the hardware comprehensive resource pressure acquisition node. The calculated memory pressure parameter is 0.95, and the computing power pressure parameter is 0.75. Following step S5, the memory pressure parameter 0.95 and the computing power pressure parameter 0.75 are weighted and summed in the comprehensive resource pressure calculation node, with a weighting coefficient of 0.5 for each parameter, to obtain the comprehensive resource pressure parameter. The value is 0.85. This value exceeds the warning water level threshold of 0.75, and the dynamic damping coefficient is calculated in the risk correction factor calculation node. for The result is approximately 2.718. In the step size correction and the next concurrent calculation node, the basic prediction step size of 12 is divided by the dynamic damping coefficient of 2.718, resulting in a corrected concurrency level increment of approximately 4.42, which is rounded down to 4. This correction process reflects that when the underlying key-value cache spikes sharply and reaches the warning level, the system automatically compresses the prediction step size from 12 to 4 through a nonlinear dynamic damping mechanism, avoiding excessively large jumps that could lead to out-of-bounds errors.

[0066] Following the revised concurrency level increment of 4, the system progresses from concurrency level 24 to concurrency level 28. The finite state machine remains in a safe predictive stress state. The request-sending acquisition node sends requests according to the request quantity corresponding to concurrency level 28. The acquisition time for the first metaword at this concurrency level is 512ms, exceeding the Service Level Agreement (SLA) limit of 500ms. At this point, the system accurately triggers an out-of-bounds failure state, treating concurrency level 28 as the out-of-bounds concurrency level. The highest concurrency level (24) among the detected concurrency levels whose performance indicators do not exceed the preset range is taken as the most recent safe concurrency level. It then enters a state of local convergence search.

[0067] Following step S6, perform a local binary search between the out-of-bounds concurrency level 28 and the nearest safe concurrency level 24. Calculate the intermediate concurrency level. The concurrency level is set to 26. A request is sent at concurrency level 26, and the first word time is collected at 478ms, meeting the service level agreement (SLA) requirements. Therefore, the nearest safe concurrency level is updated to 26. The intermediate concurrency level is recalculated as 27, and a request is sent at concurrency level 27. The first word time is collected at 495ms, meeting the SLA requirements. Therefore, the nearest safe concurrency level is updated to 27. At this point, the difference between the out-of-bounds concurrency level 28 and the nearest safe concurrency level 27 is 1, reaching the preset convergence accuracy, and convergence terminates. The finite state machine enters the probe termination state, determining the nearest safe concurrency level 27 as the maximum safe concurrency level.

[0068] This example demonstrates that the large language model inference service, while adhering to the service level agreement that the first word time does not exceed 500ms, has a maximum safe concurrency level of 27. The entire probing process achieves rapid advancement in low-load areas through safe prediction of stress states, triggers risk correction states and automatically compresses step sizes when the overall resource stress parameter exceeds a threshold due to increased memory pressure parameters, and accurately locates critical concurrency levels through out-of-bounds failure states and local convergence search states. No memory overflow or service interruption occurred throughout the entire process.

[0069] Example 2: A concurrent critical point detection system for large language models based on performance degradation slope. This system is used to implement the concurrent critical point detection method for large language models based on performance degradation slope described in Example 1, such as... Figure 2As shown, it includes a request acquisition module, a slope calculation and sorting module, a concurrency level prediction module, a hardware feedback acquisition module, a prediction result correction module, and a detection process control module.

[0070] The system comprises the following modules: a request acquisition module, used to send requests to the large language model inference service according to the number of requests corresponding to each concurrency level under multiple consecutive concurrency levels, and to collect the performance indicators of the large language model inference service under each concurrency level; a slope calculation and sorting module, used to calculate the performance degradation slope between adjacent concurrency levels based on the performance indicators, and to sort the performance degradation slopes into a performance degradation slope sequence in ascending order of concurrency levels; a concurrency level prediction module, used to predict the critical concurrency level or the next concurrency level of the large language model inference service based on the performance degradation slope sequence; a hardware feedback acquisition module, used to acquire hardware feedback information of the large language model inference service under multiple consecutive concurrency levels; a prediction result correction module, used to correct the prediction results based on the hardware feedback information; and a detection process control module, used to control the advancement, rollback, or termination of the concurrency level detection process through a finite state machine based on the corrected prediction results.

[0071] Working Principle: This invention collects performance metrics of a large language model inference service under multiple consecutive concurrency levels, calculates the performance degradation slope between adjacent concurrency levels and arranges them into a performance degradation slope sequence. Based on this sequence, it predicts the critical concurrency level or the next concurrency level. Simultaneously, it obtains hardware feedback information to correct the prediction results and uses a finite state machine to control the advancement, rollback, or termination of the probing process. Performance metrics reflect the current load status from a service quality perspective; the performance degradation slope sequence reveals the performance degradation law with increasing concurrency from a trend perspective; hardware feedback information perceives the real-time pressure on memory and computing power from a resource perspective; and the finite state machine uniformly schedules the probing behavior from a process perspective. The prediction module provides the direction of advancement, the correction module constrains the step size, and the state machine determines when to advance, rollback, or terminate. These three form a closed-loop linkage mechanism of feedforward prediction, feedback correction, and state control. Under the premise of ensuring the safety and controllability of the probing process, it quickly approaches the critical concurrency level, solving the problems of existing fixed-step-size probing methods that are difficult to balance probing efficiency and safety, and are prone to exceeding the critical point, leading to service anomalies.

[0072] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0073] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0074] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0076] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting concurrent critical points in large language models based on performance degradation slope, characterized in that, Includes the following steps: Under multiple consecutive concurrency levels, requests are sent to the large language model inference service according to the number of requests corresponding to each concurrency level, and the performance indicators of the large language model inference service under each concurrency level are collected. Calculate the performance degradation slope between adjacent concurrency levels based on the performance index, and arrange the performance degradation slopes into a performance degradation slope sequence in order of concurrency level from low to high. Predict the critical concurrency level or the next concurrency level of the large language model inference service based on the performance degradation slope sequence; Obtain hardware feedback information of the large language model inference service under the multiple consecutive concurrency levels; The prediction results are corrected based on the hardware feedback information. Based on the revised prediction results, the concurrent level detection process is controlled by a finite state machine to advance, regress, or terminate.

2. The method for detecting concurrent critical points in large language models based on performance degradation slope according to claim 1, characterized in that, The performance metrics include at least one of the following: first word time, inter-word latency, video memory usage, key-value cache usage, and number of words output per second. Each performance metric under each concurrency level is recorded as the average of multiple requests within a statistical window.

3. The method for detecting concurrent critical points in large language models based on performance degradation slope according to claim 1, characterized in that, The performance degradation slope is calculated based on the ratio of the performance index difference between two adjacent concurrency levels to the concurrency level difference; Each performance degradation slope in the performance degradation slope sequence corresponds to a current concurrency level, which is the relatively higher concurrency level among two adjacent concurrency levels involved in the calculation.

4. The method for detecting concurrent critical points in large language models based on performance degradation slope according to claim 3, characterized in that, The step of predicting the critical concurrency level or the next concurrency level of the large language model inference service based on the performance degradation slope sequence includes: The slope change is obtained by performing a difference operation on the last two performance degradation slopes in the performance degradation slope sequence. If the slope change exceeds a preset slope change threshold, the large language model inference service is determined to be close to the critical concurrency level, and the current concurrency level is used as the predicted critical concurrency level. If the slope change does not exceed the preset slope change threshold, the next concurrency level is predicted based on the average of the M performance degradation slopes at the end of the performance degradation slope sequence, where M is a positive integer greater than or equal to 2.

5. The method for detecting concurrent critical points in large language models based on performance degradation slope according to claim 1, characterized in that, The hardware feedback information includes video memory pressure parameters; The video memory pressure parameters are calculated based on the current video memory occupancy rate, the current key-value cache occupancy rate, the key-value cache growth rate, and the key-value cache expansion amount corresponding to the estimated maximum context length. The video memory pressure parameters are monotonically increasing with the current video memory occupancy rate, the current occupancy rate of the key-value cache, the growth rate of the key-value cache, and the expansion amount of the key-value cache.

6. The method for detecting concurrent critical points in large language models based on performance degradation slope according to claim 5, characterized in that, The hardware feedback information also includes computing power pressure parameters; The computing power pressure parameters are calculated based on computing power utilization, number of requests processed per second, and average queuing latency. The computing power pressure parameters are monotonically increasing with the computing power utilization rate, the number of requests processed per second, and the average queuing delay.

7. The method for detecting concurrent critical points in large language models based on performance degradation slope according to claim 6, characterized in that, The prediction result includes the next concurrency level, and the difference between the next concurrency level and the highest concurrency level among the concurrency levels of the currently collected performance indicators is used as the concurrency level increment. When correcting the prediction results based on the hardware feedback information, the memory pressure parameter and the computing power pressure parameter are weighted and summed to obtain the comprehensive resource pressure parameter. The dynamic damping coefficient is calculated based on the comprehensive resource pressure parameters, and the dynamic damping coefficient has a monotonically increasing relationship with the comprehensive resource pressure parameters. The concurrent level increment is reduced using the dynamic damping coefficient, and the reduced value of the concurrent level increment is not less than 1.

8. The method for detecting concurrent critical points in large language models based on performance degradation slope according to claim 7, characterized in that, The finite state machine includes a safe pressurization state, a risk correction state, an out-of-bounds convergence state, and a detection termination state. Under the aforementioned safe pressurization state, the concurrency level is increased according to the modified concurrency level increment; Under the risk correction state, the concurrency level increment is reduced, and the step of predicting the next concurrency level based on the performance degradation slope sequence is re-executed; When the performance index of the large language model inference service exceeds the preset range at the concurrency level during the detection process, the concurrency level at which the detection process is located will be regarded as an out-of-bounds concurrency level. The highest concurrency level among the detected concurrency levels whose performance indicators do not exceed the preset range is taken as the most recent safe concurrency level; And then enter the out-of-bounds convergence state; In the out-of-bounds convergence state, local convergence is performed between the out-of-bounds concurrency level and the nearest safe concurrency level; When the termination condition is met, the probe is entered into the termination state, and the maximum security concurrency level is output.

9. The method for detecting concurrent critical points in large language models based on performance degradation slope according to claim 8, characterized in that, The process of achieving local convergence between the out-of-bounds concurrency level and the nearest safe concurrency level includes: The average of the out-of-bounds concurrency level and the nearest safe concurrency level is used as the intermediate concurrency level; Requests are sent to the large language model inference service according to the number of requests corresponding to the intermediate concurrency level, and the corresponding performance metrics are collected. If the performance metrics at the intermediate concurrency level meet the service level agreement requirements, the intermediate concurrency level is updated to the nearest safe concurrency level; otherwise, the intermediate concurrency level is updated to the out-of-bounds concurrency level. Repeat the process until the difference between the out-of-bounds concurrency level and the nearest safe concurrency level is less than the preset convergence precision, and then determine the nearest safe concurrency level as the maximum safe concurrency level.

10. A concurrent critical point detection system for large language models based on performance degradation slope, characterized in that, include: The request collection module is used to send requests to the large language model inference service according to the number of requests corresponding to each concurrency level under multiple consecutive concurrency levels, and to collect the performance indicators of the large language model inference service under each concurrency level. The slope calculation and arrangement module is used to calculate the performance degradation slope between adjacent concurrency levels based on the performance index, and arrange the performance degradation slope into a performance degradation slope sequence in order of concurrency level from low to high. The concurrency level prediction module is used to predict the critical concurrency level or the next concurrency level of the large language model inference service based on the performance degradation slope sequence. The hardware feedback acquisition module is used to acquire hardware feedback information of the large language model inference service under the multiple consecutive concurrency levels. The prediction result correction module is used to correct the prediction result based on the hardware feedback information; The detection process control module is used to control the advancement, reversal, or termination of the concurrent level detection process through a finite state machine based on the corrected prediction results.