Request batch processing method and electronic equipment

Abstract

The invention discloses a request batch processing method and electronic equipment, and relates to the technical field of computers. When a batch processing object meets a first batch processing optimization condition, the current comprehensive error of the batch processing object is determined according to the performance requirement of the batch processing object in the current application scene and the current actual performance; when a first batch processing optimization condition is satisfied, carrying out first adjustment on the size of the current batch to obtain a first target batch size, and when a second batch processing optimization condition is satisfied, carrying out second adjustment on the size of the current batch according to the length change information of the request waiting queue to obtain a second target batch size; and finally, performing batch processing on the to-be-processed request according to the first target batch size or the second target batch size. The size of the current batch is adjusted according to the performance requirement of the application scene and the current actual performance expression or the length change information of the request waiting queue, so that the size of the target batch can adapt to the current application scene and the dynamically changing load, and the waste of hardware resources and the increase of user request response delay are avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and particularly relates to a request batch processing method and an electronic device. BACKGROUND

[0002] With the continuous development of pre-training language models, the model parameter scale grows exponentially, the memory occupation of single request inference is high, and the utilization rate of computing resources is low, so batch processing technology is often used to batch process multiple requests.

[0003] In related technologies, a static batch processing method is usually used, that is, a fixed batch size is used. However, the application scenarios and loads of pre-training language models are dynamically changing, different application scenarios have different performance requirements for batch processing, and in the load surge working condition, if the fixed batch size is too small, that is, the number of requests per batch is small, not only the hardware resources will be wasted, but also a large number of user requests will be in timeout waiting, and if the fixed batch size is too large, the response delay of user requests will increase. SUMMARY

[0004] The present application provides a request batch processing method and an electronic device to at least solve the problem that the fixed batch size in related technologies may cause waste of hardware resources or lead to an increase in response delay of user requests.

[0005] The present application provides a request batch processing method, comprising: obtaining performance requirements of a batch processing object in a current application scenario, current actual performance, request waiting queue length change information, and a current batch size; in a case where the batch processing object meets a first batch processing optimization condition and does not meet a second batch processing optimization condition, determining a current comprehensive error of the batch processing object according to the performance requirements of the batch processing object in the current application scenario and the current actual performance; performing a first adjustment on the current batch size according to the current comprehensive error of the batch processing object to obtain a first target batch size; in a case where the batch processing object does not meet the first batch processing optimization condition but meets the second batch processing optimization condition, performing a second adjustment on the current batch size according to the request waiting queue length change information to obtain a second target batch size; wherein the adjustment step length of the second adjustment is greater than the adjustment step length of the first adjustment; batch processing the requests to be processed in the request waiting queue according to the first target batch size or the second target batch size.

[0006] The present application also provides a request batch processing device, comprising: The acquisition module is configured to acquire performance requirements of the batch processing object under a current application scenario, current actual performance, request waiting queue length change information, and a current batch size; The determination module is configured to determine a current comprehensive error of the batch processing object according to the performance requirements of the batch processing object under the current application scenario and the current actual performance, in a case where the batch processing object meets the first batch processing optimization condition and does not meet the second batch processing optimization condition. The first adjustment module is configured to perform first adjustment on the current batch size according to the current comprehensive error of the batch processing object, to obtain a first target batch size. The second adjustment module is configured to perform second adjustment on the current batch size according to the request waiting queue length change information, to obtain a second target batch size, in a case where the batch processing object does not meet the first batch processing optimization condition but meets the second batch processing optimization condition, wherein an adjustment step length of the second adjustment on the batch size is greater than an adjustment step length of the first adjustment on the batch size. The batch processing module is configured to perform batch processing on the to-be-processed requests in the request waiting queue according to the first target batch size or the second target batch size.

[0007] The application further provides an electronic device, including a memory configured to store a computer program, and a processor configured to execute the computer program to implement the steps of any of the request batch processing methods.

[0008] The application further provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of any of the request batch processing methods.

[0009] The application further provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the steps of any of the request batch processing methods.

[0010] According to the application, when the batch processing object meets the first batch processing optimization condition, the current comprehensive error of the batch processing object is determined according to the performance requirements of the batch processing object under the current application scenario and the current actual performance, and the first adjustment is performed on the current batch size to obtain the first target batch size, when the second batch processing optimization condition is met, the second adjustment is performed on the current batch size according to the request waiting queue length change information to obtain the second target batch size, and finally the to-be-processed requests are batch processed according to the first target batch size or the second target batch size. The current batch size is adjusted according to the performance requirements of the application scenario and the current actual performance, or the request waiting queue length change information, so that the target batch size can adapt to the current application scenario and the dynamically changing load, and the waste of hardware resources and the increase of user request response delay are avoided. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings described in the following embodiments are only some of the embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0012] Figure 1 The structure schematic diagram of the request batch processing system based on the embodiments of the present application is provided. Figure 2 The flow schematic diagram of the request batch processing method provided by the embodiments of the present application is provided. Figure 3 The flow schematic diagram of the exemplary request batch processing method provided by the embodiments of the present application is provided. Figure 4 The structure schematic diagram of the request batch processing device provided by the embodiments of the present application is provided. Figure 5 The structure schematic diagram of the electronic device provided by the embodiments of the present application is provided. DETAILED DESCRIPTION

[0013] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort fall within the protection scope of the present application.

[0014] It should be noted that, in the description of the present application, the terms “include”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0015] Pre-trained language models, also known as large language models (LLM), are deep learning models with parameter sizes reaching hundreds of millions or even tens of billions. Their core features include large parameter sizes, such as GPT-3 with 175 billion parameters, and the ability to capture complex language patterns through massive data training. The data requirements are large, with training data typically covering internet text, books, code, and other multi-source information, with a size of up to TB. Therefore, the task complexity, processing time, and dynamic resource requirements of large model training jobs make the scheduling problem of GPU clusters even more prominent. However, as the model parameter size grows exponentially, issues such as high memory usage for single request inference and low computational resource utilization have become prominent, leading to high inference costs and severely restricting large-scale deployment.

[0016] Large model batching technology (Batching) dynamically combines concurrent requests, packages multiple independent inference tasks, and then sends them to the GPU for computation and execution. Batch size represents how many samples are processed in parallel in each training step. Assuming the batch size is 32, the neural network will process 32 independent samples simultaneously in each training step. By batching multiple online requests, the efficiency of GPU usage can be improved.

[0017] In related technologies, static batching is usually used, that is, a fixed batch size is used. However, the application scenarios and loads of pre-trained language models are dynamically changing, and different application scenarios have different performance requirements for batching. In the case of a sudden surge in load, if the fixed batch size is too small, that is, the number of requests per batch is small, not only will it waste hardware resources, but it will also cause a large number of user requests to wait for timeout. If the fixed batch size is too large, it will cause the response delay of user requests to grow.

[0018] In order to solve the above technical problems, the embodiments of the present application provide a request batch processing method and an electronic device. The method comprises the following steps: acquiring performance requirements of a batch processing object in a current application scenario, current actual performance, request waiting queue length change information and a current batch size; in the case that the batch processing object meets first batch processing optimization conditions and does not meet second batch processing optimization conditions, determining a current comprehensive error of the batch processing object according to the performance requirements of the batch processing object in the current application scenario and the current actual performance; performing first adjustment on the current batch size according to the current comprehensive error of the batch processing object, so as to obtain a first target batch size; in the case that the batch processing object does not meet the first batch processing optimization conditions but meets the second batch processing optimization conditions, performing second adjustment on the current batch size according to the request waiting queue length change information, so as to obtain a second target batch size; wherein the adjustment step length of the second adjustment on the batch size is greater than the adjustment step length of the first adjustment on the batch size; and performing batch processing on the to-be-processed requests in the request waiting queue according to the first target batch size or the second target batch size. The method provided by the above scheme can determine the current comprehensive error of the batch processing object according to the performance requirements of the batch processing object in the current application scenario and the current actual performance when the batch processing object meets the first batch processing optimization conditions, and can perform first adjustment on the current batch size to obtain a first target batch size. When the second batch processing optimization conditions are met, the second adjustment is performed on the current batch size according to the request waiting queue length change information to obtain a second target batch size. Finally, the to-be-processed requests are batch processed according to the first target batch size or the second target batch size. According to the performance requirements of the application scenario and the current actual performance, or the request waiting queue length change information, the current batch size is adjusted, so that the target batch size can adapt to the current application scenario and the dynamically changing load, and the waste of hardware resources and the increase of user request response delay are avoided. In order to enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0019] In combination with the specific application environment architecture or the specific hardware architecture on which the execution of the request batch processing method depends, the specific application environment architecture or the specific hardware architecture is described herein.

[0020] Firstly, the structure of the request batch processing system based on the present application is described. The request batch processing method and the electronic device provided by the embodiments of the present application are suitable for dynamically adjusting the batch size according to the changes of the application scenario and the load, such as Figure 1The diagram shown is a schematic of the request batch processing system based on the embodiments of this application. It mainly includes a request batch processing device and a batch processing object. The batch processing object is used to batch process requests from the pre-trained language model. The request batch processing device adjusts the batch size of the batch processing object when performing request batch processing based on the request batch processing method provided in the embodiments of this application.

[0021] This application provides a request batch processing method for dynamically adjusting batch size based on changes in application scenarios and load. The execution subject of this application embodiment is an electronic device, such as a server, desktop computer, laptop computer, tablet computer, or other electronic devices that can be used for request batch processing.

[0022] like Figure 2 The diagram shown is a flowchart of a request batch processing method provided in an embodiment of this application. The method includes: Step 201: Obtain the performance requirements of the batch processing object in the current application scenario, the current actual performance, the change information of the request waiting queue length, and the current batch size.

[0023] Specifically, performance requirements for application scenarios include requirements for inference latency, GPU utilization, first response time, cache hit rate, and memory usage. For example, in a chat application scenario, performance requirements include inference latency not exceeding 150ms, GPU utilization not less than 85%, memory usage not less than 90%, first response time not exceeding 500ms, and cache hit rate not less than 70%. In practical applications, different performance requirement standards can be set according to different application scenarios. Current actual performance is represented by the numerical values ​​of batch processing objects in terms of inference latency, GPU utilization, first response time, cache hit rate, and memory usage at the current moment.

[0024] Accordingly, by obtaining the performance requirements and actual performance of the batch processing object in the current application scenario, the gap between the current state and the ideal state of the batch processing object can be obtained, providing direction for subsequent adjustments to the target batch size. Real-time monitoring of the number of pending requests in the request waiting queue using request waiting queue length changes allows for timely adjustments to the batch size based on these changes.

[0025] Step 202: If the batch processing object meets the first batch processing optimization conditions but does not meet the second batch processing optimization conditions, determine the current comprehensive error of the batch processing object based on the performance requirements and actual performance of the batch processing object in the current application scenario.

[0026] Specifically, the first batch processing optimization condition is that the calculation of the current comprehensive error of the batch processing object is performed periodically, that is, when a corresponding time period is reached, it is determined that the batch processing object meets the first batch processing optimization condition, and the current comprehensive error represents the gap between the actual overall performance of the batch processing object and the performance requirement of the batch processing object in the current application scenario.

[0027] Correspondingly, by periodically calculating the current comprehensive error, the adaptation degree of the current batch size to the current application scenario of the batch processing object is quantified, and the current comprehensive error provides an adjustment direction and data reference for the subsequent target batch size.

[0028] Step 203, according to the current comprehensive error of the batch processing object, the first adjustment is performed on the current batch size to obtain the first target batch size.

[0029] Specifically, the current comprehensive error includes positive deviation and negative deviation, and the closer the current comprehensive error is to 0, the closer the actual performance indicators of the batch processing object are to the performance requirements of the batch processing object in the current application scenario, and the smaller the adjustment degree is.

[0030] Step 204, in the case that the batch processing object does not meet the first batch processing optimization condition but meets the second batch processing optimization condition, according to the request waiting queue length change information, the second adjustment is performed on the current batch size to obtain the second target batch size; wherein the batch size adjustment step of the second adjustment is greater than the batch size adjustment step of the first adjustment; Specifically, the request waiting queue length is the number of pending requests contained in the request queue, when the number of pending requests changes, the request queue length changes, and the change size of the number of pending requests in the request queue is the request waiting queue length change information. The second batch processing optimization condition is that the number of pending requests in the request queue increases suddenly, in this case, the target batch size is adjusted to the second target batch size. The step refers to the different batch size growth rates, and the batch size growth rate of the second adjustment is greater than the batch size adjustment growth rate of the first adjustment, so as to quickly shorten the request waiting queue length and shorten the response delay of user requests.

[0031] Correspondingly, according to the request waiting queue length change information, the workload characteristics are automatically identified, which represents the change of the number of pending requests in the request waiting queue, so that when the load increases suddenly, the batch size can be adjusted in time, the pending requests can be processed in time, the response delay of user requests is reduced, and the utilization rate of hardware resources is improved.

[0032] Step 205, according to the first target batch size or the second target batch size, the pending requests in the request waiting queue are batch processed.

[0033] Specifically, the current batch size is adjusted according to the first target batch size or the second target batch size, and a corresponding number of pending requests in the request waiting queue are extracted for batch processing according to the first target batch size or the second target batch size.

[0034] Correspondingly, by selecting different numbers of pending requests for batch processing according to the first target batch size or the second target batch size, the actual performance of the current batch processing object is made to conform to the performance requirement of the application scenario, thereby improving the utilization of resources and shortening the response delay of user requests.

[0035] On the basis of the above embodiments, as a kind of implementable mode, in an embodiment, according to the performance requirement of batch processing object in current application scenario and current actual performance, the current comprehensive error of batch processing object is determined, including: Step 2021, according to the performance requirement of batch processing object in current application scenario and current actual performance, it is judged whether there is early warning performance index; Step 2022, in the case where there is early warning performance index, according to the current performance optimization demand represented by early warning performance index, target error calculation weight set is determined; Step 2023, according to the performance requirement of batch processing object in current application scenario, current actual performance and target error calculation weight set, the current comprehensive error of batch processing object is determined.

[0036] Specifically, the early warning includes delay overrun, GPU low utilization and preprocessing bottleneck, etc., and a plurality of performance index types are determined according to the performance requirement of batch processing object in current application scenario. The target error calculation weight set is adjusted according to the current performance optimization demand represented by the early warning performance index. Meanwhile, the actual values of a plurality of performance indexes in current actual performance are obtained, and when the actual value of the performance index exceeds or is less than the preset threshold, the early warning is represented, and there is early warning performance index.

[0037] For example, when the inference delay exceeds the preset delay threshold, the early warning of delay overrun is represented, so the index weight coefficients corresponding to inference delay, GPU utilization, first response time, cache hit rate and memory occupation are adjusted;When GPU utilization is lower than the preset utilization threshold, the early warning of GPU low utilization is represented, so the index weight coefficients corresponding to inference delay, GPU utilization and first response time are adjusted;When the proportion of user first response time in inference delay represents whether preprocessing bottleneck occurs, when the proportion is greater than the preset bottleneck threshold, the preprocessing bottleneck early warning is represented, so the index weight coefficients corresponding to inference delay, GPU utilization and first response time are adjusted. The adjusted weight indexes are determined as the target error calculation weight set.

[0038] Accordingly, by processing the performance requirements of the batch processing object in the current application scenario and the current actual performance, it is determined in a timely manner whether there is a performance warning index, and then the target error calculation weight set can be adjusted according to the performance warning index, so as to realize real-time adjustment of the influence of multiple performance indexes on the current comprehensive error according to the actual performance, and improve the authenticity of the current comprehensive error.

[0039] Specifically, in an embodiment, target values of multiple performance indexes can be determined according to the performance requirements of the batch processing object in the current application scenario; current actual values of the multiple performance indexes can be determined according to the current actual performance of the batch processing object in the current application scenario; for any performance index, an error value of the performance index can be determined according to the deviation degree between the target value and the current actual value of the performance index; and a current comprehensive error of the batch processing object can be obtained by weighted calculation of the error values of the multiple performance indexes according to a target error calculation weight set. The target error calculation weight set includes weight coefficients corresponding to the performance indexes.

[0040] Specifically, the error value is determined by the ratio of the deviation value between the target value and the current actual value of the performance index to the target value. The target values of the multiple performance indexes are set by deduction under the premise of ensuring user experience. User experience refers to the user's feeling of smoothness and naturalness, and a strict deduction process is followed.

[0041] For example, smoothness and naturalness are quantified by a delay lower than the human perception threshold, and the value is usually determined to be less than 200 ms. The 200 ms total delay budget is then allocated to each link of the batch processing object, including network, preprocessing, inference and post-processing, etc. Inference is the most time-consuming part, and may be allocated to 150 ms. In order to complete the inference within 150 ms, hardware efficiency needs to be maximized. According to multiple experiments, when about 85% of GPU utilization and about 90% of video memory occupancy can achieve this performance target, while taking into account the cost. Then introduce optimization techniques such as caching, and set a challenging but achievable caching target according to the business characteristics, for example, 70%. Finally, fine-tune according to the improvement of user experience and resource efficiency.

[0042] Accordingly, the error value determined by the target value and the current actual value of the multiple performance indexes quantifies the deviation degree between the ideal state and the actual state of each performance index, providing data support for determining the current comprehensive error of the batch processing object.

[0043] Specifically, in an embodiment, the current comprehensive error of the batch processing object can be determined based on the following formula:

[0044] wherein, represents a current comprehensive error of a batch processing object, the target error calculation weight set includes 、 、 、 and , represents an error value of a batch processing delay index, represents a current actual value of the batch processing delay index, represents a target value of the batch processing delay index, represents an error value of a computing resource utilization rate, represents a current actual value of the computing resource utilization rate, represents a target value of the computing resource utilization rate, represents an error value of a request response delay index, represents a current actual value of the request response delay index, represents a target value of the request response delay index, represents an error value of a request processing efficiency, represents a current actual value of the request processing efficiency, represents a target value of the request processing efficiency, represents an error value of a memory occupancy rate, represents a current actual value of the memory occupancy rate, represents a target value of the memory occupancy rate, the performance index includes the batch processing delay index, the computing resource utilization rate, the request response delay index, the request processing efficiency, and the memory occupancy rate.

[0045] Specifically, among the plurality of performance index types, an inference delay is quantified by a P99 end-to-end delay, which corresponds to the batch processing delay index, wherein the P99 end-to-end delay refers to a total processing time from start to end of 99% of all processed requests being lower than the value. A GPU utilization rate is quantified by a stream multiprocessor activity rate (SM unit activity rate), which corresponds to the computing resource utilization rate. A first response time is quantified by a user response waiting time, which corresponds to the request response delay index. A cache hit rate is quantified by a number of processed requests per second, which corresponds to the request processing efficiency. A memory occupancy is quantified by a GPU video memory usage rate, which corresponds to the memory occupancy rate.

[0046] Specifically, the smaller the current actual value and the target value of the batch processing delay index represent the faster the speed of processing the pending requests of a batch and the better the performance. The larger the current actual value and the target value of the resource utilization rate represent that the GPU resources are fully called and the batch size is set more reasonably. The smaller the current actual value and the target value of the request response delay index represent the faster the speed of responding to a pending request and the better the performance. The larger the current actual value and the target value of the request processing efficiency represent the higher the cache hit rate, the faster the speed of reading data, and the faster the processing speed of the pending request. The larger the current actual value and the target value of the memory occupancy rate represent the higher the resource utilization rate and the larger the throughput. When the memory occupancy rate approaches the limit, it also represents that a memory overflow may be triggered under the current batch size.

[0047] Specifically, the basic weight set of the target error calculation weight set is usually When the delay is over the limit, the target error calculation weight set is usually When the GPU utilization is low, the target error calculation weight set is usually When the preprocessing bottleneck is warned, the target error calculation weight set is usually .

[0048] Correspondingly, different warning conditions represent different problems of the actual state of the batch processing object under the current application scenario. By adjusting the target error calculation weight set, the weight of the problem index is improved, and then it is reflected in the current comprehensive error, so that the first adjustment according to the current comprehensive error solves the problem, and the current actual performance of the batch processing object is more in line with the performance requirements under the current application scenario.

[0049] On the basis of the above embodiment, as a kind of implementable mode, in an embodiment, according to the current comprehensive error of batch processing object, the first adjustment is carried out to the current batch size, to obtain first target batch size, including: Step 2031, according to the error adjustment demand represented by the current comprehensive error of batch processing object, determine target first adjustment coefficient set; Step 2032, according to the current comprehensive error of batch processing object and target first adjustment coefficient set, determine the first adjustment value of batch size; Step 2033, the first adjustment value of batch size is added to the current batch size, to obtain first target batch size.

[0050] Specifically, the target first adjustment coefficient includes a speed coefficient, a static coefficient and a time coefficient, and the target first adjustment coefficient is adjusted according to the current comprehensive error. The speed coefficient determines the rapidity of the batch processing object, the greater the speed coefficient, the faster the response speed of the batch processing object, and the higher the adjustment accuracy, but the excessive speed coefficient leads to instability of the batch processing object, and the small speed coefficient reduces the adjustment accuracy. The static coefficient eliminates the static error of the batch processing object, and the excessive static coefficient increases the oscillation frequency of the batch processing object, and the small static coefficient reduces the adjustment accuracy of the batch processing object. The time coefficient improves the dynamic characteristics of the batch processing object, and the excessive time coefficient leads to large overshoot and long adjustment time of the batch processing object, and the small time coefficient also leads to large overshoot and long adjustment time.

[0051] Specifically, when the actual value is greater than the target value, the current comprehensive error value is a positive deviation, and when the actual value is less than the target value, the current comprehensive error value is a negative deviation, and the value range of the current comprehensive error is When the current comprehensive error value is a positive deviation, the current actual value representing the plurality of performance indicators is greater than the performance requirement, and the error adjustment requirement is to reduce the current batch size; when the current comprehensive error value is a negative deviation, the current actual value representing the plurality of performance indicators is less than the performance requirement, and the error adjustment requirement is to increase the current batch size.

[0052] Correspondingly, the error adjustment requirement represents the situation between the current actual value and the target value, and then the positive deviation or the negative deviation represented by the error adjustment requirement is used to determine the adjustment direction of the current batch size.

[0053] Specifically, in an embodiment, the first adjustment value of the batch size can be determined based on the following formula:

[0054] wherein, the first adjustment value of the batch size is represented by ΔN, the target first adjustment coefficient set includes , and , the speed coefficient is represented by Kp, the static coefficient is represented by Ki, the time coefficient is represented by Kd, the current comprehensive error is represented by e(t), the cumulative information of the comprehensive error from the initial time to the current time is represented by e(t-1), and the change rate of the current comprehensive error is represented by de / dt.

[0055] Specifically, the target first adjustment coefficient set usually takes the value of , , .

[0056] Specifically, in an embodiment, the first target batch size can be determined based on the following formula:

[0057] wherein, represents the first target batch size after the first adjustment of the current batch size, represents the current batch size, represents the first adjustment value of the batch size. After the first adjustment value is determined, the first target batch size is determined according to the first adjustment value. Finally, the first target batch size is constrained to ensure that it is between the minimum allowed batch size and the maximum allowed batch size.

[0058] Correspondingly, the speed coefficient enables the current batch size to be quickly adjusted when it needs to be adjusted, the static coefficient ensures the adjustment accuracy of the first target batch size, and the time coefficient enables the current batch size to be adjusted steadily.

[0059] Specifically, in an embodiment, the static coefficient is frozen when the first target batch size reaches a preset upper limit value and the current comprehensive error of the batch processing object indicates that the current actual performance is higher than the performance requirement, or when the first target batch size reaches a preset lower limit value and the current comprehensive error of the batch processing object indicates that the current actual performance is lower than the performance requirement.

[0060] Specifically, the preset upper limit value is the maximum allowed batch size, which is determined according to the GPU memory capacity, and the preset lower limit value is the minimum allowed batch size, which is usually 1. When the current comprehensive error of the batch processing object indicates that the current actual performance is higher than the performance requirement, the current first target batch size does not meet the performance requirement and needs to be further increased. However, since the first target batch size has reached the preset upper limit value, i.e., the limit of the memory capacity of the hardware device, the static coefficient is frozen to avoid further expanding the first target batch size. When the current comprehensive error of the batch processing object indicates that the current actual performance is lower than the performance requirement, the current first target batch size is greater than the performance requirement and needs to be further reduced. However, since the first target batch size has reached the preset lower limit value, i.e., the minimum value, the static coefficient is frozen to avoid further reducing the first target batch size.

[0061] Correspondingly, when the batch processing object reaches the physical limit, the past errors of the batch processing object are eliminated by freezing the static coefficient, the accumulated summation is eliminated, the steady-state error is eliminated, the adjustment time is shortened, and the control precision is improved. Avoid the first target batch size to increase indefinitely, and then cause the batch processing object to collapse, also avoid the first target batch size to decrease indefinitely, causing the batch processing object to stop processing the pending request, ensuring the rationality of the first target batch size adjustment.

[0062] Specifically, in an embodiment, when the request waiting queue length change information indicates that the request waiting queue has a request backlog, it is determined that the batch processing object meets the second batch processing optimization condition.

[0063] Specifically, when the number of pending requests in the request waiting queue suddenly increases, but the current batch size cannot meet the growth rate of the increased number of pending requests, the request waiting queue has a request backlog, and it is determined that the batch processing object meets the second batch processing optimization condition, the second adjustment is performed, and the second target batch size is determined.

[0064] Correspondingly, by using the request waiting queue length change information to represent the number of pending requests in the request waiting queue, the batch processing object can discover the request backlog in advance according to the number change, and then perform the second adjustment in time to determine the second target batch size. In the scenario of traffic surge, the cluster utilization can be effectively improved.

[0065] On the basis of the above-mentioned embodiments, as a kind of implementable mode, in an embodiment, according to request waiting queue length change information, the second adjustment is carried out to current batch size, to obtain second target batch size, including: Step 2051, according to request waiting queue length change information, determine queue length growth factor; Step 2052, according to queue length growth factor, the second adjustment is carried out to current batch size, to obtain second target batch size.

[0066] Specifically, based on the following formula, the queue length growth factor is determined:

[0067] Wherein, Indicate the queue length growth factor, Indicate the number of pending requests in the changed request waiting queue length, Indicate the number of pending requests in the changed request waiting queue length.

[0068] For example, when the number of pending requests in the request waiting queue increases from 10 to 100, the request waiting queue has a request backlog, and the queue length growth factor is:

[0069] The second target batch size is determined based on the following formula:

[0070] in, This represents the second target batch size after adjusting the current batch size based on the queue length growth factor. Indicates the current batch size. This represents the queue length growth factor.

[0071] Accordingly, the current batch size is steadily adjusted by using a queue length growth factor. At the same time, boundary constraints are applied by multiplying the current batch size by the queue length growth factor, thereby improving the stability of batch processing objects.

[0072] For example, such as Figure 3 The diagram illustrates an exemplary request batch processing method provided in this application. When a batch processing object meets the first batch processing optimization conditions but not the second batch processing optimization conditions, target values ​​and current actual values ​​of multiple performance indicators are determined based on the performance requirements and actual performance of the batch processing object in the current application scenario. This leads to the determination of error values ​​for multiple performance indicators. A weight set is calculated based on the target error, and the error values ​​of the multiple performance indicators are weighted to obtain the current comprehensive error of the batch processing object—a multi-target weighted fusion. A first adjustment coefficient set is then determined using the current comprehensive error, leading to the determination of a first adjustment value for the batch size (i.e., the PID controller). A first target batch size is obtained based on the first adjustment value, and the pending requests in the request waiting queue are batch-processed according to the first target batch size. When a batch processing object does not meet the first batch processing optimization conditions but meets the second batch processing optimization conditions, a queue length growth factor is determined using a feedforward controller based on the request waiting queue length change information. The current batch size is then adjusted a second time based on the queue length growth factor to obtain the second target batch size. The pending requests in the request waiting queue are batch-processed according to the second target batch size, and then inference tests are performed based on the determined target batch size.

[0073] For example, taking DeepSeek-R1-Distill-Qwen-32B as an example, it is deployed on an 8-GPU A100 node using the VLLM framework. The performance requirements in the current application scenario are an inference latency of 150ms, GPU utilization of no less than 85%, VRAM utilization of no less than 90%, initial response time of approximately 500ms, and cache hit rate of no less than 70%. The current actual performance is as follows: For a fixed batch size of 16, the latency is 230ms, and the utilization is 78%. The multi-objective PID optimization process is as follows: Round 1: High latency detected, weight on latency, batch reduced to 12; Round 2: Latency improved (195ms), but utilization dropped to 75%, increase utilization weight; Round 3: Batch increased to 14, latency 205ms, utilization 82%; Round 4: Predicted request peak, feedforward control reduced to 11 in advance; Steady state: Batch 13-15 dynamically adjusted, latency 190±10ms, utilization 86%.

[0074] The present application can also be applied to various model architectures such as Transformer, CNN or RNN.

[0075] Specifically, in an embodiment, when the batch processing object meets the first batch processing optimization condition and meets the second batch processing optimization condition, the first target batch size and the second target batch size are determined respectively, the third target batch size is an adjustment value of the final current batch size, the first target batch size and the second target batch size are weighted and calculated to determine the third target batch size, and the third target batch size is used to batch process the pending requests in the request waiting queue.

[0076] Based on the following formula, the third target batch size is determined:

[0077] wherein, the third target batch size is represented by T, the first target batch size after the first adjustment of the current batch size is represented by T1, the second target batch size after the second adjustment of the current batch size is represented by T2, and are the weight coefficients of the first target batch size and the second target batch size respectively, and in general cases, , .

[0078] The method for request batch processing provided in the embodiments of the present application comprises the following steps: obtaining performance requirements of a batch processing object in a current application scenario, current actual performance, request waiting queue length change information and a current batch size; in the case that the batch processing object meets first batch processing optimization conditions and does not meet second batch processing optimization conditions, determining a current comprehensive error of the batch processing object according to the performance requirements of the batch processing object in the current application scenario and the current actual performance; performing first adjustment on the current batch size according to the current comprehensive error of the batch processing object, so as to obtain a first target batch size; in the case that the batch processing object does not meet the first batch processing optimization conditions but meets the second batch processing optimization conditions, performing second adjustment on the current batch size according to the request waiting queue length change information, so as to obtain a second target batch size; wherein the adjustment step length of the second adjustment on the batch size is greater than the adjustment step length of the first adjustment on the batch size; and performing batch processing on the to-be-processed requests in the request waiting queue according to the first target batch size or the second target batch size. The method provided in the above scheme can determine the current comprehensive error of the batch processing object according to the performance requirements of the batch processing object in the current application scenario and the current actual performance when the batch processing object meets the first batch processing optimization conditions, and can perform first adjustment on the current batch size to obtain a first target batch size, and can perform second adjustment on the current batch size according to the request waiting queue length change information when the batch processing object meets the second batch processing optimization conditions, to obtain a second target batch size, and finally can perform batch processing on the to-be-processed requests according to the first target batch size or the second target batch size. The current batch size is adjusted according to the performance requirements of the application scenario and the current actual performance, or the request waiting queue length change information, so that the target batch size can adapt to the current application scenario and the dynamically changing load, thereby avoiding the waste of hardware resources and the increase of the user request response delay.

[0079] Moreover, by obtaining the performance requirement of the batch processing object under the current application scenario and the current actual performance, the gap between the current state and the ideal state of the batch processing object can be obtained, providing a direction for subsequent adjustment of the target batch size. By monitoring the number of pending requests in the request waiting queue through the request waiting queue length change information, the batch size can be adjusted in a timely manner according to the request waiting queue length change. By periodically calculating the current comprehensive error, the adaptation degree of the current batch size to the current application scenario of the batch processing object is quantified, and the current comprehensive error provides a direction and data reference for subsequent adjustment of the target batch size. Through the current comprehensive error, the trade-off of multiple indicators is realized, the batch size is automatically reduced to ensure the delay under high load, and the batch size is automatically increased to improve the throughput under low load, achieving dynamic balance between throughput and delay. According to the request waiting queue length change information, the workload characteristics are automatically identified, representing the change in the number of pending requests in the request waiting queue, so that the batch size can be adjusted in a timely manner when the load suddenly increases or decreases, and the pending requests can be processed in a timely manner, reducing the response delay of user requests and improving the utilization of hardware resources. By selecting different numbers of pending requests for batch processing according to the first target batch size or the second target batch size, the actual performance of the current batch processing object and the performance requirement of the application scenario can be met, thereby improving the utilization of resources and shortening the response delay of user requests. By the performance requirement of the batch processing object under the current application scenario and the current actual performance, it is determined in a timely manner whether there is a warning performance indicator, and then the target error weight set can be adjusted according to the warning performance indicator, realizing real-time adjustment of the influence of multiple performance indicators on the current comprehensive error according to the actual performance, and improving the authenticity of the current comprehensive error.

[0080] Meanwhile, the error value determined by the target value and the current actual value of the plurality of performance indicators quantifies the deviation degree between the ideal state and the actual state of each performance indicator, and provides data support for determining the current comprehensive error of the batch processing object. Different early warning situations reflect different problems of the actual state of the batch processing object in the current application scenario. By adjusting the target error calculation weight set, the weight of the problem indicator is improved, and then it is reflected in the current comprehensive error, so that the first adjustment according to the current comprehensive error solves the problem, and the current actual performance of the batch processing object is more in line with the performance requirements in the current application scenario. The error adjustment requirement represents the situation between the current actual value and the target value, and then the adjustment direction of the current batch size is judged according to the positive deviation or negative deviation represented by the error adjustment requirement. Through the speed coefficient, the current batch size can be quickly responded when it needs to be adjusted, and the static coefficient guarantees the adjustment accuracy of the first target batch size, and the time coefficient makes the current batch size can be adjusted steadily. When the batch processing object reaches the physical limit, the static coefficient is frozen, the accumulated summation of the past error of the batch processing object is eliminated, the steady-state error is eliminated, the adjustment time is shortened, and the control precision is improved. Avoiding the first target batch size from increasing indefinitely, which may cause the batch processing object to collapse, and avoiding the first target batch size from decreasing indefinitely, which may cause the batch processing object to stop processing the pending requests, the rationality of the adjustment of the first target batch size is guaranteed. The request waiting queue length change information represents the number of pending requests in the request waiting queue, so that the batch processing object can discover that the request backlog has occurred in advance according to the number change, and then perform the second adjustment in time to determine the second target batch size. In the scene of traffic surge, the cluster utilization rate can be effectively improved. The queue length growth factor is used to realize the steady adjustment of the current batch size, and the current batch size is multiplied by the queue length growth factor to perform boundary constraint, thereby improving the stability of the batch processing object.

[0081] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment.

[0082] Embodiments of the present application also provide a request batch processing device for executing the request batch processing method provided by the above embodiments.

[0083] As shown in Figure 4 Fig. 1 is a structural schematic diagram of a request batch processing device provided by an embodiment of the present application. The request batch processing device 40 comprises an acquisition module 401, a determination module 402, a first adjustment module 403, a second adjustment module 404 and a batch processing module 405.

[0084] The obtaining module is configured to obtain performance requirements of the batch processing object in the current application scenario, current actual performance, request waiting queue length change information, and a current batch size. The determining module is configured to determine a current comprehensive error of the batch processing object according to the performance requirements of the batch processing object in the current application scenario and the current actual performance, in a case where the batch processing object meets the first batch processing optimization condition and does not meet the second batch processing optimization condition. The first adjusting module is configured to perform first adjustment on the current batch size according to the current comprehensive error of the batch processing object, to obtain a first target batch size. The second adjusting module is configured to perform second adjustment on the current batch size according to the request waiting queue length change information, to obtain a second target batch size, in a case where the batch processing object does not meet the first batch processing optimization condition but meets the second batch processing optimization condition, wherein a batch size adjustment step of the second adjustment is greater than a batch size adjustment step of the first adjustment. The batch processing module is configured to perform batch processing on the to-be-processed requests in the request waiting queue according to the first target batch size or the second target batch size.

[0085] The description of the features in the embodiments of the request batch processing apparatus can refer to the related description of the embodiments of the request batch processing method, which will not be repeated here.

[0086] Embodiments of the present application also provide an electronic device, as shown in the accompanying drawings, which is a structural schematic diagram of the electronic device provided by the embodiments of the present application, comprising a processor 10 and a memory 20, wherein the memory 20 stores a computer program, and the processor 10 is configured to run the computer program to execute the steps in any of the embodiments of the request batch processing method described above. Figure 5

[0087] Embodiments of the present application also provide a computer readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps in any of the embodiments of the request batch processing method described above when running.

[0088] In an exemplary embodiment, the computer readable storage medium described above can include but is not limited to: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0089] ​The embodiment of the present application further provides a computer program product, the computer program product comprising a computer program, the computer program being executed by a processor to implement the steps in any of the request batch processing method embodiments.

[0090] The embodiment of the present application further provides another computer program product, comprising a non-volatile computer readable storage medium, the non-volatile computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the steps in any of the request batch processing method embodiments.

[0091] Those skilled in the art will further appreciate that the units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the various examples have been described generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0092] The above describes in detail a request batch processing method and an electronic device provided by the present application. The principles and implementation manners of the present application are described by applying specific examples in the present document, and the above description of the examples is only applicable to help understand the method of the present application and its core idea. It should be noted that, for those skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A request batch processing method, characterized in that, The method comprises the following steps: obtaining the performance requirement of a batch processing object under the current application scenario, the current actual performance, the request waiting queue length change information and the current batch size; in the case that the batch processing object meets the first batch processing optimization condition and does not meet the second batch processing optimization condition, determining the current comprehensive error of the batch processing object according to the performance requirement of the batch processing object under the current application scenario and the current actual performance; performing first adjustment on the current batch size according to the current comprehensive error of the batch processing object to obtain a first target batch size; in the case that the batch processing object does not meet the first batch processing optimization condition but meets the second batch processing optimization condition, performing second adjustment on the current batch size according to the request waiting queue length change information to obtain a second target batch size; wherein the adjustment step length of the second adjustment is greater than that of the first adjustment; performing batch processing on the requests to be processed in the request waiting queue according to the first target batch size or the second target batch size.

2. The request batch processing method of claim 1, wherein, The method comprises the following steps: determining whether there is a warning performance index according to the performance requirement of the batch processing object under the current application scenario and the current actual performance; in the case that there is the warning performance index, determining a target error calculation weight set according to the current performance optimization demand represented by the warning performance index; determining the current comprehensive error of the batch processing object according to the performance requirement of the batch processing object under the current application scenario, the current actual performance and the target error calculation weight set.

3. The request batch processing method of claim 2, wherein, The method comprises the following steps: determining the target values of a plurality of performance indexes according to the performance requirement of the batch processing object under the current application scenario; determining the current actual values of a plurality of the performance indexes according to the current actual performance of the batch processing object under the current application scenario; for any performance index, determining the error value of the performance index according to the deviation degree between the target value and the current actual value of the performance index; performing weighted calculation on the error values of a plurality of the performance indexes according to the target error calculation weight set to obtain the current comprehensive error of the batch processing object; wherein the target error calculation weight set comprises weight coefficients corresponding to each performance index.

4. The request batch processing method of claim 3, wherein, The method comprises the following steps: determining the current comprehensive error of the batch processing object based on the following formula: wherein, represents a current comprehensive error of the batch processing object, the target error calculation weight set includes , , , and , represents an error value of a batch delay indicator, represents a current actual value of the batch delay indicator, represents a target value of the batch delay indicator, represents an error value of a computing resource utilization rate, represents a current actual value of the computing resource utilization rate, represents a target value of the computing resource utilization rate, represents an error value of a request response delay indicator, represents a current actual value of the request response delay indicator, represents a target value of the request response delay indicator, represents an error value of a request processing efficiency, represents a current actual value of the request processing efficiency, represents a target value of the request processing efficiency, represents an error value of a memory occupancy rate, represents a current actual value of the memory occupancy rate, represents a target value of the memory occupancy rate, and the performance indicator includes the batch delay indicator, the computing resource utilization rate, the request response delay indicator, the request processing efficiency, and the memory occupancy rate.

5. The request batch processing method of claim 1, wherein, The method comprises the following steps: determining a target first adjustment coefficient set according to the error adjustment demand represented by the current comprehensive error of the batch processing object; determining a first adjustment value of the batch size according to a current comprehensive error of the batch processing object and a target first adjustment coefficient set; accumulating the first adjustment value of the batch size to the current batch size to obtain a first target batch size.

6. The request batch processing method of claim 5, wherein, The determining a first adjustment value of the batch size according to a current comprehensive error of the batch processing object and a target first adjustment coefficient set comprises: determining the first adjustment value of the batch size based on the following formula: wherein, denotes a first adjustment value of the batch size, the target first adjustment coefficient set comprises , and , denotes a speed coefficient, denotes a static coefficient, denotes a time coefficient, denotes the current comprehensive error, denotes cumulative information of the comprehensive error from an initial time to a current time, denotes a rate of change of the current comprehensive error.

7. The request batch processing method of claim 6, wherein, The method further comprises: freezing the static coefficient when the first target batch size reaches a preset upper limit value and the current comprehensive error of the batch processing object indicates that the current actual performance is higher than the performance requirement, or when the first target batch size reaches a preset lower limit value and the current comprehensive error of the batch processing object indicates that the current actual performance is lower than the performance requirement.

8. The request batch processing method of claim 1, wherein, The method further comprises: determining that the batch processing object satisfies the second batch processing optimization condition when the request waiting queue length change information indicates that the request waiting queue has request backlog.

9. The request batch processing method of claim 1, wherein, The second adjusting the current batch size according to the request waiting queue length change information to obtain a second target batch size comprises: determining a queue length growth factor according to the request waiting queue length change information; second adjusting the current batch size according to the queue length growth factor to obtain a second target batch size.

10. An electronic device, comprising: comprise: a memory for storing a computer program; a processor for executing the computer program to implement the steps of the request batch processing method according to any one of claims 1 to 9.

Images