Request batch processing method and electronic device

Abstract

The application discloses a request batch processing method and electronic equipment, and relates to the technical field of computers. When a batch processing object satisfies a first batch processing optimization condition, a current comprehensive error of the batch processing object is determined according to performance requirements of the batch processing object in a current application scenario and current actual performance, and a first adjustment is performed on a current batch size to obtain a first target batch size. When a second batch processing optimization condition is satisfied, a second adjustment is performed on the current batch size according to request waiting queue length change information to obtain a second target batch size. Finally, the to-be-processed requests are batch processed according to the first target batch size or the second target batch size. The current batch size is adjusted according to the performance requirements of the application scenario and the current actual performance, or the request waiting queue length change information, so that the target batch size can adapt to the current application scenario and the dynamically changing load, and the waste of hardware resources and the increase of user request response delay are avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and particularly relates to a request batch processing method and an electronic device. BACKGROUND

[0002] With the continuous development of pre-training language models, the model parameter scale grows exponentially, the memory occupation of single request inference is high, and the utilization rate of computing resources is low, so batch processing technology is often used to batch process multiple requests.

[0003] In related technologies, a static batch processing method is usually used, that is, a fixed batch size is used. However, the application scenarios and loads of pre-training language models are dynamically changing, different application scenarios have different performance requirements for batch processing, and in the load surge working condition, if the fixed batch size is too small, that is, the number of requests per batch is small, not only the hardware resources will be wasted, but also a large number of user requests will be in timeout waiting, and if the fixed batch size is too large, the response delay of user requests will increase. SUMMARY

[0004] The present application provides a request batch processing method and an electronic device to at least solve the problem that the fixed batch size in related technologies may cause waste of hardware resources or lead to an increase in response delay of user requests.

[0005] The present application provides a request batch processing method, comprising:

[0006] obtaining performance requirements of a batch processing object in a current application scenario, current actual performance, request waiting queue length change information, and a current batch size;

[0007] In a case where the batch processing object meets a first batch processing optimization condition and does not meet a second batch processing optimization condition, determining a current comprehensive error of the batch processing object according to the performance requirements of the batch processing object in the current application scenario and the current actual performance;

[0008] According to the current comprehensive error of the batch processing object, performing a first adjustment on the current batch size to obtain a first target batch size;

[0009] In a case where the batch processing object does not meet the first batch processing optimization condition but meets the second batch processing optimization condition, performing a second adjustment on the current batch size according to the request waiting queue length change information to obtain a second target batch size; wherein the adjustment step length of the second adjustment is greater than the adjustment step length of the first adjustment;

[0010] According to the first target batch size or the second target batch size, batch processing the requests to be processed in the request waiting queue.

[0011] The application further provides a request batch processing apparatus, comprising:

[0012] an acquisition module, configured to acquire performance requirements of the batch processing object in the current application scenario, current actual performance, request waiting queue length change information and the current batch size;

[0013] a determination module, configured to determine a current comprehensive error of the batch processing object according to the performance requirements of the batch processing object in the current application scenario and the current actual performance, in a case where the batch processing object meets the first batch processing optimization condition and does not meet the second batch processing optimization condition;

[0014] a first adjustment module, configured to perform first adjustment on the current batch size according to the current comprehensive error of the batch processing object, to obtain a first target batch size;

[0015] a second adjustment module, configured to perform second adjustment on the current batch size according to the request waiting queue length change information, to obtain a second target batch size, in a case where the batch processing object does not meet the first batch processing optimization condition but meets the second batch processing optimization condition; wherein the adjustment step length of the second adjustment on the batch size is greater than the adjustment step length of the first adjustment on the batch size;

[0016] a batch processing module, configured to perform batch processing on the to-be-processed requests in the request waiting queue according to the first target batch size or the second target batch size.

[0017] The application further provides an electronic device, comprising a memory configured to store a computer program, and a processor configured to execute the computer program to implement the steps of any of the request batch processing methods.

[0018] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of any of the request batch processing methods.

[0019] The application further provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the steps of any of the request batch processing methods.

[0020] According to the application, when the batch processing object meets the first batch processing optimization condition, the current comprehensive error of the batch processing object is determined according to the performance requirement of the current application scenario and the current actual performance, and the first adjustment is performed on the current batch size to obtain a first target batch size; when the second batch processing optimization condition is met, the second adjustment is performed on the current batch size according to the request waiting queue length change information to obtain a second target batch size; finally, the target batch size is determined according to the first target batch size or the second target batch size, and the batch processing is performed on the to-be-processed requests. According to the performance requirement of the application scenario and the current actual performance, or the request waiting queue length change information, the adjustment is performed on the current batch size, so that the target batch size can adapt to the current application scenario and the dynamically changing load, and the waste of hardware resources and the increase of user request response delay are avoided. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0022] Figure 1 The structure schematic diagram of the request batch processing system provided by the embodiments of the present application is shown in the figure.

[0023] Figure 2 The flow schematic diagram of the request batch processing method provided by the embodiments of the present application is shown in the figure.

[0024] Figure 3 The flow schematic diagram of the exemplary request batch processing method provided by the embodiments of the present application is shown in the figure.

[0025] Figure 4 The structure schematic diagram of the request batch processing device provided by the embodiments of the present application is shown in the figure.

[0026] Figure 5 The structure schematic diagram of the electronic device provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0028] It should be noted that in the description of the present application, the term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article or device. The terms "first", "second", and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0029] Pre-trained language models, also known as large models (Large Language Model, LLM for short), refer to deep learning models with parameter sizes reaching hundreds of millions or even tens of billions. Its core features are: large parameter size, for example, GPT-3 has 175 billion parameters, and it captures complex language rules through massive data training. Large data demand, training data usually covers multiple sources of information such as Internet text, books, and code, and the scale can reach TB. Therefore, the task complexity of large model training job load, long processing time and dynamic changes in resource demand make the scheduling problem of GPU cluster more prominent. However, with the exponential growth of model parameter size, the high memory occupancy of single request inference and low utilization of computing resources highlight the problem, resulting in high inference cost, which seriously restricts the large-scale landing.

[0030] Large model batching technology (Batching) dynamically merges concurrent requests, packages multiple independent inference tasks, and then sends them to the GPU for computation and execution at once. Batch size represents how many samples are processed in parallel in each training step. Assuming the batch size is 32, the neural network will process 32 independent samples at the same time in each training step. By batching multiple online requests, the efficiency of GPU usage can be improved.

[0031] In related technologies, a static batching method is usually used, that is, a fixed batch size is used. However, the application scenarios and loads of pre-trained language models are dynamically changing, different application scenarios have different performance requirements for batching, and in the load surge working condition, if the fixed batch size is too small, that is, the number of requests per batch is small, not only will it cause waste of hardware resources, but also will cause a large number of user requests to wait for timeout, and if the fixed batch size is too large, it will cause the response delay of user requests to increase.

[0032] In order to solve the above technical problems, the embodiments of the present application provide a request batch processing method and an electronic device. The method comprises the following steps: acquiring performance requirements of a batch processing object in a current application scenario, current actual performance, request waiting queue length change information and a current batch size; in the case that the batch processing object meets first batch processing optimization conditions and does not meet second batch processing optimization conditions, determining a current comprehensive error of the batch processing object according to the performance requirements of the batch processing object in the current application scenario and the current actual performance; performing first adjustment on the current batch size according to the current comprehensive error of the batch processing object, so as to obtain a first target batch size; in the case that the batch processing object does not meet the first batch processing optimization conditions but meets the second batch processing optimization conditions, performing second adjustment on the current batch size according to the request waiting queue length change information, so as to obtain a second target batch size; wherein the adjustment step length of the second adjustment on the batch size is greater than the adjustment step length of the first adjustment on the batch size; and performing batch processing on the to-be-processed requests in the request waiting queue according to the first target batch size or the second target batch size. The method provided by the above scheme can determine the current comprehensive error of the batch processing object according to the performance requirements of the batch processing object in the current application scenario and the current actual performance when the batch processing object meets the first batch processing optimization conditions, and can perform first adjustment on the current batch size to obtain a first target batch size. When the second batch processing optimization conditions are met, the second adjustment is performed on the current batch size according to the request waiting queue length change information to obtain a second target batch size. Finally, the to-be-processed requests are batch processed according to the first target batch size or the second target batch size. According to the performance requirements of the application scenario and the current actual performance, or the request waiting queue length change information, the current batch size is adjusted, so that the target batch size can adapt to the current application scenario and the dynamically changing load, and the waste of hardware resources and the increase of user request response delay are avoided. In order to enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0033] In combination with the specific application environment architecture or the specific hardware architecture on which the execution of the request batch processing method depends, the specific application environment architecture or the specific hardware architecture is described herein.

[0034] Firstly, the structure of the request batch processing system based on the present application is described.

[0035] The request batch processing method and the electronic device provided by the embodiments of the present application are suitable for dynamically adjusting the batch size according to the changes of the application scenario and the load, such as Figure 1As shown in FIG. 1, it is a structural schematic diagram of a request batch processing system based on which the embodiments of the present application are implemented, mainly including a request batch processing device and a batch processing object, the batch processing object is used for batch processing of requests of a pre-trained language model, and the request batch processing device adjusts the batch size of the batch processing object when performing request batch processing based on the request batch processing method provided by the embodiments of the present application.

[0036] The embodiments of the present application provide a request batch processing method for dynamically adjusting the batch size according to the changes of application scenarios and loads. The execution subject of the embodiments of the present application is an electronic device, such as a server, a desktop computer, a notebook computer, a tablet computer and other electronic devices that can be used for request batch processing.

[0037] As shown in FIG. 2, it is a flowchart of the request batch processing method provided by the embodiments of the present application, the method comprises the following steps. Figure 2

[0038] In step 201, the performance requirement of the batch processing object in the current application scenario, the current actual performance, the request waiting queue length change information and the current batch size are obtained.

[0039] Specifically, the performance requirement in the application scenario includes the requirements on inference delay, GPU utilization, first response time, cache hit rate and memory occupation, and the application scenario is, for example, a chat scenario, and the performance requirements in the chat scenario include that the inference delay is not more than 150 ms, the GPU utilization is not less than 85%, the memory occupation rate is not less than 90%, the first response time is not more than 500 ms and the cache hit rate is not less than 70%. In actual application, different performance requirement standards can be set according to different application scenarios. The current actual performance is the numerical value of the batch processing object in inference delay, GPU utilization, first response time, cache hit rate and memory occupation at the current time.

[0040] Correspondingly, by obtaining the performance requirement of the batch processing object in the current application scenario and the current actual performance, the gap between the current state and the ideal state of the batch processing object can be obtained, which provides a direction for subsequent adjustment of the target batch size. The request waiting queue length change information is used to monitor the number of requests to be processed in the request waiting queue in real time, so that the batch size can be adjusted in time according to the request waiting queue length change.

[0041] In step 202, when the batch processing object meets the first batch processing optimization condition and does not meet the second batch processing optimization condition, the current comprehensive error of the batch processing object is determined according to the performance requirement of the batch processing object in the current application scenario and the current actual performance.

[0042] ​Specifically, the first batch processing optimization condition is that the calculation of the current comprehensive error of the batch processing object is performed periodically, that is, when a corresponding time period is reached, it is determined that the batch processing object meets the first batch processing optimization condition, and the current comprehensive error represents the gap between the actual overall performance of the batch processing object and the performance requirement of the batch processing object in the current application scenario.

[0043] Correspondingly, by periodically calculating the current comprehensive error, the adaptation degree of the current batch size to the current application scenario of the batch processing object is quantified, and the current comprehensive error provides an adjustment direction and data reference for the subsequent target batch size.

[0044] Step 203, according to the current comprehensive error of the batch processing object, the first adjustment is performed on the current batch size to obtain the first target batch size.

[0045] Specifically, the current comprehensive error includes positive deviation and negative deviation, and the closer the current comprehensive error is to 0, the closer the actual performance indicators of the batch processing object are to the performance requirements of the batch processing object in the current application scenario, and the smaller the adjustment degree is.

[0046] Step 204, in the case that the batch processing object does not meet the first batch processing optimization condition but meets the second batch processing optimization condition, according to the request waiting queue length change information, the second adjustment is performed on the current batch size to obtain the second target batch size; wherein the batch size adjustment step of the second adjustment is greater than the batch size adjustment step of the first adjustment.

[0047] Specifically, the request waiting queue length is the number of pending requests contained in the request queue, when the number of pending requests changes, the request queue length changes, and the change size of the number of pending requests in the request queue is the request waiting queue length change information. The second batch processing optimization condition is that the number of pending requests in the request queue increases suddenly, in this case, the target batch size is adjusted to the second target batch size. The step refers to the different batch size growth rates, and the batch size growth rate of the second adjustment is greater than the batch size adjustment growth rate of the first adjustment, so as to quickly shorten the request waiting queue length and shorten the response delay of user requests.

[0048] Correspondingly, according to the request waiting queue length change information, the workload characteristics are automatically identified, which represents the change of the number of pending requests in the request waiting queue, so that when the load increases suddenly, the batch size can be adjusted in time, the pending requests can be processed in time, the response delay of user requests is reduced, and the utilization rate of hardware resources is improved.

[0049] Step 205, according to the first target batch size or the second target batch size, the pending requests in the request waiting queue are batch processed.

[0050] Specifically, the current batch size is adjusted according to the first target batch size or the second target batch size, and a corresponding number of pending requests in the request waiting queue are extracted for batch processing according to the first target batch size or the second target batch size.

[0051] Correspondingly, by selecting different numbers of pending requests for batch processing according to the first target batch size or the second target batch size, the actual performance of the current batch processing object is consistent with the performance requirement of the application scenario, thereby improving the utilization of resources and shortening the response delay of user requests.

[0052] On the basis of the above embodiments, as a kind of implementable mode, in an embodiment, according to the performance requirement of batch processing object in current application scenario and current actual performance, the current comprehensive error of batch processing object is determined, including:

[0053] Step 2021, according to the performance requirement of batch processing object in current application scenario and current actual performance, judge whether there is early warning performance index;

[0054] Step 2022, in the case where there is early warning performance index, according to the current performance optimization demand represented by early warning performance index, determine target error calculation weight set;

[0055] Step 2023, according to the performance requirement of batch processing object in current application scenario, current actual performance and target error calculation weight set, determine the current comprehensive error of batch processing object.

[0056] Specifically, the early warning includes delay overrun, GPU low utilization and preprocessing bottleneck, etc., and according to the performance requirement of batch processing object in current application scenario, a plurality of performance index types are determined. The target error calculation weight set is adjusted according to the current performance optimization demand represented by the early warning performance index. At the same time, the actual values of a plurality of performance indexes in current actual performance are obtained, when the actual value of performance index exceeds or is less than the preset threshold, it is represented that early warning occurs, and there is early warning performance index.

[0057] For example, when the inference delay exceeds the preset delay threshold, a pre-warning of delay overrun is represented, and therefore the index weight coefficients corresponding to the five aspects of inference delay, GPU utilization, first response time, cache hit rate and memory occupation are all adjusted; when the GPU utilization is lower than the preset utilization threshold, a GPU low utilization pre-warning is represented, and therefore the index weight coefficients corresponding to the three aspects of inference delay, GPU utilization and first response time are adjusted; when the proportion of the first response time in the inference delay represents whether a pre-processing bottleneck occurs, when the proportion is greater than the preset bottleneck threshold, a pre-processing bottleneck pre-warning occurs, and therefore the index weight coefficients corresponding to the three aspects of inference delay, GPU utilization and first response time are adjusted. The adjusted weight indicators are determined as the target error calculation weight set.

[0058] Correspondingly, by the performance requirements of the batch processing object in the current application scenario and the current actual performance, it is determined in time whether there is a pre-warning performance indicator, and then the target error calculation weight set can be adjusted according to the pre-warning performance indicator, so as to realize real-time adjustment of the influence of multiple performance indicators on the current comprehensive error according to the actual performance, and improve the authenticity of the current comprehensive error.

[0059] Specifically, in an embodiment, the target values of the multiple performance indicators can be determined according to the performance requirements of the batch processing object in the current application scenario; the current actual values of the multiple performance indicators can be determined according to the current actual performance of the batch processing object in the current application scenario; for any performance indicator, the error value of the performance indicator can be determined according to the deviation degree between the target value and the current actual value of the performance indicator; and the error values of the multiple performance indicators can be weighted and calculated according to the target error calculation weight set to obtain the current comprehensive error of the batch processing object. The target error calculation weight set includes the weight coefficients corresponding to the performance indicators.

[0060] Specifically, the error value is determined by the ratio of the deviation value between the target value and the current actual value of the performance indicator to the target value. The target values of the multiple performance indicators are derived and set under the premise of ensuring user experience. User experience refers to the user's feeling of smoothness and naturalness, and a strict derivation process is followed.

[0061] For example, the smooth natural passes the quantification of the delay below the human perception threshold, usually the value is determined to be less than 200ms. Again, the 200ms total delay budget is allocated to each link of the batch object, including network, preprocessing, inference and post-processing, etc., and the inference is the most time-consuming part, which may be allocated to 150ms. In order to complete the inference within 150ms, it is necessary to maximize the hardware efficiency. According to many experiments, when about 85% of GPU utilization and about 90% of video memory occupancy can achieve this performance target, while taking into account the cost. Again, introduce optimization techniques such as caching, set a challenging but achievable cache target according to the business characteristics, for example, 70%. Finally, according to the improvement of user experience and resource efficiency, fine-tune.

[0062] Correspondingly, the error value determined by the target value and the current actual value of the plurality of performance indicators quantifies the deviation between the ideal state and the actual state of each performance indicator, providing data support for determining the current comprehensive error of the batch object.

[0063] Specifically, in an embodiment, the current comprehensive error of the batch object can be determined based on the following formula:

[0064]

[0065] Among them, the current comprehensive error of the batch object, the target error calculation weight set includes , , , and , the error value of the batch delay indicator, the current actual value of the batch delay indicator, the target value of the batch delay indicator, the error value of the computing resource utilization rate, the current actual value of the computing resource utilization rate, the target value of the computing resource utilization rate, the error value of the request response delay indicator, the current actual value of the request response delay indicator, the target value of the request response delay indicator, the error value of the request processing efficiency, the current actual value of the request processing efficiency, the target value of the request processing efficiency, the error value of the memory occupancy rate, the current actual value of the memory occupancy rate, The target value representing the memory occupancy rate, the performance indicators including a batch processing delay indicator, a computing resource utilization rate, a request response delay indicator, a request processing efficiency, and a memory occupancy rate.

[0066] Specifically, the inference delay is quantified by a P99 end-to-end delay in a plurality of performance indicator types, corresponding to the batch processing delay indicator, wherein the P99 end-to-end delay is a total processing time from start to end of 99% of requests in all processed requests. The GPU utilization rate is quantified by a stream multiprocessor activity rate (SM unit activity rate), corresponding to the computing resource utilization rate. The first response time is quantified by a user waiting time for receiving a response, corresponding to the request response delay indicator. The cache hit rate is quantified by a number of processed requests per second, corresponding to the request processing efficiency. The memory occupancy is quantified by a GPU video memory usage rate, corresponding to the memory occupancy rate.

[0067] Specifically, the smaller the current actual value and the target value of the batch processing delay indicator, the faster the speed of processing a batch of pending requests, and the better the performance. The larger the current actual value and the target value of the computing resource utilization rate, the more reasonable the batch size setting. The smaller the current actual value and the target value of the request response delay indicator, the faster the speed of responding to a pending request, and the better the performance. The larger the current actual value and the target value of the request processing efficiency, the higher the cache hit rate, the faster the speed of reading data, and the faster the processing speed of the pending request. The larger the current actual value and the target value of the memory occupancy rate, the higher the resource utilization rate and the throughput, and when the memory occupancy rate approaches the limit, it also indicates that a memory overflow may be triggered under the current batch size.

[0068] Specifically, the basic weight set of the target error calculation weight set is usually When the delay is over the limit, the target error calculation weight set is usually When the GPU utilization rate is low, the target error calculation weight set is usually When the preprocessing bottleneck is warned, the target error calculation weight set is usually .

[0069] Correspondingly, different warning conditions reflect different problems in the actual state of the batch processing object under the current application scenario. By adjusting the target error calculation weight set, the weight of the problem indicator is improved, and then reflected in the current comprehensive error, so that the first adjustment according to the current comprehensive error solves the problem, and the current actual performance of the batch processing object is more in line with the performance requirements under the current application scenario.

[0070] On the basis of the above embodiments, as an implementable manner, in an embodiment, according to the current comprehensive error of the batch processing object, the current batch size is first adjusted to obtain a first target batch size, including:

[0071] In step 2031, according to the error adjustment demand represented by the current comprehensive error of the batch processing object, a target first adjustment coefficient set is determined.

[0072] In step 2032, according to the current comprehensive error of the batch processing object and the target first adjustment coefficient set, a first adjustment value of the batch size is determined.

[0073] In step 2033, the first adjustment value of the batch size is added to the current batch size to obtain the first target batch size.

[0074] Specifically, the target first adjustment coefficient includes a speed coefficient, a static coefficient and a time coefficient, and the target first adjustment coefficient is adjusted according to the current comprehensive error. The speed coefficient determines the rapidity of the batch processing object. The greater the speed coefficient, the faster the response speed of the batch processing object, and the higher the adjustment accuracy. However, if the speed coefficient is too large, the batch processing object will be unstable, and if the speed coefficient is too small, the adjustment accuracy will be reduced. The static coefficient eliminates the static error of the batch processing object. If the static coefficient is too large, the oscillation frequency of the batch processing object will increase, and if the static coefficient is too small, the adjustment accuracy of the batch processing object will be reduced. The time coefficient improves the dynamic characteristics of the batch processing object. If the time coefficient is too large, the overshoot will be large, the adjustment time of the batch processing object will be long, and if the time coefficient is too small, the overshoot will also be large, and the adjustment time will also be long.

[0075] Specifically, when the actual value is greater than the target value, the current comprehensive error value is a positive deviation, and when the actual value is less than the target value, the current comprehensive error value is a negative deviation. The value range of the current comprehensive error is When the current comprehensive error value is a positive deviation, the current actual value representing a plurality of performance indicators is greater than the performance demand, and the error adjustment demand is to reduce the current batch size. When the current comprehensive error value is a negative deviation, the current actual value representing a plurality of performance indicators is less than the performance demand, and the error adjustment demand is to increase the current batch size.

[0076] Correspondingly, the error adjustment demand represents the situation between the current actual value and the target value, and then the adjustment direction of the current batch size is determined according to the positive deviation or the negative deviation represented by the error adjustment demand.

[0077] Specifically, in an embodiment, the first adjustment value of the batch size can be determined based on the following formula:

[0078]

[0079] wherein, a first adjustment value representing a batch size, the target first adjustment coefficient set comprises 、 and , a speed coefficient, a static coefficient, a time coefficient, a current comprehensive error, cumulative information of the comprehensive error from an initial time to a current time, a change rate of the current comprehensive error.

[0080] Specifically, the target first adjustment coefficient set usually takes values of {1, 1, 1}. , , }.

[0081] Specifically, in an embodiment, the first target batch size can be determined based on the following formula:

[0082]

[0083] wherein, the first target batch size after the first adjustment of the current batch size, the current batch size, a first adjustment value representing a batch size. After determining the first adjustment value, the first target batch size is determined according to the first adjustment value. Finally, the first target batch size is constrained to ensure that it is between the minimum allowed batch size and the maximum allowed batch size.

[0084] Correspondingly, the speed coefficient enables the current batch size to be quickly adjusted when it needs to be adjusted, the static coefficient ensures the adjustment accuracy of the first target batch size, and the time coefficient enables the current batch size to be adjusted steadily.

[0085] Specifically, in an embodiment, when the first target batch size reaches a preset upper limit value and the current comprehensive error of the batch processing object indicates that the current actual performance is higher than the performance requirement, or when the first target batch size reaches a preset lower limit value and the current comprehensive error of the batch processing object indicates that the current actual performance is lower than the performance requirement, the static coefficient is frozen.

[0086] Specifically, the preset upper limit value is the maximum value allowed by the batch size, which is determined according to the GPU memory capacity, and the preset lower limit value is the minimum value allowed by the batch size, which is usually 1. In the case where the current comprehensive error of the batch processing object indicates that the current actual performance is higher than the performance requirement, the current first target batch size does not meet the performance requirement, and the first target batch size needs to be further increased. However, since the first target batch size has reached the preset upper limit value, that is, the limit of the memory capacity of the hardware device, the static coefficient is frozen to avoid further expanding the first target batch size. In the case where the current comprehensive error of the batch processing object indicates that the current actual performance is lower than the performance requirement, the current first target batch size is greater than the performance requirement, and the first target batch size needs to be further reduced. However, since the first target batch size has reached the preset lower limit value, that is, has been reduced to the minimum value, the static coefficient is frozen to avoid further reducing the first target batch size.

[0087] Correspondingly, when the batch processing object reaches the physical limit, the static coefficient is frozen to eliminate the accumulation of past errors of the batch processing object, eliminate the steady-state error, shorten the adjustment time, and improve the control accuracy. Avoiding the unlimited increase of the first target batch size, which may lead to the collapse of the batch processing object, and avoiding the unlimited reduction of the first target batch size, which may lead to the stop of the batch processing object to process the pending requests, ensures the rationality of the adjustment of the first target batch size.

[0088] Specifically, in an embodiment, when the request waiting queue length change information indicates that the request waiting queue has request backlog, it is determined that the batch processing object meets the second batch processing optimization condition.

[0089] Specifically, when the number of pending requests in the request waiting queue suddenly increases, but the current batch size cannot meet the growth rate of the increased number of pending requests, the request waiting queue has request backlog, and it is determined that the batch processing object meets the second batch processing optimization condition, the second adjustment is performed to determine the second target batch size.

[0090] Correspondingly, the request waiting queue length change information represents the number of pending requests in the request waiting queue, so that the batch processing object can discover the occurrence of request backlog in advance according to the number change, and then perform the second adjustment in time to determine the second target batch size. In the scenario of traffic surge, the cluster utilization can be effectively improved.

[0091] On the basis of the above-mentioned embodiments, as a kind of implementable mode, in an embodiment, according to request waiting queue length change information, the current batch size is secondly adjusted to obtain second target batch size, comprising:

[0092] In step 2051, a queue length growth factor is determined according to the request waiting queue length change information.

[0093] At step 2052, a second adjustment is made to the current batch size according to the queue length growth factor to obtain a second target batch size.

[0094] Specifically, the queue length growth factor is determined based on the following formula:

[0095]

[0096] wherein, represents the queue length growth factor, represents the number of pending requests in the request waiting queue after the change, represents the number of pending requests in the request waiting queue before the change.

[0097] For example, when the number of pending requests in the request waiting queue suddenly increases from 10 to 100, the request backlog occurs in the request waiting queue, and the queue length growth factor is:

[0098]

[0099] The second target batch size is determined based on the following formula:

[0100]

[0101] wherein, represents the second target batch size after the second adjustment to the current batch size according to the queue length growth factor, represents the current batch size, represents the queue length growth factor.

[0102] Correspondingly, the current batch size is steadily adjusted by the queue length growth factor, and the boundary constraint is performed by multiplying the current batch size and the queue length growth factor, thereby improving the stability of the batch processing object.

[0103] For example, Figure 3As shown, the flowchart of the example request batch processing method provided by the embodiment of the application is shown. In the case that the batch processing object meets the first batch processing optimization condition and does not meet the second batch processing optimization condition, the target value of the plurality of performance indicators and the current actual value of the performance indicators are determined according to the performance requirements and the current actual performance of the batch processing object in the current application scenario, and then the error values of the plurality of performance indicators are determined. The target error is calculated to obtain the weight set, and the error values of the plurality of performance indicators are weighted and calculated to obtain the current comprehensive error of the batch processing object, that is, the multi-objective weighted fusion. Then the target first adjustment coefficient set is determined through the current comprehensive error, and then the first adjustment value of the batch size is determined, that is, the PID controller. The first target batch size is obtained according to the first adjustment value, and the requests in the request waiting queue are batch processed according to the first target batch size. In the case that the batch processing object does not meet the first batch processing optimization condition but meets the second batch processing optimization condition, the queue length growth factor is determined by using the feedforward controller according to the request waiting queue length change information, and the second target batch size is obtained by second adjusting the current batch size according to the queue length growth factor. The requests in the request waiting queue are batch processed according to the second target batch size, and then inference testing is performed according to the determined target batch size.

[0104] For example, DeepSeek-R1-Distill-Qwen-32B is used as an example, and is deployed on an 8-card A100 node using the vllm framework. The performance requirements in the current application scenario are inference delay of 150ms, GPU utilization rate of not less than 85%, memory occupancy rate of not less than 90%, first response time of generally 500ms, and cache hit rate of not less than 70%. The current actual performance is fixed batch size of 16, delay of 230ms, and utilization rate of 78%. The multi-objective PID optimization process is as follows:

[0105] Round 1: High delay is detected, and the weight is focused on delay, so the batch size is reduced to 12;

[0106] Round 2: The delay is improved (195ms), but the utilization rate is reduced to 75%, so the utilization rate weight is increased;

[0107] Round 3: The batch size is increased to 14, the delay is 205ms, and the utilization rate is 82%;

[0108] Round 4: The request peak is predicted, and the feedforward control is reduced to 11 in advance;

[0109] Steady state: The batch size is dynamically adjusted to 13-15, the delay is 190±10ms, and the utilization rate is 86%.

[0110] The application can also be applied to various model architectures such as Transformer, CNN or RNN.

[0111] Specifically, in an embodiment, when the batch processing object satisfies the first batch processing optimization condition and satisfies the second batch processing optimization condition, the first target batch size and the second target batch size are determined respectively, the third target batch size is an adjustment value of the final current batch size, the first target batch size and the second target batch size are weighted to calculate the third target batch size, and the third target batch size is used to batch process the pending requests in the request waiting queue.

[0112] The third target batch size is determined based on the following formula:

[0113]

[0114] wherein, the third target batch size is represented by T3, the first target batch size after the first adjustment of the current batch size is represented by T1, the second target batch size after the second adjustment of the current batch size is represented by T2, and the weight coefficients of the first target batch size and the second target batch size are respectively represented by a and b, and in general cases, , .

[0115] The method for request batch processing provided in the embodiments of the present application comprises the following steps: obtaining the performance requirement of a batch processing object in a current application scenario, the current actual performance, the request waiting queue length change information and the current batch size; in the case that the batch processing object meets the first batch processing optimization condition and does not meet the second batch processing optimization condition, determining the current comprehensive error of the batch processing object according to the performance requirement of the batch processing object in the current application scenario and the current actual performance; performing the first adjustment on the current batch size according to the current comprehensive error of the batch processing object, so as to obtain the first target batch size; in the case that the batch processing object does not meet the first batch processing optimization condition but meets the second batch processing optimization condition, performing the second adjustment on the current batch size according to the request waiting queue length change information, so as to obtain the second target batch size; wherein the adjustment step length of the second adjustment on the batch size is greater than the adjustment step length of the first adjustment on the batch size; and performing the batch processing on the to-be-processed requests in the request waiting queue according to the first target batch size or the second target batch size. The method provided in the above scheme can determine the current comprehensive error of the batch processing object according to the performance requirement of the batch processing object in the current application scenario and the current actual performance when the batch processing object meets the first batch processing optimization condition, and can perform the first adjustment on the current batch size to obtain the first target batch size, and can perform the second adjustment on the current batch size according to the request waiting queue length change information when the batch processing object meets the second batch processing optimization condition to obtain the second target batch size, and finally can perform the batch processing on the to-be-processed requests according to the first target batch size or the second target batch size. The current batch size is adjusted according to the performance requirement and the current actual performance of the application scenario or the request waiting queue length change information, so that the target batch size can adapt to the current application scenario and the dynamically changing load, and the waste of hardware resources and the increase of the user request response delay are avoided.

[0116] Moreover, by obtaining the performance requirement of the batch processing object under the current application scenario and the current actual performance, the gap between the current state and the ideal state of the batch processing object can be obtained, providing a direction for subsequent adjustment of the target batch size. By monitoring the number of pending requests in the request waiting queue through the request waiting queue length change information, the batch size can be adjusted in a timely manner according to the request waiting queue length change. By periodically calculating the current comprehensive error, the adaptation degree of the current batch size to the current application scenario of the batch processing object is quantified, and the current comprehensive error provides a direction and data reference for subsequent adjustment of the target batch size. Through the current comprehensive error, the trade-off of multiple indicators is realized, the batch size is automatically reduced to ensure the delay under high load, and the batch size is automatically increased to improve the throughput under low load, achieving dynamic balance between throughput and delay. According to the request waiting queue length change information, the workload characteristics are automatically identified, representing the change in the number of pending requests in the request waiting queue, so that the batch size can be adjusted in a timely manner when the load suddenly increases or decreases, and the pending requests can be processed in a timely manner, reducing the response delay of user requests and improving the utilization of hardware resources. By selecting different numbers of pending requests for batch processing according to the first target batch size or the second target batch size, the actual performance of the current batch processing object and the performance requirement of the application scenario can be met, thereby improving the utilization of resources and shortening the response delay of user requests. By the performance requirement of the batch processing object under the current application scenario and the current actual performance, it is determined in a timely manner whether there is a warning performance indicator, and then the target error weight set can be adjusted according to the warning performance indicator, realizing real-time adjustment of the influence of multiple performance indicators on the current comprehensive error according to the actual performance, and improving the authenticity of the current comprehensive error.

[0117] Meanwhile, the error value determined by the target value and the current actual value of the plurality of performance indicators quantifies the deviation degree between the ideal state and the actual state of each performance indicator, and provides data support for determining the current comprehensive error of the batch processing object. Different early warning situations reflect different problems of the actual state of the batch processing object in the current application scenario. By adjusting the target error calculation weight set, the weight of the problem indicator is improved, and then it is reflected in the current comprehensive error, so that the first adjustment according to the current comprehensive error solves the problem, and the current actual performance of the batch processing object is more in line with the performance requirements in the current application scenario. The error adjustment requirement represents the situation between the current actual value and the target value, and then the adjustment direction of the current batch size is judged according to the positive deviation or negative deviation represented by the error adjustment requirement. Through the speed coefficient, the current batch size can be quickly responded when it needs to be adjusted, and the static coefficient guarantees the adjustment accuracy of the first target batch size, and the time coefficient makes the current batch size can be adjusted steadily. When the batch processing object reaches the physical limit, the static coefficient is frozen, the accumulated summation of the past error of the batch processing object is eliminated, the steady-state error is eliminated, the adjustment time is shortened, and the control precision is improved. Avoiding the first target batch size from increasing indefinitely, which may cause the batch processing object to collapse, and avoiding the first target batch size from decreasing indefinitely, which may cause the batch processing object to stop processing the pending requests, the rationality of the adjustment of the first target batch size is guaranteed. The request waiting queue length change information represents the number of pending requests in the request waiting queue, so that the batch processing object can discover that the request backlog has occurred in advance according to the number change, and then perform the second adjustment in time to determine the second target batch size. In the scene of traffic surge, the cluster utilization rate can be effectively improved. The queue length growth factor is used to realize the steady adjustment of the current batch size, and the current batch size is multiplied by the queue length growth factor to perform boundary constraint, thereby improving the stability of the batch processing object.

[0118] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment.

[0119] Embodiments of the present application also provide a request batch processing device for executing the request batch processing method provided by the above embodiments.

[0120] As shown in Figure 4 , it is a structure schematic diagram of the request batch processing device provided by the embodiments of the present application. The request batch processing device 40 comprises: an acquisition module 401, a determination module 402, a first adjustment module 403, a second adjustment module 404 and a batch processing module 405.

[0121] The obtaining module is configured to obtain performance requirements of the batch processing object in the current application scenario, current actual performance, request waiting queue length change information, and a current batch size.

[0122] The determining module is configured to determine a current comprehensive error of the batch processing object according to the performance requirements of the batch processing object in the current application scenario and the current actual performance, in a case where the batch processing object meets the first batch processing optimization condition and does not meet the second batch processing optimization condition.

[0123] The first adjusting module is configured to perform first adjustment on the current batch size according to the current comprehensive error of the batch processing object, to obtain a first target batch size.

[0124] The second adjusting module is configured to perform second adjustment on the current batch size according to the request waiting queue length change information, to obtain a second target batch size, in a case where the batch processing object does not meet the first batch processing optimization condition but meets the second batch processing optimization condition, and wherein an adjustment step length of the second adjustment on the batch size is greater than an adjustment step length of the first adjustment on the batch size.

[0125] The batch processing module is configured to perform batch processing on the to-be-processed requests in the request waiting queue according to the first target batch size or the second target batch size.

[0126] The description of the features in the embodiments of the request batch processing apparatus can be referred to the related description of the embodiments of the request batch processing method, which will not be repeated here.

[0127] Embodiments of the present application also provide an electronic device, as shown in the accompanying drawings, which includes a processor 10 and a memory 20. The memory 20 stores a computer program, and the processor 10 is configured to run the computer program to perform the steps in any of the above-described embodiments of the request batch processing method. Figure 5

[0128] Embodiments of the present application also provide a computer readable storage medium, which stores a computer program. When the computer program is run, it performs the steps in any of the above-described embodiments of the request batch processing method.

[0129] In an example embodiment, the above-described computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0130] ​The embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program realizes the steps in any of the request batch processing method embodiments when executed by a processor.

[0131] The embodiment of the present application further provides another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program realizes the steps in any of the request batch processing method embodiments when executed by a processor.

[0132] Those skilled in the art can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in general terms. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0133] The above describes in detail a request batch processing method and an electronic device provided by the present application. The principles and implementation modes of the present application are described by applying specific examples in this paper, and the above description of the examples is only applicable to help understand the method of the present application and its core idea. It should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A request batch processing method, characterized in that, The method comprises the following steps: obtaining the performance requirement of a batch processing object under a current application scenario, the current actual performance, the request waiting queue length change information and the current batch size; in the case that the batch processing object meets the first batch processing optimization condition and does not meet the second batch processing optimization condition, determining the current comprehensive error of the batch processing object according to the performance requirement of the batch processing object under the current application scenario and the current actual performance; performing first adjustment on the current batch size according to the current comprehensive error of the batch processing object to obtain a first target batch size; in the case that the batch processing object does not meet the first batch processing optimization condition but meets the second batch processing optimization condition, performing second adjustment on the current batch size according to the request waiting queue length change information to obtain a second target batch size; wherein the adjustment step length of the second adjustment is greater than that of the first adjustment; performing batch processing on the requests in the request waiting queue according to the first target batch size or the second target batch size; the first adjustment on the current batch size according to the current comprehensive error of the batch processing object to obtain a first target batch size comprises the following steps: determining a target first adjustment coefficient set according to the error adjustment demand represented by the current comprehensive error of the batch processing object; determining a first adjustment value of the batch size according to the current comprehensive error of the batch processing object and the target first adjustment coefficient set; adding the first adjustment value of the batch size to the current batch size to obtain the first target batch size; the determination of the first adjustment value of the batch size according to the current comprehensive error of the batch processing object and the target first adjustment coefficient set comprises the following steps: determining the first adjustment value of the batch size according to the following formula: wherein, denotes a first adjustment value of the batch size, the target first adjustment coefficient set comprises , and , denotes a speed coefficient, denotes a static coefficient, denotes a time coefficient, denotes the current comprehensive error, denotes cumulative information of the comprehensive error from an initial time to a current time, denotes a rate of change of the current comprehensive error.

2. The request batch processing method of claim 1, wherein, the determination of the current comprehensive error of the batch processing object according to the performance requirement of the batch processing object under the current application scenario and the current actual performance comprises the following steps: judging whether there is a pre-warning performance index according to the performance requirement of the batch processing object under the current application scenario and the current actual performance; in the case that the pre-warning performance index exists, determining a target error calculation weight set according to the current performance optimization demand represented by the pre-warning performance index; determining the current comprehensive error of the batch processing object according to the performance requirement of the batch processing object under the current application scenario, the current actual performance and the target error calculation weight set.

3. The request batch processing method of claim 2, wherein, the determination of the current comprehensive error of the batch processing object according to the performance requirement of the batch processing object under the current application scenario, the current actual performance and the target error calculation weight set comprises the following steps: determining the target values of multiple performance indexes according to the performance requirement of the batch processing object under the current application scenario; determining the current actual values of the multiple performance indexes according to the current actual performance of the batch processing object under the current application scenario; for any performance index, determining the error value of the performance index according to the deviation degree between the target value and the current actual value of the performance index; According to the target error calculation weight set, error values of multiple performance indicators are weighted and calculated to obtain a current comprehensive error of the batch processing object; The target error calculation weight set includes a weight coefficient corresponding to each performance indicator.

4. The request batch processing method of claim 3, wherein, The target error calculation weight set, error values of multiple performance indicators are weighted and calculated to obtain a current comprehensive error of the batch processing object, includes: Based on the following formula, the current comprehensive error of the batch processing object is determined: wherein, represents a current comprehensive error of the batch processing object, the target error calculation weight set includes , , , and , represents an error value of a batch delay indicator, represents a current actual value of the batch delay indicator, represents a target value of the batch delay indicator, represents an error value of a computing resource utilization rate, represents a current actual value of the computing resource utilization rate, represents a target value of the computing resource utilization rate, represents an error value of a request response delay indicator, represents a current actual value of the request response delay indicator, represents a target value of the request response delay indicator, represents an error value of a request processing efficiency, represents a current actual value of the request processing efficiency, represents a target value of the request processing efficiency, represents an error value of a memory occupancy rate, represents a current actual value of the memory occupancy rate, represents a target value of the memory occupancy rate, and the performance indicator includes the batch delay indicator, the computing resource utilization rate, the request response delay indicator, the request processing efficiency, and the memory occupancy rate.

5. The request batch processing method of claim 1, wherein, The method further includes: In a case where the first target batch size reaches a preset upper limit value and the current comprehensive error of the batch processing object represents that the current actual performance is higher than the performance requirement, or in a case where the first target batch size reaches a preset lower limit value and the current comprehensive error of the batch processing object represents that the current actual performance is lower than the performance requirement, the static coefficient is frozen.

6. The request batch processing method of claim 1, wherein, The method further includes: When the request waiting queue length change information represents that the request waiting queue has request backlog, it is determined that the batch processing object satisfies the second batch processing optimization condition.

7. The request batch processing method of claim 1, wherein, According to the request waiting queue length change information, the current batch size is second adjusted to obtain a second target batch size, including: According to the request waiting queue length change information, a queue length growth factor is determined; According to the queue length growth factor, the current batch size is second adjusted to obtain a second target batch size.

8. An electronic device, comprising: It includes: A memory for storing a computer program; A processor for executing the computer program to implement the steps of the request batch processing method according to any one of claims 1 to 7.

Images