A batch processing optimization method and system for deep learning model reasoning

Abstract

The present invention discloses a method and system for optimizing batch processing of deep learning model reasoning. The system includes a load container batch processing analysis tool module, a batch processing merging tool module, and an algorithm service calling module; the load container batch processing analysis tool module is used to save the reasoning input parameters in the reasoning service request to the preprocessing data set, generate training data to perform performance tests on the models in each container, and then determine the optimal parameters according to the test result indicators; the batch processing merging tool module predicts and obtains the reasoning service request of the next time window according to the optimal parameters to generate a batch processing task and send it to the corresponding container for execution. The present invention optimizes the performance of deep learning reasoning services under a serverless architecture, effectively utilizes multi-core parallel computing capabilities, and can efficiently perform reasoning service requests under high concurrency, so that resource utilization, task execution delay, and throughput are greatly optimized.

Description

Technical Field

[0001] The present invention relates to a deep learning model reasoning batch processing optimization method and system, and belongs to the field of computer artificial intelligence and cloud computing. Background Art

[0002] With the rapid development of emerging technologies such as 5G, Internet of Things, big data, and cloud computing, artificial intelligence technology has emerged and is becoming a decisive force in driving mankind into the intelligent era. The global industry has fully realized the great significance of artificial intelligence technology in leading a new round of industrial transformation, and has transformed and developed to lay out an artificial intelligence innovation ecosystem. With the vigorous development of the artificial intelligence industry, the further maturity of technology, and the increasing investment from the government and industry, the cloudification of artificial intelligence applications will continue to accelerate, and its impact on society will become increasingly prominent. As the core force of a new round of scientific and technological revolution and industrial transformation, it is promoting the upgrading of traditional industries, driving the rapid development of the "unmanned economy", and having a positive impact on people's livelihood areas such as smart transportation, smart homes, and smart medical care.

[0003] Machine Learning is the core method for realizing artificial intelligence. It is a process of analyzing and processing data through computers, using certain algorithms such as classification, regression, and clustering to seek internal laws, and then making predictions or decisions about events. Deep Learning is a technical means to realize machine learning, which shows excellent performance in feature extraction. By building a neural network with many hidden layers and then training with massive data, useful features are learned, thereby ultimately improving the accuracy of classification or prediction. With the continuous maturity of deep learning technology, a series of relatively classic and common algorithms and models have been formed in the fields of computer vision, image processing, natural language processing, etc., such as AlexNet and VGGNet in the field of image processing, and BERT and Transformer in the field of natural language processing. As a result, more and more algorithm developers deploy the trained deep learning algorithm models and publish them as inference services, which has become the basis of many scientific research works.

[0004] At the same time, serverless architecture has become an emerging cloud architecture paradigm with the development of cloud computing, providing users with on-demand service calls and hiding the task management details of the instance from users. Users only need to provide functions and their triggering events to publish services. Currently, a large number of applications have been deployed on serverless platforms, such as intelligent transportation systems, IoT frameworks, subscription services, video / image processing, and deep learning related services.

[0005] Usually, function algorithms in serverless architecture tend to be short, reusable code blocks with the characteristics of short life cycle, statelessness and no long connection service. In deep learning, the trained algorithm can be exported as a model file. The function code only needs to provide the implementation of loading the model and reasoning functions. The amount of code is relatively small, and the reasoning process is stateless. In some low-latency reasoning tasks, the execution time is often in seconds. There is no need to provide long connection services. In terms of usage scenarios, deep learning algorithms meet the requirements of serverless functions. Therefore, deploying deep learning algorithms in serverless architecture provides a great opportunity. The trained deep learning algorithm model can be deployed in the serverless architecture through service release, and reasoning prediction services can be provided to reduce the repeated development and deployment of the algorithm. The main challenges are as follows:

[0006] The deployment of deep learning model inference services in a serverless architecture faces three challenges: (1) How to respond well to sudden inference service requests; (2) How to provide inference services quickly and with low latency to ensure the overall throughput of the system; and (3) How to automatically identify the optimal allocation of deep learning model resources to maximize system utilization under reasonable resource allocation and reduce the overall inference cost of the system. Summary of the invention

[0007] Aiming at the problem of deep learning inference, the present invention proposes a method and system for optimizing batch processing of deep learning model inference. Since the deep learning algorithm model can be accelerated by batch processing according to hardware computing resources during inference calculation, a method for analyzing batch processing parameters of load containers is proposed to obtain the optimal batch processing parameters of the service, and an adaptive sliding window batch processing merging algorithm is proposed to improve the performance of real-time online inference. The system architecture of the present invention is as follows: Figure 1 shown.

[0008] The technical solution adopted by the present invention is as follows:

[0009] A batch processing optimization method for deep learning model reasoning, the steps of which include:

[0010] 1) Obtain the online reasoning service request through the request interceptor, and save the reasoning input parameters in the reasoning service request into the preprocessing data set;

[0011] 2) According to the type of resources occupied by the inference service request, the inference service request is divided into CPU type tasks and GPU type tasks; multiple containers are set for CPU type tasks, and different containers have different memory sizes and CPU core numbers; for GPU type tasks, several GPU type task containers with set memory sizes and GPU core numbers are generated;

[0012] 3) For the inference service of the same model, if the inference service is a CPU type task, training data sets of different batch sizes are generated for the model according to the inference input parameters in the preprocessing data set and input into each container to perform a performance test on the model, and then a load performance table is generated according to the test result indicators; then the batch merge execution time Batchtime and the non-merge execution time NoBatchtime in the load performance table are compared, and the records in the load performance table are screened by comparing the ratio with the set threshold δ; then, based on the screened records, several records with the highest batch size occurrence frequency are selected, and the records with the smallest memory are selected from them; if the inference service is a GPU type task, training data sets of different batch sizes are generated according to the inference input parameters in the preprocessing data set and input into the GPU type task container to perform a performance test on the model, and the record with the largest batch size that meets the threshold indicator condition is selected;

[0013] 4) Determine the optimal parameter based on the selected records in step 3), that is, the batch size Y in the selected records batch and batch execution time T batch ;

[0014] 5) Cache the inference service requests received in real time into the task cache queue, and then the workload aggregator calculates the batch execution time T batch Determine a time interval, count the number of inference service requests in each time interval in the task cache queue, and obtain a time series;

[0015] 6) The inference service requests in the most recent period are intercepted from the time series for prediction, and the number of service requests in the next time interval is obtained and input into the adaptive window algorithm model. The adaptive window algorithm model adjusts the left and right boundaries of the current window according to the input information, and then takes out the inference service requests in the window from the task cache queue to generate batch tasks and sends them to the corresponding container for execution according to the type of resources occupied by the inference service requests.

[0016] Furthermore, the method for predicting the number of service requests, the optimal batch size and the optimal batch execution time in the next time interval is as follows: let the time series be {X1, X2, X3, … Xt}, Xt is the reasoning service request in the t-th time interval, let α be the smoothing coefficient; according to the recursive relationship Calculates the quadratic exponential smoothing value, where S i (2) is the quadratic exponential smoothing value of Xi, S i (1) is the exponential smoothing value of Xi; then the number of service requests Y in the next time interval is predicted t+1 =a t +bt ,

[0017] Furthermore, the method for the adaptive window algorithm model to adjust the left and right boundaries of the current window according to the input information is as follows: first determine the window discrimination function Determine whether the window is expanded or reduced; then according to the formula T length+1 =T length +(P(t)×σ(Z)×T batch ) determines the adjusted window; wherein, ΔF=|Y t+1 -y t |;T length+1 is the adjusted window length, y t Indicates the real-time request size in t time intervals.

[0018] Furthermore, the strategy for adjusting the left and right borders of the current window is: left+1 =W left +T finished , W right+1 =W left+1 +T length+1 ; Among them, W left+1 , W right+1 are the adjusted left and right boundaries, T finished The length of time that the batch request has completed within the window.

[0019] Furthermore, if the corresponding ratio is recorded The record is retained.

[0020] A batch processing optimization system for deep learning model reasoning, characterized by comprising a load container batch processing analysis tool module and a batch processing merging tool module; wherein

[0021] The load container batch analysis tool module is used to obtain the online reasoning service request through the request interceptor, and save the reasoning input parameters in the reasoning service request into the preprocessing data set; then, according to the type of resources occupied by the reasoning service request, the reasoning service request is divided into CPU type tasks and GPU type tasks; multiple containers are set for CPU type tasks, and different containers have different memory sizes and CPU core numbers; for GPU type tasks, several GPU type task containers with set memory sizes and GPU core numbers are generated; then, for the reasoning service of the same model, if the reasoning service is a CPU type task, training data sets of different batch sizes are generated for the model according to the reasoning input parameters in the preprocessing data set and input into each container to perform performance testing on the model, and then Then, a load performance table is generated according to the test result indicators; then, the batch merge execution time Batchtime and the non-merge execution time NoBatchtime in the load performance table are compared, and the records in the load performance table are screened by comparing the ratio with the set threshold δ; then, based on the screened records, several records with the highest batch size occurrence frequency are selected, and the records with the smallest memory are selected from them; if the inference service is a GPU type task, training data sets with different batch sizes are generated according to the inference input parameters in the preprocessing data set and input them into the GPU type task container to perform performance testing on the model, and the records with the largest batch size that meet the threshold indicator conditions are selected; then, the optimal parameter is determined according to the selected records, that is, the batch size Y in the selected records. batch and batch execution time T batch ;

[0022] The batch merging tool module is used to cache the inference service requests received in real time into the task cache queue, and then the workload aggregator calculates the batch execution time T. batch Determine a time interval, count the number of inference service requests in each time interval in the task cache queue, and obtain a time series; then intercept the inference service requests in the most recent period from the time series for prediction, obtain the number of service requests in the next time interval and input it into the adaptive window algorithm model, the adaptive window algorithm model adjusts the left and right boundaries of the current window according to the input information, then takes out the inference service requests in the window from the task cache queue to generate batch tasks and sends them to the corresponding container for execution according to the type of resources occupied by the inference service requests.

[0023] The load container batch processing parameter analysis method of the present invention adopts the "service first, then strategy" approach, which allows the algorithm model to be put online for a period of time first, obtains service request data through request interception, automatically generates service requests for different container environments and different batch sizes, and then performs load performance testing to generate a load performance table for the algorithm model, and selects appropriate container resource configuration, batch size and batch execution time through parameter analysis methods. Figure 2 As shown, the following steps are included:

[0024] Steps to intercept deep learning inference service requests: forward the online service HTTP request (i.e. inference service request) through the request interceptor to obtain the various parameters of this request. Save the HTTP parameters of the inference service request in the request parameter data set, such as the request sending time and response time, and save the inference input parameters in the preprocessing data set to generate the training data set.

[0025] Steps to generate containers of different resource sizes through the container generator: According to the type of inference service request task, divide it into CPU type tasks and GPU type tasks according to the type of occupied resources. For CPU type tasks, divide them into multiple groups of containers according to different memory sizes and CPU core numbers, where the memory size and CPU core number are based on multiple pre-defined specifications; for GPU type tasks, directly generate GPU type task containers with specific memory and GPU core number.

[0026] Steps to generate training data sets of different batch sizes by inferring input parameters: By formulating batches of different sizes, multiple data sets are generated for the same model. This data set is placed in the corresponding container as an input parameter for execution.

[0027] Furthermore, the method for performing load performance testing is as follows: for CPU type tasks, for the inference service of the same model, batch data sets of different sizes are respectively used for performance testing in different containers, and then the test result indicators are saved, such as memory size, batch size, batch execution time, and non-merged execution time, and a load performance table is generated based on these indicators. For GPU type tasks, training data sets of different batch sizes are generated based on the inference input parameters in the preprocessing data set and input into the GPU type task container to perform performance testing on the model, and the record with the largest batch size that meets the threshold indicator condition is selected; then the optimal parameter is determined based on the selected record, that is, the batch size Y in the selected record. batch and batch execution time T batch ;.

[0028] Further, the batch processing parameter analysis method comprises the following steps:

[0029] Compare the batch merge execution time Batchtime and the non-merge execution time NoBatchtime in the load performance table, and compare the ratio with the set threshold δ. Compare and select the appropriate records. The selection rules are as follows:

[0030] For CPU-type tasks, the records that meet the threshold are further screened, because there will be multiple sets of container records that meet the conditions, and it is necessary to continue to select from the remaining container records. At this time, the selection rule is to count the candidate record set of the batch size with the highest frequency in the remaining container records, and then select the record with the smallest memory as the final result. For GPU-type tasks, it is only necessary to select the record with the largest batch size under the condition that the threshold indicator is met. The parameters of this batch are important parameters in the subsequent adaptive sliding window algorithm.

[0031] A batch merging algorithm based on an adaptive sliding window is used to merge and optimize deep learning inference service requests, which mainly includes a timing prediction algorithm for inference service requests and an adaptive sliding window adjustment algorithm. Figure 3 shown.

[0032] Inference service time series prediction algorithm, users provide conditions for subsequent inference service merging, which includes three parts: task cache queue, request time series, time series prediction, and prediction result evaluation.

[0033] Task cache queue: The task cache queue caches inference service requests in the order they arrive. The queue needs to mark the arrival time of each request. Since the existing serverless platform does not support caching of service requests, it is necessary to add a task cache queue outside the serverless platform to enable batch processing support for the inference service.

[0034] Furthermore, the generation of request time series requires the statistics of service request data within a fixed time interval. The size of the interval determines the granularity of time series data statistics, which can also reflect different regular characteristics and directly determine the time span of algorithm service prediction. Since different algorithms have different service states and different statistical granularities of time series, this paper selects the batch execution time of the records obtained in the load container batch parameter analysis method stage as the reference standard to generate request time series.

[0035] Furthermore, for time series prediction, the quadratic exponential smoothing algorithm is selected for service prediction, which is actually to take the weighted average of historical data as the prediction result for the future moment, and to process the data at different times with unequal weights to get close to the real data. In this way, the predicted value of the service and the size of the trend of change are obtained.

[0036] Furthermore, it also includes a visualization module for graphically displaying the process of the automated machine learning module and is responsible for the collection and organization of process indicator data and progress data.

[0037] Furthermore, the prediction evaluation indicators are used to evaluate the prediction results through RMSE and MAE, and the prediction effect is judged by the error value of the predicted results.

[0038] Adaptive sliding window adjustment algorithm. Through the above-mentioned load container parameter analysis method and time series prediction work, the basic work of dynamically merging model requests has been completed. In the real workload scenario, the adaptive sliding window adjustment algorithm dynamically adjusts the adaptive sliding window size according to the trend change of the number of service requests, so as to meet the merging according to the size of the traffic under different window sizes, thereby prompting the performance of the inference service. It mainly includes the following steps:

[0039] To determine the trend change of service requests, the P(t) window discriminant function is used to determine whether the service request data is increasing or decreasing at the current time point. If it is increasing, the corresponding window becomes larger; if it is decreasing, the corresponding window becomes smaller.

[0040] Further, the service for window adjustment is determined. The batch size and batch execution time parameters obtained in the batch parameter analysis phase are introduced into the load container parameters.

[0041] Further, the increment ΔF of the predicted flow size is determined, and then the ratio Z of the predicted flow increment ΔF to the optimal batch size is calculated.

[0042] Furthermore, Z is optimized by the Sigmoid function to obtain σ(Z)

[0043] Furthermore, by multiplying the window discriminant function P(t), σ(Z) with the batch execution time, we can get the value of length Accumulate and get the new window size T length+1 .

[0044] Furthermore, the left and right boundaries of the window are determined. The left boundary movement strategy is to add the current left boundary to the completed service time as the new left boundary, and the right boundary is the new left boundary plus T length+1 As the new bounded size.

[0045] The advantage of the present invention compared with the prior art is that since the selection of the window is time-sensitive, the window size directly affects the efficiency of batch merging. Traditional methods for selecting the window size include sliding windows based on a fixed number of requests and sliding windows based on a fixed time. The window sizes of these two methods are limited to specific conditions and cannot meet the needs of dynamic changes in time series data in a short period of time. The adaptive window size in this article can be automatically scaled according to the traffic size, and the adjusted time window size is used as the left and right boundaries of the service request merge, so that different traffic sizes have different window sizes at different request times. The focus is on adaptively adjusting the window size. The biggest difference is that the batch merging algorithm of the adaptive sliding window allows multiple batches in the window, that is, when the window is expanded, when the request in the window is greater than the maximum batch size, it will be divided into multiple batches according to the greedy algorithm for calling, and when there are remaining requests in the window, they will be merged into the next window. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is the overall architecture diagram of the batch optimization system of the present invention.

[0047] Figure 2 It is a flow chart of batch processing parameter analysis of the present invention.

[0048] Figure 3 Merge flow chart for batch processing.

[0049] Figure 4 Merge architecture diagram for batch processing.

[0050] Figure 5 Publish flowcharts for platform algorithms.

[0051] Figure 6 Deploy the call flow graph for OpenFaaS. DETAILED DESCRIPTION

[0052] The technical solution of the present invention can be expressed as Figure 1 As shown, it mainly includes: load container batch analysis tool module and batch merging tool module, algorithm service calling module. Through the cooperation of the above two modules, the degree learning model reasoning batch optimization technology and system proposed in the present invention can be realized.

[0053] In the above modules, load container batch analysis tool module:

[0054] like Figure 4As shown in the left branch, the load container batch analysis tool solves the problem of optimal matching of container performance and service performance. By storing historical service requests to generate a service request dataset, the Batch generator divides the inference input parameters in the service request dataset into sets of different batch sizes, and then the container generator constructs containers of different memory sizes, automatically adjusts the request:cpu and request:memory parameters of spec:containers:resources in the OpenFaaS configuration file pod.yaml, and deploys the function through faas-cli deploy-f. / functon.yml.

[0055] After that, the load trainer will simulate the service request and send different batches of data in the Batch training set for reasoning. After each batch reasoning is completed, the load trainer also needs to enable multi-threaded asynchronous reasoning for each record in the Batch data to test the service performance without batch processing, and count the service performance indicators. When this environment container has completed testing all training sets, it will destroy this container and generate the next memory size container to continue the above steps. Finally, a load performance table is generated, which contains the performance parameters of all different batch sizes. The parameter selector will select the batch size and batch execution time parameters of this algorithm model in a certain memory size through custom algorithm rules. These two parameters will serve as key data to guide the batch merging algorithm of the adaptive sliding window in subsequent real-time traffic scenarios, and use this memory size as the online container environment of this algorithm model. The specific implementation is as follows:

[0056] The load performance table is as follows, and the parameter selection strategy is as follows

[0057]

[0058]

[0059] The following interface configuration is provided for the implementation process of this module.

[0060] Interface Name Interface meaning getArrayList(<request[input]>) Returns a dataset for a service request outMultMemoryPod(merory,core) Then return a created container outMultSizeBatch(size) Returns a fixed-size batch dataset collectTrainResult(pod, size) Returns the execution results of a specific container and batch size selectBestParam(result) Returns the optimal batch size and memory size

[0061] In the above modules, the batch merging tool module:

[0062] like Figure 4As shown in the right branch, the batch merging tool solves the problem of how to merge batches in real-time scenarios, and is used to improve service performance. For real-time service requests, enable the Nginx configuration to forward service requests to the batch merging tool, which is cached by the task cache queue, which is implemented by the LinkedBlockingQueue blocking queue. Next, the workload aggregator will count the request tasks in the queue according to the time interval. The time interval size is selected based on the batch execution time generated by the service container load stage of this algorithm as a reference standard to generate a time series of service requests. This sequence saves the traffic size of the current and historical time intervals.

[0063] Finally, through the time series prediction algorithm, recent inference service requests are intercepted from the time series to start prediction. The prediction result parameters obtained by the prediction algorithm, that is, the number of service requests and batch processing parameters (optimal batch processing size and optimal batch processing execution time) in the next time interval, are input into the adaptive window algorithm model. The adaptive window algorithm model adjusts the left and right boundaries of the current window, and then takes out the inference service requests in the window from the request cache queue to generate batch processing tasks and sends them to the corresponding OpenFaaS cluster algorithm container for execution according to the type of resources occupied by the inference service requests.

[0064] The specific implementation process is as follows: First, implement the prediction method of the service. Calculate the exponential smoothing value, infer the time series of service requests {X1, X2, X3, ... Xt}, is the smoothing value of the i-th period, Xi is the actual value of the i-th period, α is the smoothing coefficient (between 0 and 1, the smaller the α value, the more relevant the predicted value is to the recent data), and the recursive relationship is as follows: Then calculate the second exponential smoothing value. Second exponential smoothing needs to predict the sequence with trend elements by performing another weighted average. So S i (2) is the quadratic exponential smoothing value of the i-th period, S i (1) is the exponential smoothing value of the i-th period, and the formula is as follows: When there are T periods of data, we can predict the t+T predicted value. If we make a prediction for the t+1 period, we can set T to 1. t+T =a t +b t ×T, The parameter bt is used to analyze the future trend of service requests, and the future traffic size is determined based on the predicted value Yt+T. The prediction result directly determines the size change of the adaptive window.

[0065] Then comes the implementation of the adaptive sliding window adjustment algorithm. The first step of the adaptive sliding window adjustment algorithm is to determine the window discriminant function: determine whether the window is expanded or reduced. As shown in the formula, t represents the predicted current time point interval. When bt is greater than 0, it means that the traffic has increased, otherwise it has decreased.

[0066] Next, we need to determine the magnitude of the window adjustment. This article introduces the parameters for obtaining the batch size and batch execution time in the load container batch parameter analysis phase. The formula is as follows:

[0067] T length+1 =T length +(P(t)×σ(Z)×T batch )

[0068]

[0069]

[0070] ΔF=|Y t+1 -y t |

[0071] Among them, T length+1 Refers to the length of the adjusted window; Y batch ,T batch is the optimal batch size and execution time obtained through batch parameter analysis; Z is the predicted traffic increment as a percentage of the optimal batch size Y batch The ratio of y t Indicates the size of the real-time request volume in the previous time interval; σ(Z) is the activation function, and the Sigmoid function is used here. To solve the problem of Z value being too large due to the surge in service traffic, the Sigmoid activation function is used to map the Z value to the range of (0,1). Finally, the position movement strategy of the left and right boundaries of the window is:

[0072] W left+1 =W left +T finished

[0073] W right+1 =W left+1 +T length+1

[0074] Among them, W left+1 , W right+1 Refers to the adjusted left and right boundaries; T finishedis the time length of the batch requests completed in the window. From the above formula, it can be concluded that the maximum step size of each window adjustment is the execution time length of the optimal batch size in the load container batch parameter analysis. However, when the traffic is increasing, the window size will be extended each time, so the number of requests in the window will exceed the optimal batch size. At this time, the requests in the window are grouped into multiple batches for execution. When the service traffic decreases, the window is reduced until the left and right boundaries of the window overlap. At this time, each service request is not batched and called separately.

[0075] The following interface configuration is provided for the implementation process of this module.

[0076] Interface Name Interface meaning addQueue(<request[input]> ) Returns the saved service request queue countTimeInterval(<request[input]> ) Return request time queue initAndResizeWindow(ArrayList) Returns the initial and resized window size predictTimeSerires(ArrayList) Returns the parameters of a time series forecast Invoking(batch) Returns batch execution results

[0077] In the above modules, the algorithm service calls the module:

[0078] like Figure 5 , 6 As shown in the figure, when the release process enters the serverless platform stage, the deployment algorithm request is first forwarded to the OpenFaaS Provider through the Gateway. The Provider deploys through faas-netes according to the function template that the user has written, including configuring the Docker image, generating Kubernetes's Deployment, Service, Secret and other resource components, deploying the algorithm to the Pod container, and then publishing the function as a service. The resource components are managed by CRD through OpenFaaSOperator. When the algorithm makes a service call, the request route first reaches the Gateway, which forwards the request to the Watchdog monitor on port 8080 in the container, and transmits the HTTP request information through stdin and stdout to complete the service call.

[0079] Although the specific embodiments of the present invention are disclosed for the purpose of illustration, the purpose is to help understand the content of the present invention and implement it accordingly, those skilled in the art will understand that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the best embodiment, and the scope of the present invention is subject to the scope defined in the claims.

Claims

1. A batch processing optimization method for deep learning model reasoning, the steps comprising: 1) Obtain the online reasoning service request through the request interceptor, and save the reasoning input parameters in the reasoning service request into the preprocessing data set; 2) According to the type of resources occupied by the inference service request, the inference service request is divided into CPU type tasks and GPU type tasks; multiple containers are set for CPU type tasks, and different containers have different memory sizes and CPU core numbers; for GPU type tasks, several GPU type task containers with set memory sizes and GPU core numbers are generated; 3) For the inference service of the same model, if the inference service is a CPU type task, training data sets of different batch sizes are generated for the model according to the inference input parameters in the preprocessed data set and input into each container to perform a performance test on the model, and then a load performance table is generated according to the test result indicators; then the batch merge execution time Batchtime and the non-merge execution time NoBatchtime in the load performance table are compared, and the records in the load performance table are screened by comparing the ratio with the set threshold δ; then, based on the screened records, several records with the highest batch size occurrence frequency are selected, and the records with the smallest memory are selected from them; If the inference service is a GPU type task, generate training data sets of different batch sizes according to the inference input parameters in the preprocessed data set and input them into the GPU type task container to perform performance testing on the model, and select the record with the largest batch size that meets the threshold indicator condition; 4) Determine the optimal parameter based on the selected records in step 3), that is, the batch size Y in the selected records batch and batch execution time T batch ; 5) Cache the inference service requests received in real time into the task cache queue, and then the workload aggregator calculates the batch execution time T batch Determine a time interval, count the number of inference service requests in each time interval in the task cache queue, and obtain a time series; 6) The inference service requests in the most recent period are intercepted from the time series for prediction, and the number of service requests in the next time interval is obtained and input into the adaptive window algorithm model. The adaptive window algorithm model adjusts the left and right boundaries of the current window according to the input information, and then takes out the inference service requests in the window from the task cache queue to generate batch tasks and sends them to the corresponding container for execution according to the type of resources occupied by the inference service requests.

2. The method according to claim 1, characterized in that The method for predicting the number of service requests, the optimal batch size and the optimal batch execution time in the next time interval is as follows: let the time series be {X1, X2, X3, ... Xt}, Xt is the reasoning service request in the t-th time interval, let α be the smoothing coefficient; according to the recursive relationship Calculates the quadratic exponential smoothing value, where S i (2) is the quadratic exponential smoothing value of Xi, S i (1) is the exponential smoothing value of Xi; then the number of service requests Y in the next time interval is predicted t+1 =a t +b t , 3. The method according to claim 2, characterized in that The method for the adaptive window algorithm model to adjust the left and right boundaries of the current window according to the input information is as follows: first determine the window discriminant function Determine whether the window is expanded or reduced; then according to the formula T length+1 =T length +(P(t)×σ(Z)×T batch ) determines the adjusted window; wherein, ΔF=|Y t+1 -y t |;T length+1 is the adjusted window length, y t Indicates the real-time request size in t time intervals.

4. The method according to claim 1, 2 or 3, characterized in that: The strategy for adjusting the left and right borders of the current window is: left+1 =W left +T finished , W right+1 =W left+1 +T length+1 ; Among them, W left+1 , W right+1 are the adjusted left and right boundaries, T finished The length of time that the batch request has completed within the window.

5. The method according to claim 1, 2 or 3, characterized in that: If the corresponding ratio is recorded The record is retained.

6. A batch processing optimization system for deep learning model reasoning, characterized in that: It includes a load container batch analysis tool module and a batch merging tool module; in The load container batch analysis tool module is used to obtain the online reasoning service request through the request interceptor, and save the reasoning input parameters in the reasoning service request into the preprocessing data set; then, according to the type of resources occupied by the reasoning service request, the reasoning service request is divided into CPU type tasks and GPU type tasks; multiple containers are set for CPU type tasks, and different containers have different memory sizes and CPU core numbers; for GPU type tasks, several GPU type task containers with set memory sizes and GPU core numbers are generated; then, for the reasoning service of the same model, if the reasoning service is a CPU type task, training data sets of different batch sizes are generated for the model according to the reasoning input parameters in the preprocessing data set and input into each container to perform performance testing on the model, and then a load performance table is generated according to the test result indicators; then the batch merge execution time Batchtime and the non-merged execution time NoBatchtime in the load performance table are compared, and the records in the load performance table are screened by comparing the ratio with the set threshold δ; then, based on the screened records, several records with the highest batch size occurrence frequency are selected, and the records with the smallest memory are selected from them; If the inference service is a GPU type task, then a training data set with different batch sizes is generated according to the inference input parameters in the preprocessed data set and input into the GPU type task container to perform a performance test on the model, and the record with the largest batch size that meets the threshold indicator condition is selected; then the optimal parameter is determined according to the selected record, that is, the batch size Y in the selected record batch and batch execution time T batch ; The batch merging tool module is used to cache the inference service requests received in real time into the task cache queue, and then the workload aggregator calculates the batch execution time T. batch Determine a time interval, count the number of inference service requests in each time interval in the task cache queue, and obtain a time series; then intercept the inference service requests in the most recent period from the time series for prediction, obtain the number of service requests in the next time interval and input it into the adaptive window algorithm model, the adaptive window algorithm model adjusts the left and right boundaries of the current window according to the input information, then takes out the inference service requests in the window from the task cache queue to generate batch tasks and sends them to the corresponding container for execution according to the type of resources occupied by the inference service requests.

7. The system according to claim 6, characterized in that The method for predicting the number of service requests, the optimal batch size and the optimal batch execution time in the next time interval is as follows: let the time series be {X1, X2, X3, ... Xt}, Xt is the reasoning service request in the t-th time interval, let α be the smoothing coefficient; according to the recursive relationship Calculates the quadratic exponential smoothing value, where S i (2) is the quadratic exponential smoothing value of Xi, S i (1) is the exponential smoothing value of Xi; then the number of service requests Y in the next time interval is predicted t+1 =a t +b t , 8. The system according to claim 7, characterized in that The method for the adaptive window algorithm model to adjust the left and right boundaries of the current window according to the input information is as follows: first determine the window discriminant function Determine whether the window is expanded or reduced; then according to the formula T length+1 =T length +(P(t)×σ(Z)×T batch ) determines the adjusted window; wherein, ΔF=|Y t+1 -y t |;T length+1 is the adjusted window length, y t Indicates the real-time request size in t time intervals.

9. The system according to claim 6, 7 or 8, characterized in that: The strategy for adjusting the left and right borders of the current window is: left+1 =W left +T finished , W right+1 =W left+1 +T length+1 ; Among them, W left+1 , W right+1 are the adjusted left and right boundaries, T finished The length of time that the batch request has completed within the window.

10. The system according to claim 6, 7 or 8, characterized in that: If the corresponding ratio is recorded The record is retained.

Images