Dynamic batch processing and delay optimization method for deep learning model reasoning service

Abstract

The invention discloses a dynamic batch processing and delay optimization method for a deep learning model reasoning service, and the method comprises the steps: constructing a system architecture which comprises a request classification module, a dynamic batch processing scheduling module, a delay prediction and optimization module, and a reasoning execution module; accurate classification of reasoning requests, intelligent scheduling of dynamic batch processing and effective prediction and optimization of reasoning delay are realized. Wherein the dynamic batch processing scheduling module adopts a self-adaptive batch processing strategy based on reinforcement learning, and dynamically adjusts the batch processing size in combination with the priority, the type and the delay demand of a request; and the delay prediction and optimization module predicts reasoning delay by using a deep learning model, and performs optimization through resource dynamic allocation and model parameter adjustment. Experimental results show that the batch processing efficiency can be remarkably improved and the reasoning delay can be reduced on the premise of ensuring the reasoning precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of edge computing, in particular to a dynamic batch processing and delay optimization method for deep learning model inference service. BACKGROUND

[0002] A common way for terminal devices to provide intelligent services using deep learning is to run deep learning inference based on pre-trained models. Deploy the trained model to the production environment to provide real-time inference results for users. However, deep learning model inference has high requirements in terms of energy, memory and computing engine cycles.

[0003] Dynamic batch processing is one of the important means to improve the efficiency of deep learning model inference service. By combining multiple inference requests into a batch processing task for processing, computing resources can be fully utilized and the throughput of the model can be improved. However, the existing dynamic batch processing method has the following shortcomings: during dynamic batch processing, fixed batch size or simple rule-based adjustment is mostly used, which cannot adaptively adjust according to the characteristics of the inference request (such as priority, delay requirement, etc.) and the real-time state of the system, resulting in low batch processing efficiency and increased inference delay when the request volume fluctuates greatly or there are high-priority requests; when batch processing, the inference delay of the batch processing task is not accurately predicted, and there are no special optimization measures for delay, making it difficult for the inference service to meet the delay requirements in different application scenarios, especially for delay-sensitive applications, which may cause serious consequences; existing delay optimization methods mostly focus on model compression, quantization and other model optimization, and pay less attention to delay optimization during dynamic batch processing, which cannot improve the performance of the inference service as a whole. SUMMARY

[0004] The purpose of the present application is to provide a dynamic batch processing and delay optimization method for deep learning model inference service to solve the problems raised in the background.

[0005] To achieve the above purpose, the present application provides the following technical scheme: a dynamic batch processing and delay optimization method for deep learning model inference service, comprising the following steps:

[0006] Step 1: receiving an inference request, classifying the inference request through a request classification module to obtain different types of inference requests, including priority, data type and model type;

[0007] Step 2: dynamic batch processing of inference requests, the dynamic batch processing scheduling module adopts an adaptive batch processing strategy to dynamically adjust the batch size according to the classified inference requests to form a batch processing task;

[0008] Step 3, Delay Prediction and Optimization: The delay prediction and optimization module uses a deep learning model to predict the inference delay of batch processing tasks, and optimizes the delay by dynamically allocating resources and adjusting model parameters based on the prediction results.

[0009] Step 4: Output Results. The inference execution module performs inference execution on the optimized batch processing task and returns the inference results.

[0010] Furthermore, in step one, the request classification module extracts feature information from the inference request. The feature information includes the priority identifier of the request, the data type of the input data, and the type of model to be called. Based on the preset classification rules and machine learning classification model, the extracted feature information is classified to obtain different types of inference requests.

[0011] Furthermore, in step two, the adaptive batching strategy includes: constructing a reinforcement learning state space, which includes the number of inference requests currently waiting to be processed, the priority of each inference request, and the expected inference latency; and determining the reinforcement learning action space, which is an optional value for the batch size.

[0012] Furthermore, the framework for the state space of reinforcement learning is constructed, which consists of an agent, environment, state, and action.

[0013] The state space for reinforcement learning is constructed as follows: the number of inference requests waiting to be processed is denoted by N; the priority distribution of inference requests is represented by the vector P=[p_1,p_2,...,p_k], where k is the number of priority levels, p_i represents the proportion of requests of priority level i in the waiting queue; the expected inference latency distribution is represented by the vector L=[l_1,l_2,...,l_m], where m is the number of possible values ​​for batch size, and l_j represents the expected average inference latency when the batch size is j.

[0014] System resource utilization includes CPU utilization (C), GPU utilization (G), and memory utilization (M);

[0015] The mathematical representation of the state space is S=(N,P,L,C,G,M).

[0016] Determining the action space, let the action space be:

[0017] A = {a_1, a_2, ..., a_n}, where a_i is the specific batch size value. At each decision moment, the agent selects an action from the action space, that is, determines the current batch size.

[0018] Adaptive batch processing strategy optimization and execution, the agent stores the state S, action A and next state S' in the process of interacting with the environment to the experience pool D, and in the training process, a batch of experience data containing S, A and S' is randomly sampled from the experience pool D for updating the policy of the agent.

[0019] Policy updating and execution, the policy is updated using a deep reinforcement learning algorithm, and the agent selects a batch size action space according to the current state S through the trained policy, and applies the batch size to the dynamic batch scheduling to form a batch task.

[0020] Further, in step three, the deep learning model collects historical inference data, including batch size, input data features, model parameters and corresponding inference delay;

[0021] A delay prediction model is constructed, the delay prediction model adopts a neural network structure, the historical inference data is used to train the delay prediction model, and a trained delay prediction model is obtained, the related information of the batch task is input into the trained delay prediction model, and an inference delay prediction result is obtained.

[0022] The specific process of resource dynamic allocation for delay optimization is: according to the inference delay prediction result and the current resource usage of the system, the computing resources and storage resources required by each batch task are determined; and a resource scheduling algorithm is used to allocate resources to each batch task.

[0023] The specific process of delay optimization through model parameter adjustment is: according to the inference delay prediction result and the characteristics of the model, the adjustable model parameters are determined, the adjustable model parameters include the number of network layers, the number of neurons and the activation function; and a parameter optimization algorithm is used to adjust the adjustable model parameters.

[0024] Further, in step four, the steps of monitoring the inference execution module are monitored, and the related data of the inference execution are obtained, the related data including the inference progress, the actual inference delay and the resource usage; and according to the monitoring data, the dynamic batch processing strategy and the delay optimization strategy are dynamically adjusted.

[0025] Compared with the prior art, the beneficial effects of the present application are:

[0026] (1) Through the dynamic batch scheduling module, an adaptive batch processing strategy based on reinforcement learning is adopted, a state space containing the number of requests, priority, predicted delay and other dimensions is constructed, and a dynamic action space is combined, so that the agent can learn and optimize the batch size in real time, when high-priority requests surge, the strategy will automatically reduce the batch size to speed up processing; when the request quantity surges, the batch size is increased to improve the throughput, so that efficient processing is maintained in the scene of request quantity fluctuation and significant priority difference, and the flexibility is better.

[0027] (2) Through the delay prediction and optimization module, the delay of batch processing tasks is accurately predicted by using deep learning models such as LSTM and multilayer perceptron, data support is provided for optimization, and double means of resource dynamic allocation and model parameter adjustment are combined: for tasks with high delay prediction values, more CPU / GPU resources are allocated through resource scheduling strategies such as genetic algorithm, and resource allocation is reasonable; under the premise of ensuring accuracy, the calculation amount is reduced by adjusting parameters such as the number of network layers and the number of neurons to meet the requirements of delay-sensitive applications.

[0028] (3) Through the monitoring module, real-time collection of inference progress, actual delay, resource usage and other data is performed, and batch processing strategies and optimization parameters are dynamically adjusted. When the deviation between the actual delay and the prediction value exceeds the threshold, the retraining of the delay prediction model is automatically triggered; when the resource allocation is unbalanced, the scheduling algorithm parameters are adjusted in real time, and the stability of the inference service is improved. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is the flowchart of the structure of the application;

[0030] Figure 2 is the working principle diagram of the request classification module of the application;

[0031] Figure 3 is the principle diagram of the adaptive batch processing strategy based on reinforcement learning of the application;

[0032] Figure 4 is the working process diagram of the delay prediction and optimization module of the application;

[0033] Figure 5 is the system structure diagram of the application;

[0034] Figure 6 is the diagram without dynamic batch processing;

[0035] Figure 7 is the diagram of no procrastination dynamic batch processing;

[0036] Figure 8 is the diagram of dynamic batch processing delay=X / 2. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0038] Embodiments:

[0039] Referring to Figures 1-8 , the present application provides a technical solution: a dynamic batch processing and delay optimization method for deep learning model inference service;

[0040] As Figure 1 shown, it includes the following steps:

[0041] Step 1: Receive inference request and classify

[0042] Inference requests can come from various client applications, such as image recognition applications, voice assistant applications, etc.

[0043] Classify the inference request through the request classification module, as Figure 2 shown. First, extract the feature information in the inference request, such as the priority identifier of the request (such as high, medium, low), the data type of the input data (such as image data, text data, voice data, etc.), and the model type to be called (such as convolutional neural network model, recurrent neural network model, etc.). Then, combined with the pre-set classification rules (such as requests with high priority identifier are classified as high priority type) and machine learning classification models (such as support vector machine, decision tree, etc.), the extracted feature information is classified to obtain different types of inference requests. Through classification, targeted basis can be provided for subsequent dynamic batch processing scheduling and delay optimization.

[0044] Step 2: Dynamic batch processing scheduling

[0045] The dynamic batch processing scheduling module adopts an adaptive batch processing strategy based on reinforcement learning to dynamically adjust the batch size according to the classified inference requests, forming a batch processing task. As Figure 3 shown. Specifically, it includes:

[0046] Construct the state space of reinforcement learning: the state space contains the number of current waiting processing inference requests, the priority of each inference request, the predicted inference delay, etc. These information can reflect the current system state and the characteristics of the inference request.

[0047] Determine the action space of reinforcement learning: the action space is the selectable value of the batch size, for example, the batch size can be set in the range of 1-100, and the action is to select a specific value from this range as the current batch size.

[0048] 2.1 Reinforcement learning framework construction

[0049] The reinforcement learning framework is composed of agent (Agent), environment (Environment), state (State), and action (Action). In this strategy:

[0050] Agent: Responsible for adjusting the batch size, aiming to find the optimal batch size adjustment strategy through learning to minimize inference delay and improve batch efficiency.

[0051] Environment: The deep learning model inference service system, including the queue of inference requests to be processed, computing resources, storage resources, etc. The environment will change its state according to the agent's action (batch size adjustment).

[0052] 2.2. State Space Construction

[0053] The state space is used to describe the environment state of the agent and is the basis for the agent to make decisions. In this strategy, the construction of the state space considers both the characteristics of the inference request and the real-time state of the system, and specifically includes the following dimensions:

[0054] Number of inference requests waiting to be processed: denoted by N, reflecting the current system load. N ranges from 0 to the maximum number of requests the system can accommodate, for example, 0-1000. This parameter allows the agent to understand the size of the requests to be processed, and when N is large, the batch size may need to be increased to improve processing efficiency; when N is small, the batch size may need to be reduced to reduce delay.

[0055] Priority distribution of inference requests: represented by the vector P = [p_1, p_2,..., p_k], where k is the number of priority levels (such as high, medium, and low), and p_i represents the proportion of requests of the i-th priority level in the waiting queue. For example, if high-priority requests account for p_1 = 0.6, medium-priority requests p_2 = 0.3, and low-priority requests p_3 = 0.1, the agent will be more inclined to prioritize high-priority requests when making decisions and will set a smaller batch size for high-priority requests to speed up processing.

[0056] Predicted inference delay distribution: represented by the vector L = [l_1, l_2,..., l_m], where m is the number of possible batch size values, and l_j represents the predicted average inference delay when the batch size is j. This data can be provided by a delay prediction model and helps the agent to predict the delay under different batch sizes, making more reasonable decisions.

[0057] System resource utilization: including CPU utilization C, GPU utilization G, and memory utilization M. System resource utilization reflects the current system busy degree, when C, G or M is high, it means that the system resources are tight, at this time if the batch size is increased, the delay may increase significantly, the agent needs to adjust carefully; when the resource utilization is low, the batch size can be increased appropriately to improve resource utilization.

[0058] The mathematical representation of the state space is S=(N, P, L, C, G, M), through which the agent can fully grasp the current environmental state.

[0059] 2.3 Action Space Determination

[0060] The action space is the set of actions that the agent can perform, and in this strategy, the action is the batch size adjustment value. The optional values of the batch size are determined according to the hardware performance of the system, the characteristics of the model, and the average data volume of the inference request. Let the action space be A={a_1, a_2,..., a_n}, where a_i is a specific batch size value, for example, A={1, 2, 4, 8, 16, 32, 64, 128}.

[0061] The agent selects an action from the action space at each decision-making moment, i.e., determines the current batch size. For example, when the agent selects action a_5=16, it means that the current inference requests waiting for processing will be batched in groups of 16.

[0062] 2.4. Strategy Optimization and Execution

[0063] Strategy Initialization: Use the ε-greedy strategy as the initial strategy, i.e., the agent randomly selects an action with probability ε and selects the action that it currently considers optimal with probability 1-ε. The initial value of ε is large (such as 0.5), allowing the agent to fully explore the action space; as learning progresses, the value of ε gradually decreases (such as linearly decaying to 0.1), allowing the agent to make more use of the learned optimal strategy.

[0064] Experience Replay: During the interaction with the environment, the agent stores the state S, action A, and next state S' of each time in the experience pool D. During training, a batch of experience data (S, A, S') is randomly sampled from the experience pool D to update the agent's strategy, breaking the correlation between samples and improving the stability of training.

[0065] Strategy Update: Use deep reinforcement learning algorithms (such as DQN, PPO, etc.) to update the strategy. Taking the DQN algorithm as an example, a deep neural network Q(S, A; θ) is constructed to estimate the action value function, where θ is the network parameter. The goal is to make Q(S, A; θ) as close as possible to the target action value y* max_{A'} Q(S', A'; θ'), where y is the discount factor (0 < y < 1) and θ' is the parameter of the target network. Update the network parameter θ by minimizing the loss function L(θ)=E[(y*max_{A'}Q(S',A';θ')-Q(S,A;θ))^2], and periodically copy θ to θ'.

[0066] Policy Execution: During the actual inference service process, the agent selects a batch size A according to the current system state S through the trained policy, and applies the batch size to the dynamic batch scheduling to form a batch task. At the same time, real-time environmental feedback and the next state S' are collected and stored in the experience pool for subsequent policy optimization, so that the policy can continuously adapt to changes in the environment.

[0067] Through the above adaptive batch strategy based on reinforcement learning, the dynamic batch scheduling module can dynamically adjust the batch size according to the priority, quantity of inference requests, and system resource state, etc., while improving the efficiency of batch processing and resource utilization, effectively reducing the inference delay, and meeting the delay requirements of different inference requests.

[0068] Step 3: Delay prediction and optimization

[0069] The workflow of the delay prediction and optimization module is shown in Figure 4 First, the inference delay of the batch task is predicted using a deep learning model:

[0070] Collect historical inference data: including batch size, input data features (such as data dimension, data distribution, etc.), model parameters (such as network layer number, neuron number, etc.), and corresponding inference delay.

[0071] Build a delay prediction model: use neural network structures such as multilayer perceptron, LSTM, etc. The input of this model is the relevant information of the batch task, and the output is the prediction value of the inference delay.

[0072] Train the delay prediction model: use historical inference data to train the model, adjust the model parameters through the backpropagation algorithm to minimize the prediction error of the model. Increase the "model compression parameters" (such as quantization precision, pruning rate) as input features to improve the delay prediction accuracy of the compressed model. In the model parameter adjustment stage, in addition to adjusting the network layer number and neuron number, introduce quantization strategies (such as INT8 quantization for non-critical layers) and pruning rules (such as removing neurons with activation values below a threshold); the dynamic batch scheduling module considers the computational efficiency of the compressed model (such as the speed improvement ratio of each batch) in the state space of reinforcement learning when determining the batch size, so that the batch strategy is more suitable for the characteristics of the compressed model.

[0073] Perform delay prediction: input the batch size, input data features, model parameters, etc. of the batch task into the trained delay prediction model to obtain the inference delay prediction result.

[0074] Then, according to the prediction result, the delay is optimized:

[0075] Resource dynamic allocation: According to the inference delay prediction results and the current resource usage of the system (such as CPU utilization, GPU utilization, memory usage, etc.), determine the computing resources (such as CPU core number, GPU computing power) and storage resources (such as memory space) required by each batch processing task. Adopt resource scheduling algorithms such as genetic algorithm, particle swarm optimization algorithm, etc., to minimize the overall inference delay, and allocate resources to each batch processing task. For example, for batch processing tasks with large delay prediction values, allocate more computing resources to speed up their processing speed.

[0076] Model parameter adjustment: According to the inference delay prediction results and the characteristics of the model, determine the adjustable model parameters such as the number of network layers, the number of neurons and the activation function, etc. Adopt parameter optimization algorithms such as stochastic gradient descent algorithm, simulated annealing algorithm, etc. On the premise of meeting the inference accuracy requirements, adjust the adjustable model parameters to reduce the inference delay. For example, appropriately reducing the number of network layers or the number of neurons can reduce the computational complexity of the model, thereby reducing the inference delay, but the inference accuracy of the adjusted model needs to be ensured not to be lower than the preset threshold.

[0077] Step 4: Inference execution and return of results

[0078] The dynamic batch scheduling module adjusts the action space according to the optimal batch range supported by the inference module when generating batch processing tasks, so that the batch size falls within the engine optimization interval.

[0079] During the training of the delay prediction model, "inference module optimization parameters" (such as operator fusion strategy, memory pool size) are added as features to ensure that the prediction results are consistent with the actual execution environment.

[0080] The monitoring module collects the running indicators of the inference module (such as operator execution time, memory fragmentation rate) in real time and feeds back to the dynamic batch scheduling module. When the engine performance fluctuates (such as memory reuse efficiency decreases), the batch size is automatically reduced to avoid a sharp increase in delay

[0081] The inference module performs inference execution on the optimized batch processing task, uses the allocated resources and adjusted model parameters to perform parallel processing on multiple inference requests in the batch processing task, obtains the inference results, and returns the inference results to the corresponding client.

[0082] In addition, in order to ensure the dynamic adaptability and stability of the method, the inference execution process can also be monitored. Through the monitoring module, real-time acquisition of relevant data of inference execution is performed, such as inference progress, actual inference delay and resource usage, etc. According to the monitoring data, the dynamic batch processing strategy and the delay optimization strategy are dynamically adjusted. For example, if the actual inference delay deviates greatly from the predicted delay, the delay prediction model is retrained; if the delay of some tasks is still high after resource allocation, the parameters of the resource scheduling algorithm are adjusted or resource allocation is performed again.

[0083] In the application process, as shown in Figure 6 , without dynamic batch processing, one batch is processed at a time, and each batch separately occupies X time, which consumes a lot of time.

[0084] As shown in Figure 7 , ideal dynamic batch processing, combined requests, reduced delay and waiting, total time compression to 2X, reduced delay and waiting time.

[0085] As shown in Figure 8 , dynamic batch processing with delay, a commonly used compromise strategy, total time 2X (as ideal dynamic batch processing), but with X / 2 waiting delay, more feasible in reality, with very short waiting (X / 2) to exchange for higher hardware utilization, avoiding small batch frequent execution waste of computing power.

[0086] Although embodiments of the present application have been shown and described, it will be understood by those having ordinary skill in the art that various changes, modifications, substitutions and alterations can be made hereto without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A dynamic batch processing and latency optimization method for deep learning model inference services, characterized in that, Includes the following steps: Step 1: Receive inference requests and classify them using the request classification module to obtain different types of inference requests, including priority, data type, and model type. Step 2: Dynamic batch processing of inference requests. The dynamic batch processing scheduling module dynamically adjusts the batch size based on the classified inference requests using an adaptive batch processing strategy to form batch processing tasks. Step 3, Delay Prediction and Optimization: The delay prediction and optimization module uses a deep learning model to predict the inference delay of batch processing tasks, and optimizes the delay by dynamically allocating resources and adjusting model parameters based on the prediction results. Step 4: Output Results. The inference execution module performs inference execution on the optimized batch processing task and returns the inference results.

2. The dynamic batch processing and latency optimization method for deep learning model inference services according to claim 1, characterized in that: In step one, the request classification module extracts feature information from the inference request. The feature information includes the priority identifier of the request, the data type of the input data, and the type of model to be called. Based on the preset classification rules and machine learning classification model, the extracted feature information is classified to obtain different types of inference requests.

3. The dynamic batch processing and latency optimization method for deep learning model inference services according to claim 1, characterized in that: In step two, the adaptive batching strategy includes: constructing the state space of reinforcement learning, which includes the number of inference requests currently waiting to be processed, the priority of each inference request, and the expected inference latency; and determining the action space of reinforcement learning, which is an optional value of the batch size.

4. The dynamic batch processing and latency optimization method for deep learning model inference services according to claim 3, characterized in that: The framework for constructing the state space of reinforcement learning consists of an agent, environment, state, and action. The state space for reinforcement learning is constructed as follows: the number of inference requests waiting to be processed is denoted by N; the priority distribution of inference requests is represented by the vector P=[p_1,p_2,...,p_k], where k is the number of priority levels, p_i represents the proportion of requests of priority level i in the waiting queue; the expected inference latency distribution is represented by the vector L=[l_1,l_2,...,l_m], where m is the number of possible values ​​for batch size, and l_j represents the expected average inference latency when the batch size is j. System resource utilization includes CPU utilization (C), GPU utilization (G), and memory utilization (M); The mathematical representation of the state space is S=(N,P,L,C,G,M).

5. The dynamic batch processing and latency optimization method for deep learning model inference services according to claim 4, characterized in that: Determining the action space, let the action space be: A = {a_1, a_2, ..., a_n}, where a_i is the specific batch size value. At each decision moment, the agent selects an action from the action space, that is, determines the current batch size.

6. The dynamic batch processing and latency optimization method for deep learning model inference services according to claim 5, characterized in that: Adaptive batch processing strategy optimization and execution: During the interaction between the agent and the environment, the agent stores each state S, action A, and next state S' into the experience pool D. During training, a batch of experience data containing S, A, and S' is randomly sampled from the experience pool D to update the agent's strategy.

7. The dynamic batch processing and latency optimization method for deep learning model inference services according to claim 6, characterized in that: Policy update and execution: Deep reinforcement learning algorithm is used to update the policy. The agent selects the batch size action space according to the current state S through the trained policy, and applies the batch size to the dynamic batch scheduling to form batch tasks.

8. The dynamic batch processing and latency optimization method for deep learning model inference services according to claim 1, characterized in that: In step three, the deep learning model collects historical inference data, including batch size, input data features, model parameters, and corresponding inference latency; A delay prediction model is constructed using a neural network structure. The model is trained using historical inference data to obtain a trained delay prediction model. Relevant information of the batch processing task is input into the trained delay prediction model to obtain the inference delay prediction result.

9. The dynamic batch processing and latency optimization method for deep learning model inference services according to claim 1, characterized in that: In step three, the specific process of dynamic resource allocation for latency optimization is as follows: based on the inference latency prediction results and the current resource usage of the system, determine the computing and storage resources required for each batch processing task; and use a resource scheduling algorithm to allocate resources to each batch processing task. The specific process of delay optimization through model parameter adjustment is as follows: Based on the inference delay prediction results and the characteristics of the model, determine the adjustable model parameters, including the number of network layers, the number of neurons, and the activation function; and use a parameter optimization algorithm to adjust the adjustable model parameters.

10. The dynamic batch processing and latency optimization method for deep learning model inference services according to claim 1, characterized in that: In step four, the steps of the inference execution module are monitored to obtain relevant data on inference execution, including inference progress, actual inference latency, and resource usage. Based on the monitoring data, the dynamic batch processing strategy and latency optimization strategy are dynamically adjusted.

Images