Edge server adaptive batch processing method for edge inference tasks

Abstract

The application discloses an edge server adaptive batch processing method for edge inference tasks, and relates to how to adaptively adjust the batch size during inference according to the arrival of tasks in the edge scene to optimize the completion time of inference tasks. The method firstly measures the calculation time of the server under various batch sizes to perform inference tasks, thereby fitting the corresponding mapping function of batch size to calculation time; then a certain window size is set to predict the task arrival time in the window; the dynamic programming method is used to calculate the optimal batch processing mode of the tasks to obtain the corresponding batch size; the corresponding batch processing is performed until the number of executed tasks exceeds the window size, the window is slid, and the next prediction and dynamic programming are performed, so that the optimal batch size is adaptively adjusted during the edge server operation.

Description

Technical Field

[0001] This invention relates to edge servers, and more particularly, to an adaptive batch processing method for edge servers oriented towards edge inference tasks. Background Technology

[0002] In recent years, with the continuous emergence of deep learning applications, deploying these applications and performing inference tasks in edge scenarios has become a hot topic. In edge scenarios, the timeliness of task processing is a primary concern for users. However, due to the limited computing power of edge devices, the timeliness of deep learning tasks is difficult to guarantee.

[0003] Batch processing refers to packaging several similar tasks together and then processing these packages simultaneously on a server. In batch processing, deep learning tasks can make fuller use of hardware computing power, improving task completion efficiency and reducing overall completion time. Therefore, batch processing has become an important technical approach for processing deep learning tasks on edge devices.

[0004] There are two main types of existing batch processing methods: static batch processing and opportunistic batch processing.

[0005] Static batch processing sets a fixed batch size on the server. Whenever the number of tasks arriving on the server reaches this batch size, the tasks are packaged and sent to the server for execution. This method typically sets the batch size based on experience and cannot be adjusted according to real-time arrival data, resulting in poor flexibility. Often, long task arrival intervals lead to lengthy waiting times to meet the fixed batch size, significantly reducing task completion timeliness; or a large number of tasks arrive in a short period, resulting in only a small number of tasks being packaged at a time, leading to low completion efficiency. In short, a fixed batch size cannot effectively handle the dynamic nature of task arrivals in real-world scenarios.

[0006] Opportunistic batching sets a fixed time threshold on the server. If the next task arrives within the time threshold, it is packaged together with the current task, and the time threshold is refreshed. This process continues until no new tasks arrive when the time threshold expires. Similar to static batching, a fixed time threshold cannot effectively handle the dynamic nature of task arrivals in real-world scenarios.

[0007] In summary, this invention addresses the problem of how edge servers can adjust the batch size of deep learning inference tasks in edge scenarios based on the actual arrival status of tasks. It designs an adaptive batch processing mechanism to optimize task completion time. Summary of the Invention

[0008] To address the technical challenge of adjusting batch size based on task arrival status in deep learning inference tasks at edge environments, this invention proposes an adaptive batch processing method for edge servers. First, the computation time of the server performing inference tasks at various batch sizes is measured to fit a mapping function between batch size and computation time. Then, a window size is set, and the arrival time of tasks within the window is predicted. Dynamic programming is used to calculate the optimal batch processing method for these tasks, yielding the corresponding batch size. The corresponding batch processing is executed until the number of tasks exceeds the window size. The window is then slid, and the next prediction and dynamic programming iteration is performed. This adaptively adjusts the batch size at runtime, achieving the technical effect of reducing task completion time.

[0009] Compared with the prior art, the technical advantages of the method of the present invention are as follows:

[0010] ① This invention effectively models the relationship between edge server computing time and batch size, enabling the benefits of batch processing to be quantified and allowing for more precise optimization of batch size.

[0011] ② This invention uses a dynamic programming algorithm, which can set different batch sizes for different batches of tasks, enhancing the flexibility of batch processing and improving the efficiency of batch processing.

[0012] ③ This invention uses a sliding window to predict task arrival, which can more effectively capture the characteristics of task arrival, thereby setting the batch size for different batches of tasks more accurately and improving the batch processing effect.

[0013] See Figure 2 As shown, an adaptive batch processing method for edge servers oriented towards edge inference tasks according to the present invention includes the following steps:

[0014] Step 1: Model the relationship between server batch size and computation time;

[0015] Step 101: Calculate the computation time for different batch sizes;

[0016] Execute tasks of different batch sizes on edge servers and calculate the task computation time.

[0017] Step 102: Fit the functional relationship between server parallelism and batch size;

[0018] Step 102A, the server parallelism η can be calculated from the task computation time L of different batch sizes b and the single task computation time c, that is...

[0019] Step 102B: Fit the relationship between server parallelism η and batch size b, η = f(b), thus obtaining the fitting function of parallelism-batch size, as follows. Figure 3 As shown (using Tesla V100 and Jetson AGX Xavier as examples).

[0020] Step 103: Fit the functional relationship between calculation time and batch size;

[0021] By fitting the computation time to the batch size, i.e., L = f(b) × b × c, a fitting function for computation time versus batch size is obtained, such as... Figure 4 As shown (using Tesla V 100 and Jetson AGX Xavier as examples).

[0022] Step 2: Set the batch size based on the prediction window;

[0023] Step 201, set the prediction window size;

[0024] On the edge server, a fixed prediction window size ω is set, and the size of the window is the number of prediction tasks.

[0025] Step 202: Predict the task arrival time;

[0026] Step 202A: On the edge server, record the arrival times of tasks within the prediction window ω, denoted as a1, a2, ..., a ω Then calculate the average arrival interval I of the task, and

[0027] Step 202B, it is estimated that subsequent tasks will follow... Uniform arrival, the arrival time of any subsequent task is denoted as . Record the predicted arrival time series of subsequent tasks as A 预测 And A 预测 ={a1,a2,…,a ω ,a ω+1 ,…,a n-后 ,…,a N}

[0028] n represents any task.

[0029] a ω This represents the arrival time of the ω-th task.

[0030] a ω+1 This represents the arrival time of the (ω+1)th task.

[0031] a n-后 Indicates that it is located at a ω The arrival time of the subsequent tasks.

[0032] a N This indicates the arrival time of the last task.

[0033] Step 203: Perform dynamic planning on the batch size;

[0034] Step 203A: Initialize the completion time E0, and assign the value 0 to E0 (E0 = 0);

[0035] Initialization start time S 0,(1-N) And S 0,(1-N) Assigned the value 0(S) 0,(1-N) =0);

[0036] Initialize the dynamic programming batch size B0, and assign B0 a value of 0 (B0 = 0);

[0037] Step 203B: Set the task parameter n, and assign the value 1 to n (n = 1);

[0038] Step 203C, Set the current completion time. min , and end min =∞;

[0039] Step 203D, set the current maximum batch size b max , and b max = min(n, max_b), where max_b is the maximum batch size that the server can execute;

[0040] Step 203E: Set the batch size parameter b, and assign b a value of 1 (b = 1);

[0041] Step 203F: Calculate the end time of the previous batch task. 上 , and end 上 =E n-b E n-b This represents the end time of a task with a batch size of b that is located before the nth task.

[0042] Step 203G: Calculate the start time S of the current batch processing task. n,b And S n,b =max(A 预测,n end 上 A 预测,n This represents the predicted arrival time series of tasks preceding the nth task.

[0043] Step 203H: Calculate the end time of the current batch processing task, end, and end = S. n,b +f(b)×b×c;

[0044] Step 203I, if end < endmin If yes, then execute steps 203J-203L; otherwise, execute step 203M.

[0045] Step 203J, record the current completion time. min =end;

[0046] Step 203K: Record the completion time E of the nth task. n And E n The value is assigned to end(E) n =end);

[0047] Step 203L: Record the dynamic programming batch size B of the nth task. n And B n Assigned the value b(B) n =b);

[0048] Step 203M, set the batch size parameter b = 2,...,b max Repeat steps 203F to 203I;

[0049] Step 203N: Set task parameters n = 2, ..., N, and repeat steps 203C to 203I;

[0050] Step 2030: Set the traversal variable t, and assign t the value N (t = N);

[0051] Step 203P: Set up an array called Batch to record the batch size;

[0052] Step 203Q: Add the dynamic programming batch size belonging to the traversal variable t to the beginning of the array Batch.

[0053] Step 203R: Update the iterable variable t, and assign t the value...

[0054] Step 203S, if Then repeat steps 203Q-203R; otherwise, return an array Batch that records the batch size.

[0055] Step 204: Execute the batch processing task in the window;

[0056] Based on the obtained batch size array Batch, all tasks (1,2,…,n,…,N) are packaged and handed over to the edge server for batch processing until the number of tasks exceeds the window size ω.

[0057] Step 205: Determine whether the task has been completed;

[0058] Determine whether the server has finished processing all tasks (1,2,…,n,…,N). If it has, end the process; otherwise, when the server has finished processing the assigned batch tasks, move the window forward by ω and repeat steps 202-205. Attached Figure Description

[0059] Figure 1 This is a structural block diagram of the edge reasoning system.

[0060] Figure 2 This is a flowchart of the edge server adaptive batch processing for edge inference tasks according to the present invention.

[0061] Figure 3 This is a graph showing the relationship between parallelism and batch size for different types of servers.

[0062] Figure 4 This is a graph showing the relationship between computation time and batch size for different types of servers.

[0063] Figure 5 This is a comparison chart of the completion times of various methods for task Poisson arrival on the edge server (Tesla V100).

[0064] Figure 6 This is a comparison chart of the completion times of various methods for task Poisson arrival, with the edge server being Jetson AGX Xavier.

[0065] Figure 7 This is a comparison chart of the completion times of various methods for tasks generated based on real data, with the edge server being a Tesla V100.

[0066] Figure 8 The task is generated based on real data, and the edge server is Jetson AGX Xavier. The completion time of each method is shown in the graph. Detailed Implementation

[0067] The method of the present invention will now be described in further detail with reference to the accompanying drawings. The examples of the parameters listed are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

[0068] In this invention, the edge reasoning system is as follows: Figure 1 As shown, the task requester (taking a camera as an example) transmits the data of the generated task (taking image recognition as an example) to the edge server via a wireless network. After receiving these tasks, the edge server divides the tasks into different batches according to a certain batch size and performs batch processing.

[0069] For edge servers, the relationship between batch size and computation time is represented as L = f(b) × b × c. Here, L is the computation time for batch processing b tasks on the edge server, c is the computation time for processing a single task on the edge server, b represents the batch size, and f(b) is a fitting function with b as the independent variable and server parallelism η as the dependent variable, i.e., η = f(b). Server parallelism refers to the efficiency of server hardware relative to processing a single task when batching tasks. Batch size b also refers to the number of tasks (N) divided into batches.

[0070] The arrival time series of multiple tasks (1,2,…,n,…,N) to the edge server is denoted as A={a1,a2,…,a…}. n ,…,a N}, where: a1 represents the arrival time of the first task, a2 represents the arrival time of the second task, a n Let a represent the arrival time of the nth task. N This indicates the arrival time of the last task. The subscript n represents the task number, and N represents the total number of tasks. For ease of explanation, a... n Also known as the arrival time of any task. In this invention, the predicted arrival times of tasks within the window recorded on the edge server are a1, a2, ..., a ω And (a1,a2,…,a) ω )∈A.

[0071] This invention utilizes computer hardware, and the software is developed and implemented using the Python language, and is simulated in a Python environment (version 3.8).

[0072] In the simulation experiment, four simulation scenarios were set up to verify the effectiveness of the invention.

[0073] Scenario 1: Tasks follow a Poisson process with an average arrival interval of 10ms. The total number of tasks is N=200, and the edge server is a Tesla V 100. A comparison of the effectiveness of the method of this invention with existing methods is provided. Figure 5 As shown, the numbers in the bar chart represent the speedup compared to not using batch processing. Scenario 2: The tasks follow a Poisson process with an average arrival interval of 10ms, the total number of tasks N=200, and the edge server is Jetson AGX Xavier. The effect comparison between the method of this invention and existing methods is as follows: Figure 6 As shown.

[0074] The results from scenarios one and two demonstrate that the method described in this invention significantly reduces completion time and improves batch processing speedup compared to static batch processing or opportunistic batch processing. This result can be verified on two different servers. On a high-performance Tesla V 100 server, the speedup of this invention's method reaches 83.9 times compared to methods without batch processing, significantly exceeding the best result of previous methods (57.5 times). Even on a lower-performance Jetson AGX Xavier server, our method achieves a 10.1 times speedup, surpassing the best performance of previous methods (9.1 times). The effectiveness of this method has been successfully verified.

[0075] Scenario 3: Simulation tasks are generated based on a Microsoft Azure dataset, with a total of N=1000 tasks. The edge server is a Tesla V100. The effectiveness of the method described in this invention is compared with existing methods. Figure 7 As shown. Scenario 4: Generating simulation tasks based on a Microsoft Azure dataset, with a total of N=1000 tasks, and using a Jetson AGX Xavier edge server. The effectiveness of the method of this invention compared to existing methods is as follows: Figure 8 As shown.

[0076] In scenarios three and four, the tasks arrive from real-world datasets, making the arrival of tasks more dynamic. In these scenarios, the method of this invention still achieves optimal results on both types of servers.

[0077] In summary, it can be seen that the method of the present invention achieves the best results under various task arrival conditions on different servers.

[0078] This invention addresses the technical problem of adaptively adjusting the batch size of deep learning inference tasks based on task arrival status in edge scenarios. First, the invention measures the computation time of inference tasks on the server under various batch sizes, thereby fitting a mapping function from batch size to computation time. Then, a certain window size is set, and the arrival time of tasks within the window is predicted. Dynamic programming is used to calculate the optimal batch processing method for these tasks, obtaining the corresponding batch size. The corresponding batch processing is executed until the number of tasks executed exceeds the window size. The window is then slid, and the next prediction and dynamic programming are performed. This allows for adaptive adjustment to the optimal batch size during runtime, achieving the technical effect of reducing task completion time.

Claims

1. An adaptive batch processing method for edge servers for edge inference tasks, characterized in that... It includes the following steps: Step 1: Model the relationship between server batch size and computation time; Step 101: Calculate the computation time for different batch sizes; Execute tasks of different batch sizes on edge servers and analyze the task computation time. Step 102: Fit the functional relationship between server parallelism and batch size; Step 102A, the server parallelism η is calculated from the task computation time L of different batch sizes b and the single task computation time c, that is... Step 102B: Fit the relationship between server parallelism η and batch size b, η = f(b), to obtain the fitting function of parallelism - batch size; Step 103: Fit the functional relationship between calculation time and batch size; The computation time and batch size are fitted together, i.e., L = f(b) × b × c, thus obtaining the fitting function of computation time - batch size; Step 2: Set the batch size based on the prediction window; Step 201, set the prediction window size; On the edge server, a fixed prediction window size ω is set, and the size of the window is the number of prediction tasks; Step 202: Predict the task arrival time; Step 202A: On the edge server, record the arrival times of tasks within the prediction window ω, denoted as a1, a2, ..., a ω Then calculate the average arrival interval I of the task, and Step 202B, it is estimated that subsequent tasks will follow... Uniform arrival, let a be the arrival time of any subsequent task. n-后 =a ω +(n-ω)×I, Record the predicted arrival time series of subsequent tasks as A 预测 And A 预测 ={a1,a2,...,a ω ,a ω+1 ,...,a n-后 ,...,a N }; n represents any task; N represents the total number of tasks; a1 represents the arrival time of the first task; a2 represents the arrival time of the second task; a ω This represents the arrival time of the ω-th task; a ω+1 This represents the arrival time of the (ω+1)th task; a n-后 Indicates that it is located at a ω The arrival time of subsequent tasks; a N Indicates the arrival time of the last task; Step 203: Perform dynamic planning on the batch size; Step 203A: Initialize the completion time E0, and assign the value 0 to E0; Initialization start time S 0,(1-N) And S 0,(1-N) The value is assigned to 0; Initialize the dynamic programming batch size B0, and assign B0 a value of 0; Step 203B: Set the task parameter n, and assign the value 1 to n; Step 203C, Set the current completion time. min , and end min =∞; Step 203D, set the current maximum batch size b max , and b max = min(n, max_b), where max_b is the maximum batch size that the server can execute; Step 203E: Set the batch size parameter b, and assign b a value of 1; Step 203F: Calculate the end time of the previous batch task. 上 , and end 上 =E n-b E n-b This represents the end time of a task with a batch size of b that is located before the nth task. Step 203G: Calculate the start time S of the current batch processing task. n,b And S n,b =max(A 预测,n end 上 A 预测,n This represents the predicted arrival time series of tasks preceding the nth task; Step 203H: Calculate the end time of the current batch processing task, end, and end = S. n,b +f(b)×b×c; Step 203I, if end < end min If yes, then execute steps 203J-203L; otherwise, execute step 203M. Step 203J, record the current completion time. min =end; Step 203K: Record the completion time E of the nth task. n And E n The value is assigned to end; Step 203L: Record the dynamic programming batch size B of the nth task. n And B n Assign the value b; Step 203M, set the batch size parameter b = 2,...,b max Repeat steps 203F to 203I; Step 203N: Set task parameters n = 2, ..., N, and repeat steps 203C to 203I; Step 2030: Set the traversal variable t, and assign the value N to t; Step 203P: Set up an array called Batch to record the batch size; Step 203Q: Add the dynamic programming batch size belonging to the traversal variable t to the beginning of the array Batch. Step 203R: Update the iterable variable t, and assign t the value... Step 203S, if Then repeat steps 203Q-203R; otherwise, return the array Batch that records the batch size. Step 204: Execute the batch processing task in the window; According to the obtained batch size array Batch, all tasks 1, 2, ..., n, ..., N are packaged and handed over to the edge server for batch processing until the number of tasks exceeds the window size ω. Step 205: Determine whether the task has been completed; Determine whether the server has finished processing all tasks 1, 2, ..., n, ..., N. If it has, then end; otherwise, when the server has finished processing the assigned batch tasks, move the window forward by ω and repeat steps 202-205.

2. The edge server adaptive batch processing method for edge inference tasks according to claim 1, characterized in that: This is used to address deep learning inference tasks in edge scenarios by adjusting the batch size of batch processing on the edge server based on the arrival status of the tasks.

Images