A neural network inference task scheduling method for edge intelligent servers

Abstract

The present invention discloses a method for scheduling neural network inference tasks of an edge intelligent server, the steps of which include: using GPU virtualization technology to virtualize a GPU into multiple virtual GPUs; allocating preset resources to the virtual GPU according to a preset allocation strategy, and using a queuing service system to allocate inference tasks to the virtual GPU corresponding to the category of each inference task according to a preset execution batch; collecting the average service delay and computing resource amount of each type of inference task, judging whether the allocation strategy needs to be adjusted, and if so, using a reinforcement learning algorithm to calculate a new allocation strategy; according to the new allocation strategy, allocating corresponding resources to the virtual GPU, and using a queuing service system to allocate neural network inference tasks to the virtual GPU corresponding to the category of each task according to the corresponding execution batch. The present invention meets the real-time requirements of dynamic scenes with low computational complexity, and effectively solves the load balancing problem in larger-scale edge computing scenes.

Description

Technical Field

[0001] The present invention relates to the field of edge computing, and in particular to a method for scheduling neural network inference tasks for an edge intelligent server. Background Art

[0002] Edge computing (EC) is a new cloud computing model. Servers are deployed at the edge of the network to provide computing services to users. The network edge is not a terminal device, but a network location close to the terminal device, which is characterized by being able to communicate with the terminal device with low latency. However, in complex scenarios in real life, an edge server, especially an edge intelligent server, should be responsible for multiple types of highly concurrent neural network inference tasks. The problem this brings is how to perform appropriate task and resource scheduling to increase the speed at which edge servers process these tasks and improve throughput.

[0003] The queuing system, also known as the "queuing service system", is a service system composed of one or more parallel, serial and mixed service stations, serving customers or work objects with different needs, and determining the service order according to given queuing rules. Most of the manufacturing and service systems in reality belong to queuing systems. The service objects can be natural persons, work to be completed, or workpieces to be processed. The batch queuing system is a derivative system of it. The tasks in the batch queuing system are not processed directly, but the tasks in the system are accumulated to a certain number, and these tasks are processed as a batch at the same time. While the processing speed is improved, the resource overhead is also gradually increased. If the batch size is set too large, some tasks will be queued in the system due to resource size limitations, making the queuing delay significantly higher than expected. If it is set too small, the advantages of batch processing cannot be effectively utilized. Summary of the invention

[0004] The technical problem to be solved by the present invention is: In response to the technical problems existing in the prior art, the present invention provides a method for scheduling neural network inference tasks for edge intelligent servers. When the edge intelligent server faces high-concurrency and multi-type neural network inference task scenarios, the method analyzes the characteristics of the inference tasks and allocates corresponding computing resources to them. It also obtains relevant evaluation indicators of the system through modeling and simulation of the dynamic batch queuing service system, and finally uses the reinforcement learning algorithm (D3QN) to determine the system resources and task scheduling scheme to improve the execution speed of the inference tasks, so as to solve the problem of low throughput of multi-type and high-concurrency inference tasks.

[0005] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0006] A method for scheduling neural network inference tasks on an edge intelligent server comprises the following steps:

[0007] S1. Use GPU virtualization technology to virtualize the GPU into multiple virtual GPUs;

[0008] S2, allocating preset resources to the virtual GPU, and using the queuing service system to allocate inference tasks to the virtual GPU corresponding to the category of each inference task according to a preset execution batch;

[0009] S3. Collect the average service delay and the amount of computing resources allocated for each type of inference task, determine whether the allocation strategy needs to be adjusted, and use the reinforcement learning algorithm to calculate a new allocation strategy;

[0010] S4. Allocate resources corresponding to the new allocation strategy to the virtual GPU, and use the queuing service system to allocate neural network inference tasks to the virtual GPU corresponding to each task category according to the execution batch corresponding to the new allocation strategy.

[0011] Furthermore, the specific steps of step S3 include:

[0012] Collect the average service delay E(W) of all inference tasks i ) and the amount of computing resources α allocated by the edge intelligence server to all inference tasks i , as the state value s at this time t , use ∈-greedy method to select the current state value s t The corresponding action A is as follows;

[0013] According to action A, reallocate computing resources and perform task scheduling to obtain the next state s t′ , according to the remaining resources and the inferred task execution status, a reward R is obtained using the reward and punishment mechanism;

[0014] Get the current status t The task execution status and resource allocation status are combined into the current status data s t , get the next state s t′ The task execution status and resource allocation status are combined into the next state data s t′ , the current state data s t , behavior A, next state data s t′ The sum reward R is stored in the priority experience replay pool in the form of array D, and the sampling probability of array D in the priority experience replay pool is calculated by the td-error algorithm;

[0015] According to the sampling probability, the array D in the experience replay pool is extracted to the D3QN network, and the gradient descent error training of the D3QN network is performed, and it is determined whether the termination condition is met. If so, the trained GPU resource allocation and task scheduling model is obtained, and the next step is executed. Otherwise, the next state s t′ As the current state t , execute the ∈-greedy method to select the current state value s t The following steps correspond to action A;

[0016] Import the trained GPU resource allocation and task scheduling model into the edge intelligent server.

[0017] Furthermore, we collect the average service delay E(W i ) and the amount of computing resources α allocated by the edge intelligence server to all inference tasks i Previously included:

[0018] An environment capable of virtualizing GPU and simulating task submission model is constructed on the edge server, and the task submission model is placed in the environment for autonomous scheduling and execution using the D3QN network.

[0019] Furthermore, we collect the average service delay E(W i The step of ) specifically includes: calculating the average service delay of each type of inference task according to the type of arriving inference task and the corresponding average number of tasks, and the expression is:

[0020]

[0021] In the above formula, i is the type number of the inference task, λ i ′ is the effective arrival rate of the i-th inference task, E(L i ) is the average number of tasks of the i-th type of inference task in the queue service system.

[0022] Furthermore, according to action A, the specific steps of reallocating computing resources and performing task scheduling include:

[0023] If there is an average service delay E(W) in the inference task in action A that is greater than the preset threshold i ), then increase the amount of computing resources or the execution batch size, or reduce the amount of computing resources or the execution batch size. If there is no inference task average service delay E(W i ), the amount of computing resources or the execution batch size remains unchanged.

[0024] Furthermore, the preset threshold is the average service delay E(W i ) is the latency of executing the corresponding inference task individually.

[0025] Furthermore, the expression of reward R obtained by using the reward and punishment mechanism is as follows:

[0026]

[0027] In the above formula, i is the type number of the inference task, represents the execution time of the i-th task under the current allocation strategy, T i is the execution time of the i-th task alone, α′ i Indicates the percentage of GPU resources allocated to the i-th task.

[0028] Furthermore, we collect the average service delay E(W i ) and the amount of computing resources α allocated by the edge intelligence server to all inference tasks i Previously, it also included steps to calculate the resource usage of each type of inference task, including:

[0029] According to the type of neural network, the time required to initialize each type of inference task, and the amount of computing resources allocated by the edge intelligent server to each type of inference task, the server rate of each type of inference task is calculated as follows:

[0030]

[0031] In the above formula, i is the type number of the inference task, ε i represents the time required to initialize the i-th inference task, j i is the base of the logarithmic function, which is related to the GFLOPS of the neural network model itself. i Represents the amount of computing resources allocated by the edge intelligent server to the i-th type of inference task.

[0032] The present invention also proposes an edge intelligent server neural network inference task scheduling system, which is programmed or configured to execute any of the edge intelligent server neural network inference task scheduling methods.

[0033] The present invention also proposes a computer-readable storage medium, which stores a computer program programmed or configured to execute any of the edge intelligent server neural network inference task scheduling methods.

[0034] Compared with the prior art, the advantages of the present invention are:

[0035] The present invention adjusts server resources and task scheduling strategies through mathematical modeling quantitative analysis and reinforcement learning algorithms to solve the problem of low throughput of multi-type, high-concurrency inference tasks. First, based on experimental analysis and mathematical modeling methods, the task volume is analyzed and the average service delay of each type of task in the queuing service system is calculated. According to all the average service delays, it is judged whether there is an unreasonable strategy at this time. If so, the strategy adjustment algorithm based on reinforcement learning is started to adjust the strategy, and this process is repeated until the strategy is reasonable. The present invention has low computational complexity, meets the real-time requirements of dynamic scenarios, and effectively solves the load balancing problem in larger-scale edge computing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of a specific embodiment of the present invention.

[0037] Figure 2 It is a schematic diagram of task scheduling by a queuing service system in a specific embodiment of the present invention.

[0038] Figure 3 Schematic diagram of the relationship between the amount of computing resources allocated to the residual network inference task by the edge intelligence server and the inference time.

[0039] Figure 4 Schematic diagram of the relationship between the amount of computing resources allocated to the convolutional neural network inference task by the edge intelligent server and the inference time.

[0040] Figure 5 Schematic diagram of the relationship between the amount of computing resources allocated to the dense convolutional network inference task by the edge intelligent server and the inference time. DETAILED DESCRIPTION

[0041] The present invention is further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the present invention is not limited thereby.

[0042] According to our research, there is a clear trend in the current neural network models: the same task, such as computer vision tasks, uses models with similar structures. Some tasks can even be implemented by the same model. For these tasks, we can process them in a batch to improve the throughput of the neural network inference task. Increasing the batch size of the inference task can significantly improve the processing speed of the neural network inference task. While the processing speed is improved, the resource overhead is also gradually increased. However, once the batch size is set, the problem that follows is that if the batch size is set too large, some tasks will be queued in the system due to video memory limitations, making the queue delay significantly higher than expected. If the batch size is set small, the advantages of batch processing cannot be effectively utilized. According to the above preliminary analysis, it can be clearly concluded that the setting of the inference task batch size will be limited by the configuration of the edge intelligent server device. The batch size will significantly affect the speed at which the queuing service system processes inference tasks. Therefore, there is a boundary for the setting of the batch size. In addition, in real scenarios, the arrival of tasks is random, and multiple inference tasks will also affect each other due to problems such as resource preemption. These factors bring trouble to our quantitative analysis. In order to solve the problem of resource competition among multiple tasks, we use GPU virtualization technology. This technology can virtualize a physical GPU into multiple independent GPUs, thereby reducing the impact of resource contention on the efficiency of the server in performing inference tasks.

[0043] On this basis, on the one hand, we calculate the computational load of different types of tasks and guide the system to allocate resources by modeling the resource-speed model of the task. On the other hand, we model the execution process of various tasks in the system based on the theory of dynamic batch queuing model. The optimization problem is finally guided to our task scheduling.

[0044] Based on the above analysis results and technical ideas, this embodiment proposes a method for scheduling edge intelligent server neural network inference tasks, such as Figure 1 As shown, the following steps are included:

[0045] S1. Use GPU virtualization technology to virtualize the GPU into multiple virtual GPUs;

[0046] S2, allocating preset resources to the virtual GPU according to a preset allocation strategy, and using a queuing service system to allocate inference tasks to the virtual GPU corresponding to the category of each inference task according to a preset execution batch; Figure 2 As shown, for different categories of tasks one, two and three, the queuing service system sequentially allocates each batch of tasks one, two and three to the corresponding virtual GPU;

[0047] S3. Collect the average service delay and the amount of computing resources allocated for each type of inference task, determine whether the allocation strategy needs to be adjusted, and use the reinforcement learning algorithm to calculate a new allocation strategy;

[0048] S4. Allocate corresponding resources to the virtual GPU according to the new allocation strategy, and use the queuing service system to allocate neural network inference tasks to the virtual GPU corresponding to each task category according to the corresponding execution batches.

[0049] Through the above steps, the edge intelligent server neural network inference task scheduling method of this embodiment first uses GPU virtualization technology to divide the GPU into multiple shareable blocks, each block can independently allocate GPU resources, and then calculates the average service delay of the inference task. Finally, the average service delay is used to adjust the allocation strategy including resource allocation and task scheduling by the reinforcement learning algorithm. With lower computational complexity, it meets the real-time requirements of dynamic scenarios and effectively solves the load balancing problem in larger-scale edge computing scenarios.

[0050] Step S3 of this embodiment is used to determine whether the queuing service system has an unreasonable scheduling method. If so, the strategy adjustment algorithm based on reinforcement learning is started to adjust the resource allocation and task scheduling. After the adjustment, if it is determined that the current scenario is still unreasonable in scheduling and allocation, the adjustment will continue until the strategy is reasonable. After the strategy is reasonable, as tasks continue to arrive, unreasonable strategies may still occur after a period of time. At this time, the above behavior needs to be repeated until it is reasonable again. And the system status at this time is recorded for subsequent offline adjustments. The specific steps include:

[0051] Step 1: Build a virtual GPU environment and simulated task submission model on the edge server, use the D3QN network, and place the task submission model in the environment for autonomous scheduling and execution;

[0052] Step 2: Collect the average service delay E(W) of all inference tasks i ) and the amount of computing resources α allocated by the edge intelligence server to all inference tasks i , as the state value s at this time t , use ∈-greedy method to select the current state value s t The corresponding action A is as follows;

[0053] Step 3: According to action A, reallocate computing resources and perform task scheduling to obtain the next state s t′ , according to the remaining resources and the inferred task execution status, a reward R is obtained using the reward and punishment mechanism;

[0054] Step 4: Get the current state s tThe task execution status and resource allocation status are combined into the current status data s t , get the next state s t′ The task execution status and resource allocation status are combined into the next state data s t′ , the current state data s t , behavior A, next state data s t′ The sum reward R is stored in the priority experience replay pool in the form of array D, and the sampling probability of array D in the priority experience replay pool is calculated by the td-error algorithm;

[0055] Step 5: Extract the array D in the experience replay pool to the D3QN network according to the sampling probability, perform gradient descent error training on the D3QN network, and determine whether the termination condition is met. If so, obtain the trained GPU resource allocation and task scheduling model and execute the next step. Otherwise, set the next state s t′ As the current state t , execute the ∈-greedy method to select the current state value s t The following steps correspond to action A;

[0056] Step 6: Import the trained GPU resource allocation and task scheduling model into the edge intelligent server, perform resource allocation and task scheduling in the real environment, and obtain the final allocation and scheduling plan.

[0057] For the second step, the amount of computing resources α allocated to all inference tasks is collected i The process is that the speed at which the server processes the neural network inference task, that is, the service rate, is related to the computing resources allocated by the server to the task and the computing amount of the task itself. In order to quantitatively analyze the relationship between server resource allocation, task computing amount and service rate, we model them. In our analysis, we found that when the computing resources allocated to a certain inference task are increased, the server's processing speed for the task first increases and then tends to stabilize. Moreover, this rule applies to different inference tasks. Therefore, assuming that all the computing resources of the server are 1, the amount of computing resources allocated by the server to the i-th inference task is α i , the relationship between the resource occupancy and service time of this type of task can be expressed by the following formula:

[0058]

[0059] where μ i represents the server rate of the i-th type of inference task. i represents the time required to initialize the i-th type of inference task. i It is the base of the logarithmic function and is related to the GFLOPS of the neural network model itself. irepresents the amount of computing resources allocated by the server to the i-th inference task. We tested the relationship between inference time and resource allocation for some common models, such as Figures 3 to 5 shown.

[0060] For the average service delay E(W) of all inference tasks collected in the second step i ), assuming that the task arrival process is a random process that obeys the Poisson distribution, and the arrival rate is represented by λ. In the process of task arrival at the edge server, the task flow is stable, has no aftereffect, and is ordinary. That is, the arrival of tasks only depends on the length of the time interval, and in non-overlapping time periods, the arrival of tasks is independent of each other. If the time interval is sufficiently small, a user's task will not appear twice. Therefore, the assumption is valid. After completing the server resource scheduling, the server's service time for a certain type of task can be fixed, that is, the service time obeys a fixed-length distribution. The upper and lower limits of the batch size can be represented by a and b respectively. When the number of computing tasks is less than a, the server will wait until the number of tasks arriving is greater than or equal to a, and complete the service as a batch. When the number of tasks is greater than a and less than b, all tasks are completed as a batch. When the number of tasks is more than b, only b tasks will be completed at once. The remaining tasks continue to queue and wait. Assume that X(n,r) and Y(n,r) represent the number of tasks remaining in the queuing service system when the nth batch of tasks is completed and the number of tasks arriving when the nth batch of tasks is served. r is the number of tasks executed in the nth batch. Then X(n+1,r) can be expressed as follows:

[0061]

[0062] From this we can see that X(n+1, r) is only related to X(n, l). It has nothing to do with the value of n. So {X(n, r)}, n∈N 0 ,r∈M a,b ,M a,b ={a,a+1,...,b} is a homogeneous Markov chain. Therefore, the queue service system can be described by M / D(a,b) / 1 / N queues.

[0063] According to relevant theories, the average service delay of a certain type of task in the queuing service system is calculated as follows:

[0064]

[0065] In the above formula, i is the type number of the inference task, λ i ′ is the effective arrival rate of the i-th inference task, E(L i ) is the average number of tasks of the i-th type of inference task in the queue service system.

[0066] Therefore, the service delay of the entire queuing service system is:

[0067] E(W)=[E(W 1 ),E(W 2 ),…,E(W i )]i∈1,2,3…,N (4)

[0068] Where i represents the i-th type of inference task.

[0069] After the server calculates the service delay of the entire system in the current state, if the E(W i ) is large, it means that the resources allocated to this type of task are insufficient or the scheduling method is unreasonable. At this time, it is necessary to adjust the resource allocation and task scheduling plan. Otherwise, it means that the task can perform well under the current system strategy. When the average service delay of a certain type of task in E(W) is significantly greater than the task executed alone, we believe that the system allocation and scheduling strategy is unreasonable at this time. At this time, it is necessary to start the reinforcement learning algorithm (D3QN) to adjust the strategy. The details of the reinforcement learning algorithm include:

[0070] In this embodiment, for the state space (Station Space) of the reinforcement learning algorithm: E(W i ) can be used to describe the allocation of a certain type of task to resource α in the system i , then the set s=[E(W i ),α i ] can be used to describe the situation of the entire system. We use this as the state of the system.

[0071] In this embodiment, for the action space of the reinforcement learning algorithm: after observing the state of the environment, if it is determined that the scheduling method is unreasonable at this time, it is necessary to select the action at this time to solve the problem of poor scheduling. There are two kinds of actions in this embodiment, one is to adjust resource allocation, and the other is to adjust the batch size of task execution. As shown in Table 1.

[0072] Table 1 Scheduling action table

[0073] Maintain 0 0 0 Increase +1 *1.25 *1.5 Decrease -1 *0.75 *0.5

[0074] In Table 1, Maintain means that the resource allocation or batch size remains unchanged, Increase means increasing the resources or batch size, and the increase ranges are fine-tuning (+1), increasing by 25% (*1.25), and increasing by 50% (*1.5), respectively; Decrease means decreasing, and the ranges are the same as above.

[0075] In this embodiment, for the reward value (Reward) of the reinforcement learning algorithm: after observing a certain state and taking an action, Reward is needed to evaluate the quality of the action. We define the reward value of each type of task as R, and the expression is as follows:

[0076]

[0077] In the above formula, i is the type number of the inference task, represents the execution time of the i-th task under the current allocation strategy, T i is the execution time of the i-th task alone, α′ i It represents the percentage of GPU resources allocated to the i-th task. That is, the reward function is determined by the task execution time improvement rate and resource usage. The greater the execution time improvement, the less resource usage, and the server resources are effectively utilized. At this time, the throughput will inevitably increase and the revenue value will be higher.

[0078] Therefore, in the third step, according to action A, the specific steps of reallocating computing resources and performing task scheduling include:

[0079] If there is an average service delay E(W) in the inference task in action A that is greater than the preset threshold i ), then increase the amount of computing resources or the execution batch size, or reduce the amount of computing resources or the execution batch size. If there is no inference task average service delay E(W i ), the amount of computing resources or the execution batch size remains unchanged. Wherein, the preset threshold is the average service delay E(W i ) is the latency of executing the corresponding inference task individually.

[0080] The present invention also proposes an edge intelligent server neural network inference task scheduling system, which is programmed or configured to execute any of the edge intelligent server neural network inference task scheduling methods.

[0081] The present invention also proposes a computer-readable storage medium, which stores a computer program programmed or configured to execute any of the edge intelligent server neural network inference task scheduling methods.

[0082] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A method for scheduling neural network inference tasks on an edge intelligent server, characterized in that: The steps include: S1. Use GPU virtualization technology to virtualize the GPU into multiple virtual GPUs; S2, allocating preset resources to the virtual GPU, and using the queuing service system to allocate inference tasks to the virtual GPU corresponding to the category of each inference task according to a preset execution batch; S3. Collect the average service delay and the amount of computing resources allocated for each type of inference task, determine whether the allocation strategy needs to be adjusted, and use the reinforcement learning algorithm to calculate a new allocation strategy; S4, allocating resources corresponding to the new allocation strategy to the virtual GPU, and using the queuing service system to allocate neural network inference tasks to the virtual GPU corresponding to each task category according to the execution batch corresponding to the new allocation strategy; The specific steps of step S3 include: Collect the average service latency of all inference tasks and the amount of computing resources allocated by the edge intelligence server to all inference tasks , as the state value at this time , select the current state value The corresponding action A is as follows; According to action A, reallocate computing resources and schedule tasks to obtain the next state , according to the remaining resources and the inferred task execution status, a reward R is obtained using the reward and punishment mechanism; Get the current status The task execution status and resource allocation status are combined into the current status data , get the next state The task execution status and resource allocation status are combined into the next state data , the current state data , behavior A, next state data The sum reward R is stored in the priority experience replay pool in the form of array D, and the sampling probability of array D in the priority experience replay pool is calculated; According to the sampling probability, the array D in the experience replay pool is extracted to the D3QN network, and the gradient descent error training of the D3QN network is performed, and it is determined whether the termination condition is met. If so, the trained GPU resource allocation and task scheduling model is obtained, and the next step is executed. Otherwise, the next state As current status , execute and select the current state value The following steps correspond to action A; Import the trained GPU resource allocation and task scheduling model into the edge intelligent server.

2. The method for scheduling edge intelligent server neural network inference tasks according to claim 1 is characterized in that: In step S3, use - Greedy method selects the current state value The corresponding action A is taken below; the sampling probability of array D in the priority experience replay pool is calculated by the td-error algorithm.

3. The method for scheduling edge intelligent server neural network inference tasks according to claim 2 is characterized in that: Collect the average service latency of all inference tasks and the amount of computing resources allocated by the edge intelligence server to all inference tasks Previously included: An environment capable of virtualizing GPU and simulating task submission model is constructed on the edge server, and the task submission model is placed in the environment for autonomous scheduling and execution using the D3QN network.

4. The method for scheduling edge intelligent server neural network inference tasks according to claim 2, characterized in that: Collect the average service latency of all inference tasks The specific steps include: according to the type of arriving inference tasks and the corresponding average number of tasks, calculating the average service delay of each type of inference task, the expression is: In the above formula, i is the type number of the inference task, is the effective arrival rate of the i-th type of inference task, is the average number of tasks of the i-th type of inference task in the queue service system.

5. The method for scheduling edge intelligent server neural network inference tasks according to claim 2, characterized in that: According to action A, the specific steps of reallocating computing resources and scheduling tasks include: If there is an average service delay of the inferred task in action A that is greater than the preset threshold , then increase the amount of computing resources or the execution batch size, or reduce the amount of computing resources or the execution batch size, if there is no inference task average service delay greater than the preset threshold in action A , the amount of computing resources or the execution batch size remains unchanged.

6. The method for scheduling edge intelligent server neural network inference tasks according to claim 5, characterized in that: The preset threshold is the average service delay The latency of the corresponding inference task execution alone.

7. The method for scheduling edge intelligent server neural network inference tasks according to claim 2, characterized in that: The expression of reward R obtained by using the reward and punishment mechanism is as follows: In the above formula, i is the type number of the inference task, Indicates The execution time of the class task under the current allocation strategy, For the The execution time of a class task alone, Indicates the percentage of GPU resources allocated to the i-th task.

8. The method for scheduling edge intelligent server neural network inference tasks according to claim 2, characterized in that: Collect the average service latency of all inference tasks and the amount of computing resources allocated by the edge intelligence server to all inference tasks Previously, it also included steps to calculate the resource usage of each type of inference task, including: According to the type of neural network, the time required to initialize each type of inference task, and the amount of computing resources allocated by the edge intelligent server to each type of inference task, the server rate of each type of inference task is calculated as follows: In the above formula, i is the type number of the inference task, Indicates The time required to initialize the class inference task, It is the base of the logarithmic function and is related to the GFLOPS of the neural network model itself. Indicates that the edge intelligent server gives The amount of computing resources allocated to the class inference task.

9. An edge intelligent server neural network inference task scheduling system, characterized in that: The edge intelligent server neural network inference task scheduling system is programmed or configured to execute the edge intelligent server neural network inference task scheduling method described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that is programmed or configured to execute the edge intelligent server neural network inference task scheduling method described in any one of claims 1 to 8.

Images