Method, device and equipment for operator scheduling of artificial intelligence model and storage medium

Abstract

The invention relates to an operator scheduling method and device of an artificial intelligence model, equipment and a storage medium. The method comprises the following steps: acquiring a reference running time length of each target operator corresponding to a model, calculating an expected queuing time of each operator queue based on the number of operators in a plurality of operator queues and an expected waiting time length of the operators, and calculating the expected queuing time of each operator queue based on the reference running time length and the expected queuing time of each operator queue. And adding each target operator to the operator queue with the shortest expected queuing time. According to the method, the operators in the operator queues can be balanced to the maximum extent, so that the reasoning load on the reasoning engines is balanced to the maximum extent, the model reasoning delay corresponding to the APPs tends to be balanced, the problem of reasoning delay diffusion caused by scheduling according to the number of the operators in the operator queues is effectively avoided, and the reasoning efficiency is improved. The waiting time of each target operator is reduced, and the AI reasoning efficiency of the whole system is improved.

Description

technical field [0001] The present application relates to the technical field of data processing, and in particular, to an operator scheduling method, apparatus, device and storage medium for an artificial intelligence model. Background technique [0002] At present, most of the strategies adopted for scheduling operators of artificial intelligence models put the target operator in the queue according to the number of existing operators in the operator queue. The disadvantage of this scheduling strategy is that when an operator in the operator queue When the running time of the operator is longer than the running time of other operators in the queue, the operator will cause the other operators behind the operator in the queue to have a longer waiting time, thus increasing all subsequent operators. The reasoning delay is high, and since multiple operators of a deep learning neural network in an APP may be distributed to multiple different queues, there may be dependencies bet...

AI Technical Summary

Problems solved by technology

[0002] At present, most of the strategies adopted for scheduling the operators of the artificial intelligence model put the target operator into the queue according to the number of existing operators in the operator queue. The defect of this scheduling strategy is that when an operator in the operator queue When the running time of an operator is longer than that of other operators in the queue, this operator will cause other operators behind this operator in the queue to have a longer waiting time, thus increasing the number of all subsequent operators. reasoning delay, and because multiple operators of the deep learning neural network in an APP may be distributed to multiple different queues, there may be dependencies between multiple operators in a network. If synchronization is also performed, an operator that takes a long time to complete will not only block operator tasks in this queue, but may also block operator tasks in other queues, resulting in increased inference delays for most networks

Images