A deep learning algorithm model inference scheduling engine system and method

Abstract

The application provides a deep learning algorithm model inference scheduling engine system and method, comprising: a model management module, which realizes unified management of loading, unloading and maintenance of deep learning algorithm models, and the loading mode is based on algorithm logic as a basic unit; an inference calculation module, which sets a priority queue set for the deep learning algorithm model, and the processing logic of the priority queue set is: idle waiting, timeout checking, priority checking and waiting timeout time; the inference nodes are scheduled according to the priority in the priority queue set, the inference nodes are provided with a timeout time and a timestamp, and inference calculation is performed; and a scheduling module, which dynamically manages and schedules the deep learning algorithm model called by the algorithm logic, and asynchronously schedules the inference nodes. The application adopts the priority queue set to realize that different algorithm logics have different priorities, improves the fault tolerance and robustness of the system, sets the timestamp to improve the processing efficiency of the inference nodes, and realizes the rapid response of the algorithm logic.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of model inference request processing, in particular to a deep learning algorithm model inference scheduling engine system and method. BACKGROUND

[0002] Deep learning inference service generally refers to deploying a trained deep learning model to a server and providing a gRPC / HTTP interface request service. The functions implemented internally by the inference service include model loading, model version management, batch processing and multi-path support, and encapsulation of service interfaces. With the progress of science and technology and the continuous improvement of people's living standards, the application of deep learning algorithm models is becoming more and more widespread, such as search engines, recommendation systems, intelligent video surveillance, face recognition, OCR, robots, drones, and self-driving cars. The operation of deep learning algorithm models requires inference calculation of deep learning models. When the number of models is large and the algorithm logic is complex, it is necessary to uniformly manage and schedule the models involved.

[0003] Prior art one, CN202010537231.1 scheduling method, device, equipment and medium of deep learning inference engine, the specific implementation scheme is: in response to the scheduling request for the current inference task from the application layer, determine the type of the current inference task; calculate the total load of each inference engine after executing the current inference task of the type; compare the total load of each inference engine, and determine the target inference engine for executing the current inference task from each inference engine according to the comparison result; return the index of the target inference engine to the application layer, wherein the index represents the calling path of the inference engine. According to the actual computing power of each inference engine, the inference engine is allocated, thereby improving the system performance. And when the inference engine is applied to face recognition, the speed and execution efficiency of face recognition can be improved.

[0004] Prior art two, CN201810685004.6 a deep learning model inference period acceleration method, device and system, by acquiring an optimized deep learning model and data to be processed; the optimized deep learning model has an optimized combined parameter; the optimized deep learning model with the optimized combined parameter is used to process the data to be processed; the data processed is output. By using the optimized deep learning model with the optimized combined parameter to process the data to be processed; the additional calculation overhead in the deep learning model inference period is saved, thereby reducing the inference period calculation time and response delay in the application process of the deep learning model, and reducing the device power consumption.

[0005] Prior art three, CN202111675314.8 A deep learning model inference method, system, device and computer medium, determine the channel quantization parameter of each channel in the target deep learning model; according to the relationship between the channel and the model branch in the target deep learning model, determine the branch quantization parameter of the model branch based on the channel quantization parameter; quantize the target deep learning model based on the branch quantization parameter to obtain the quantized deep learning model; inference based on the quantized deep learning model to obtain the inference result. The channel quantization parameter of each channel in the target deep learning model is determined, the fine-grained quantization parameter determination is realized, the branch quantization parameter of the model branch is determined based on the channel quantization parameter, the inference is performed based on the quantized deep learning model, and the inference result is obtained. The deep learning model inference based on fine-grained channel quantization parameter is realized, and the inference accuracy of the deep learning model is improved.

[0006] Prior art one, prior art two and prior art three put the inference request into the inference queue when receiving the deep learning model inference request, then process the inference request in order, that is, the "first-in first-out" strategy. When the calculation hardware completes the inference once, check the inference queue. If there are multiple inference requests accumulated, batch inference can be used to simultaneously infer multiple requests, which to some extent can simultaneously calculate and save calculation time. The simple first-in first-out queue of the prior art treats different inference requests equally. If the algorithm logic requires rapid feedback, but there are still a large number of inference requests that do not require rapid feedback queued in front of the queue, the algorithm logic that needs rapid feedback will be severely timed out, and the user experience will drop sharply. A certain inference calculation service may serve multiple calculation purposes, such as having multiple input source data, each input source configuring multiple algorithms of different types, and each algorithm calling different models. The traditional scheme does not consider the efficient scheduling of these complex situations.

[0007] The current deep learning model inference calculation is performed by specific hardware, such as a graphics processing unit (GPU) and an embedded neural processing unit (NPU). Although more and more research is focusing on solving the energy consumption problem of deep learning inference calculation, for example, applying lightweight models, applying professional inference software frameworks, optimizing or fusing model internal operators, etc., but involving deep learning model high-throughput inference scenarios, especially involving the complex relationship of multiple algorithms under multiple models, the throughput capacity of deep learning model inference calculation hardware is still a challenge. Therefore, the present application constructs a deep learning algorithm model inference scheduling engine to uniformly schedule and manage inference requests, thereby improving the throughput of unit calculation hardware, reducing energy consumption, and saving costs. SUMMARY

[0008] To solve the above technical problems, the present application provides a deep learning algorithm model inference scheduling engine system, comprising:

[0009] a model management module, which implements unified management of loading, unloading and maintenance of a deep learning algorithm model, and the loading is in the form of an algorithm logic as a basic unit;

[0010] an inference calculation module, which sets a priority queue set for the deep learning algorithm model, and the processing logic of the priority queue set is: idle waiting, timeout checking, priority checking and waiting timeout time; the inference node is scheduled according to the priority in the priority queue set, the inference node is set with a timeout time and a timestamp, and inference calculation is performed;

[0011] a scheduling module, which dynamically manages and schedules the deep learning algorithm model called by the algorithm logic, and asynchronously schedules the inference node.

[0012] Optionally, the model management module comprises:

[0013] a first algorithm logic unit, which loads all deep learning algorithm models capable of completing a single algorithm logic for the single algorithm logic;

[0014] a second algorithm logic unit, which performs commonality checking on models used by all algorithm logics for different algorithm logics, if a model is shared by more than one algorithm logic, the model is used to serve multiple inference calculation purposes, and if the model is used by only a single algorithm logic, the model is only responsible for inferring a request of the single algorithm logic and completing an inference calculation purpose of the single algorithm logic;

[0015] an algorithm logic unloading unit, which, when a certain algorithm logic needs to be unloaded, first detects whether the deep learning algorithm model under the algorithm logic is shared by other algorithm logics, if yes, the deep learning algorithm model cannot be unloaded until a certain deep learning algorithm model is not used by any algorithm logic, and then the deep learning algorithm model can be unloaded to release resources.

[0016] Optionally, the inference calculation module comprises:

[0017] a priority queue setting unit, which sets a priority queue set corresponding to a single type of deep learning algorithm model for the single type of deep learning algorithm model;

[0018] an inference node setting unit, which sets a timeout time and a timestamp matched with a priority for each inference node, and determines whether the inference node is overdue according to the timestamp and the timeout time, and if the inference node is overdue, the inference node is processed preferentially; the inference node setting unit sets a timeout time and a timestamp for the inference node, and if the inference node is determined to be overdue according to the timestamp and the timeout time at a certain time, the inference node is processed preferentially regardless of the queue in which the inference node is located.

[0019] Optionally, the priority queue setting unit contains multiple queues in the priority queue set, different queues have different priorities, and a single queue is processed in order according to the first-in first-out principle, and between different priority queues, the queue with high priority is arranged for processing first, and the queue with low priority is waiting.

[0020] Optionally, the inference node setting unit sets a timeout time and a join timestamp for the inference node, and if it is determined according to the join timestamp and the timeout time that the inference node has timed out at a certain time, the inference node that has timed out is processed most preferentially regardless of the queue of which priority.

[0021] For each inference node, a timeout time matching the priority is set, inference nodes of algorithm logic requiring fast response are placed in a queue with high priority, a shorter timeout time is set to ensure that the inference nodes in the queue are processed as soon as possible, and inference nodes of algorithm logic with low response speed requirement are placed in a queue with low priority, a higher timeout time is set to ensure the minimum throughput requirement.

[0022] Optionally, the scheduling module comprises:

[0023] The algorithm logic model loading and unloading processing unit obtains the loading process of the deep learning algorithm model of the algorithm logic according to the type and quantity requirements of the deep learning algorithm model called by the algorithm logic, and the type and quantity of the deep learning algorithm model existing in the inference service.

[0024] The model scheduling logic unit schedules the deep learning algorithm model called by the algorithm logic.

[0025] The multi-thread asynchronous calling unit does not block and wait after the inference node is added to the priority queue set of the corresponding deep learning algorithm model, but continues to process the scheduling of other algorithm logics, and after the inference node is completed, the deep learning algorithm model thread is called back to notify the scheduling thread, and the scheduling thread returns the final result of the algorithm logic to the requester after receiving the notification.

[0026] The inference scheduling method of the deep learning algorithm model inference scheduling engine system provided by the application comprises the following steps:

[0027] The loading, unloading and maintenance of the deep learning algorithm model are uniformly managed.

[0028] A priority queue set is set for the deep learning algorithm model, and the processing logic of the priority queue set is: idle waiting, timeout check, priority check, and waiting timeout time; the inference nodes are scheduled according to the priorities in the priority queue set, the inference nodes are set with timeout time and time stamp, and inference calculation is performed;

[0029] The deep learning algorithm model is dynamically managed and scheduled according to the algorithm logic call, and the inference nodes are asynchronously scheduled.

[0030] Optionally, the loading of the deep learning algorithm model comprises:

[0031] For a single algorithm logic that needs to be loaded, all deep learning algorithm models capable of completing the algorithm logic are loaded;

[0032] For different algorithm logics, firstly, it is determined whether there are common deep learning algorithm models between different algorithm logics, and the common deep learning algorithm models are shared by multiple algorithm logics;

[0033] The unloading of the deep learning algorithm model comprises:

[0034] When unloading a certain algorithm logic, it is firstly determined whether the deep learning algorithm model under the algorithm logic to be unloaded is shared by other algorithm logics, and if the deep learning algorithm model is shared, the algorithm logic cannot be unloaded, and if the deep learning algorithm model is not shared, the algorithm logic is unloaded.

[0035] Optionally, the processing logic of the priority queue set of the deep learning algorithm model comprises:

[0036] 1) Idle waiting: if there is no inference node in the priority queue set corresponding to the deep learning algorithm model, the deep learning algorithm model enters a waiting state, and a set of inference nodes to be batch processed is set, which is initially empty; until the inference node is added to the priority queue set by the external caller and is notified, the deep learning algorithm model enters a working state;

[0037] 2) Timeout check: firstly, according to the order from high to low priority, the inference nodes in each priority queue are checked in a loop, and the waiting time of the inference node is calculated according to the timeout time and the joining time stamp of the inference node; if the waiting time has exceeded the timeout time of the inference node, the timeout inference node is added to the set of inference nodes to be batch processed; if the number of inference nodes in the set of inference nodes to be batch processed has reached the maximum number of batch processing in a single time during the loop or after the loop, the set of inference nodes to be batch processed is handed over to the deep learning algorithm model for batch inference calculation;

[0038] 3) Priority check: if the number of inference nodes in the set of inference nodes to be batch processed fails to reach the maximum batch processing quantity after the timeout check, the queues are checked in order of priority from high to low to see if there are inference nodes to be processed; if so, the inference nodes to be processed are added to the set of inference nodes to be batch processed; if the number of inference nodes in the set of inference nodes to be batch processed has reached the maximum batch processing quantity during the loop or after one loop, the set of inference nodes to be batch processed is submitted to the deep learning algorithm model for batch inference calculation;

[0039] 4) Wait timeout: if the number of inference nodes in the set of inference nodes to be batch processed still fails to reach the maximum batch processing quantity after the timeout check and the priority check, all inference nodes in the set of inference nodes to be batch processed are checked, and the waiting time of the inference nodes is calculated according to the timeout time and the join timestamp of the inference nodes; if the waiting time exceeds the timeout time of the inference nodes, it means that there are timeout inference nodes in the set of inference nodes to be batch processed, and the set of inference nodes to be batch processed is submitted to the deep learning algorithm model for batch inference calculation; if there are no timeout inference nodes after one loop, the minimum value of the waiting time is taken, and waiting is performed; if the external notification is received during the waiting process, steps 2) and 3) are repeated; if no notification of joining new inference nodes is received after the waiting ends, the set of inference nodes to be batch processed is submitted to the deep learning algorithm model for batch inference calculation;

[0040] 5) After the single inference calculation is completed, it is first checked whether there are inference nodes in the priority queue of the deep learning algorithm model; if so, steps 2), 3), and 4) are repeated; if not, step 1) is entered.

[0041] Optionally, the algorithm logic includes calling one or more models, including the following cases:

[0042] The deep learning algorithm models have no dependency relationship with each other, and concurrent inference calculation scheduling is performed on the deep learning algorithm models included in the algorithm logic;

[0043] The deep learning algorithm models have a dependency relationship with each other, and a deep learning algorithm model needs to wait for the inference of a preceding deep learning algorithm model to be completed before using the inference result for analysis and inference;

[0044] In a more complex algorithm logic, multiple deep learning algorithm models are called, and both the case where the deep learning algorithm models have no dependency relationship with each other and the case where the deep learning algorithm models have a dependency relationship with each other are included, at which time the two cases are processed separately.

[0045] The application improves the throughput rate of unit computing hardware, reduces energy consumption and saves cost by constructing a deep learning algorithm model inference scheduling engine to uniformly schedule and manage inference requests.

[0046] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art upon examination of the following or can be learned by practice of the present application. The objects and other advantages of the present application can be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

[0047] The technical solutions of the present application will be further described in detail below with the aid of the accompanying drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0048] The accompanying drawings, which are included to provide a further understanding of the present application and constitute a part of this specification, illustrate embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0049] Figure 1 The structure schematic diagram of the deep learning algorithm model inference scheduling engine system in the embodiment of the present application;

[0050] Figure 2 The structure schematic diagram of the model management module in the embodiment of the present application;

[0051] Figure 3 The structure schematic diagram of the inference calculation module in the embodiment of the present application;

[0052] Figure 4 The structure schematic diagram of the scheduling module in the embodiment of the present application;

[0053] Figure 5 The flow chart of the inference scheduling method of the deep learning algorithm model inference scheduling engine system in the embodiment of the present application;

[0054] Figure 6 The relationship schematic diagram between the algorithm logic and the model in the embodiment of the present application;

[0055] Figure 7 Figure 1 is a schematic diagram of the priority queue set processing logic in an embodiment of the present application;

[0056] Figure 8 Figure 2 is a schematic diagram of the model scheduling logic in the algorithm logic in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The preferred embodiments of the present application will be described herein below with reference to the accompanying drawings, in which it is understood that the preferred embodiments described herein are merely for the purpose of illustrating and explaining the present application and are not intended to limit the present application.

[0058] The terminology used in the present application is merely for the purpose of describing particular embodiments and is not intended to limit the present application. As used in the present application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or," as used herein, refer to and encompass any or all possible combinations of one or more of the associated listed items.

[0059] The following description refers to the accompanying drawings. Wherever possible, the same reference numbers in different drawings refer to the same or similar elements. The following exemplary embodiments are described herein below with reference to the drawings, in which the implementations are not limited to the specific embodiments described herein. Instead, as would be apparent to one of ordinary skill in the art, many modifications are available that achieve the same result and incorporate the same basic functions. The following exemplary embodiments are described with reference to the accompanying drawings, wherein:

[0060] Embodiment 1

[0061] As shown in FIG. 1, an embodiment of the present application provides a deep learning algorithm model inference scheduling engine system, which comprises: Figure 1 A model management module, which implements unified management of loading, unloading, and maintenance of deep learning algorithm models, and the loading is in the form of algorithm logic as a basic unit;

[0062] An inference calculation module, which sets a priority queue set for the deep learning algorithm model, and the processing logic of the priority queue set is: idle waiting, timeout checking, priority checking, and waiting timeout time; the inference node is scheduled according to the priority in the priority queue set, the inference node is set with a timeout time and a timestamp, and inference calculation is performed;

[0063]

[0064] ​The scheduling module dynamically manages and schedules the deep learning algorithm model called by the algorithm logic, and asynchronously schedules the inference node.

[0065] The working principle and beneficial effects of the technical solution are as follows: the application constructs an inference scheduling engine of a deep learning algorithm model, uniformly schedules and manages inference requests, thereby improving the throughput of unit computing hardware, reducing energy consumption, and saving costs. The application uniformly manages different deep learning algorithm models, not only saves hardware resources, but also reduces the hardware running burden and improves the processing efficiency of the hardware; different priorities are adopted for inference requests of different deep learning algorithm logics, and a priority queue set is used to realize that different algorithm logics have different priorities, which is flexible in operation and improves the fault tolerance and robustness of the system; a timestamp and an overtime are set for the inference node, and inference calculation is performed in the order of priority and overtime, which improves the processing efficiency of the inference node and realizes fast response of the algorithm logic; the deep learning algorithm model called by the algorithm logic is dynamically managed and scheduled, and the inference node is asynchronously scheduled, which ensures the response speed of the algorithm logic, saves hardware resources, and improves the intelligent control level of the system.

[0066] Embodiment 2

[0067] As shown in Figure 2 On the basis of embodiment 1, the model management module provided by the application loads all deep learning algorithm models of a single algorithm logic according to a preset inference calculation purpose of the deep learning algorithm model.

[0068] The first algorithm logic unit loads all deep learning algorithm models capable of completing the algorithm logic for a single algorithm logic.

[0069] The second algorithm logic unit checks the commonality of models used by all algorithm logics for algorithm logics of different categories. If a model is shared by more than one algorithm logic for multiple inference calculation purposes, the model is only responsible for inferring the request of a single algorithm logic and completing the inference calculation purpose of the single algorithm logic if the model is used by only the single algorithm logic.

[0070] The algorithm logic unloading unit checks whether the deep learning algorithm model under the algorithm logic is shared by other algorithm logics when the algorithm logic needs to be unloaded. If so, the deep learning algorithm model cannot be unloaded until the deep learning algorithm model is not used by any algorithm logic, and the resource is released.

[0071] The working principle and beneficial effects of the above technical solution are as follows: This invention provides unified management of deep learning algorithm models with different algorithm logics. When different algorithm logics share a deep learning algorithm model, the deep learning algorithm model is shared, saving hardware resources. When an algorithm logic is unloaded, if a model within that algorithm logic is shared by other algorithm logics, it checks whether the deep learning algorithm model needs to be unloaded to avoid errors in the algorithm logic that requires the deep learning algorithm model, thereby improving the utilization rate of hardware resources. At the same time, it also ensures the normal operation of other algorithm logics and ensures the smooth inference scheduling of deep learning algorithm models.

[0072] Example 3

[0073] like Figure 3 As shown, based on Embodiment 1, the inference calculation module provided in this embodiment of the invention includes:

[0074] The priority queue setting unit sets up a priority queue set corresponding to a single type of deep learning algorithm model. The priority queue set contains multiple queues with different priorities. Within a single queue, processing is performed sequentially according to the first-in-first-out principle. Between different priority queues, the queue with the higher priority is processed first, while the queue with the lower priority waits.

[0075] The inference node setting unit sets a timeout time and timestamp that match the priority for each inference node. It determines whether an inference node has timed out based on the timestamp of its addition and the timeout time. If it has timed out, it is processed first. The inference node setting unit sets a timeout time and an addition timestamp for each inference node. If at a certain moment, it is determined that an inference node has timed out based on the timestamp of its addition and the timeout time, then regardless of which priority queue it is in, the inference node that has timed out is processed with the highest priority.

[0076] For each inference node, set a timeout period that matches its priority. Inference nodes with algorithmic logic requiring fast response are placed in a high-priority queue with a shorter timeout period to ensure that the inference nodes in this queue can be computed and processed as quickly as possible. Examples include real-time recommendations based on user input and real-time video traffic statistics. The goal is to minimize response latency and improve the user's real-time feedback experience. Therefore, they can be placed in a high-priority queue with a shorter timeout period to ensure that the inference nodes in this queue can be computed and processed as quickly as possible.

[0077] For algorithm logic with low response speed requirement, for example, non-real-time offline text security review, offline picture or video super-resolution, frame-by-frame personnel intrusion detection, etc., the characteristics of such algorithm logic are that only low continuous request frequency needs to be guaranteed, or there is no requirement for the correlation between front and rear requests, and users also have no real-time feedback experience requirement, therefore, such algorithm logic can be put into a lower priority queue, a higher timeout time is set, and the minimum throughput requirement is guaranteed.

[0078] The working principle and beneficial effects of the technical solution are as follows: the priority queue setting unit of the application adopts a priority queue set with different priority queues, the inference nodes generated by algorithm logic with different inference calculation purposes are put into different priority queues for different treatment, the inference node setting unit sets the timeout time and the time stamp matched with the priority for the inference nodes, and it is ensured that the inference nodes with different inference calculation purposes can be completed on time and in quantity. Meanwhile, the number of priority levels is not limited, different priorities of different algorithm logics can be set according to different inference calculation purposes, and the flexibility, fault tolerance and robustness of the system program are improved.

[0079] Embodiment 4

[0080] As shown in Figure 4 On the basis of embodiment 1, the scheduling module provided by the application embodiment comprises:

[0081] The algorithm logic model loading and unloading processing unit, when the algorithm logic is loaded into the inference service request, according to the preset deep learning algorithm model calling type and quantity requirement of the algorithm logic, in combination with the existing deep learning algorithm model type and quantity in the inference service, the loading process of the deep learning algorithm model of the algorithm logic is obtained; for example, algorithm logic 1 needs to call model A and model B, and it is found that model A has been loaded by another algorithm logic in the current inference service, therefore, the algorithm logic 1 only needs to load a model B.

[0082] For each inference node, a join timestamp of its joining the priority queue set is recorded, and a completion timestamp of the inference node being completed by the inference calculation is also recorded, which is referred to as a completion timestamp. With the passage of time of the inference service, the time stamp difference between the completion timestamp and the join timestamp of all inference nodes is constantly monitored and averaged to obtain an average response time. If the average response time continuously exceeds a preset maximum average response time threshold in a period of time, it indicates that too many inference nodes in the priority queue set of the deep learning algorithm model of the type are accumulated at this time, and more deep learning algorithm models of the type are needed to process the inference nodes in the priority queue set at the same time. At this time, a deep learning algorithm model of the type can be dynamically copied and loaded to improve the user experience. If the average response time continuously falls below a preset minimum average response time threshold in a period of time, it indicates that there is little or no accumulation of inference nodes in the priority queue set of the deep learning algorithm model at this time, and a deep learning algorithm model of the type can be unloaded on demand under the premise of ensuring the correctness of the algorithm logic, thereby reducing the hardware resource occupancy rate.

[0083] The model scheduling logic unit schedules the deep learning algorithm models called by the algorithm logic, including:

[0084] The deep learning algorithm models have no dependency relationship with each other, that is, any deep learning algorithm model can be called first; at this time, concurrent inference calculation scheduling can be performed on all contained deep learning algorithm models;

[0085] The deep learning algorithm models have a dependency relationship with each other, that is, a deep learning algorithm model needs to wait for the inference of a preceding deep learning algorithm model to be completed, and then use the inference result for analysis and inference;

[0086] In a more complex algorithm logic, multiple deep learning algorithm model calls can be included, and both of the above two cases can be included. However, the above two cases can exist at the same time and do not contradict or conflict with each other.

[0087] The multi-thread asynchronous calling unit does not block and wait after scheduling the thread of the scheduling module after the inference node is added to the priority queue set of the corresponding deep learning algorithm model, but continues to process the scheduling of other algorithm logic. When the inference node is completed, the deep learning algorithm model thread is called back to notify the scheduling thread, and the scheduling thread returns the final result of the algorithm logic to the requester after receiving the notification.

[0088] The working principle and beneficial effects of the technical solution are as follows: the algorithm logic model loading and unloading processing unit of the application dynamically increases or reduces the model under the premise of ensuring the correctness of the algorithm logic, realizes intelligent increase and decrease of the number of models, ensures the response time, and avoids resource waste. The model scheduling logic unit of the application reasonably disassembles and sorts the deep learning algorithm model scheduling logic through deep learning of the algorithm logic, performs on-demand concurrent calculation or waits for inference calculation, instead of all deep learning algorithm models being treated equally and sequentially inferred, the processing speed of the entire algorithm logic is accelerated as much as possible under the premise of ensuring the correctness of the algorithm logic. Each deep learning algorithm model in the inference service is maintained by a separate thread, which ensures that different deep learning algorithm models can be concurrently calculated, and improves the utilization of hardware. At the same time, the algorithm logic is asynchronously called, which fully improves the utilization of hardware resources and improves the overall throughput of the inference service.

[0089] Embodiment 5

[0090] As shown in Figure 5 the inference scheduling method of the deep learning algorithm model inference scheduling engine system provided by the embodiment of the application based on the embodiment 1 comprises the following steps:

[0091] S501: implement unified management of loading, unloading and maintenance of deep learning algorithm models;

[0092] S502: set a priority queue set for the deep learning algorithm model, the processing logic of the priority queue set is: idle waiting, timeout checking, priority checking and waiting timeout time; schedule the inference node according to the priority in the priority queue set, set the timeout time and time stamp of the inference node, and perform inference calculation;

[0093] S503: dynamically manage and schedule the deep learning algorithm model called by the algorithm logic, and asynchronously schedule the inference node.

[0094] The working principle and beneficial effects of the above technical solution are as follows: This invention provides unified management and scheduling for different deep learning algorithm models, which not only saves hardware resources but also reduces the hardware operating burden and improves hardware processing efficiency; it sets timeout times and timestamps for inference nodes, and for inference nodes that time out, inference calculations are performed in the order of the timeout times, improving the processing efficiency of inference nodes and realizing rapid response of algorithm logic; it dynamically manages and schedules the deep learning algorithm models called by the algorithm logic, and asynchronously schedules inference nodes, ensuring the response speed of the algorithm logic and saving hardware resources. The inference node setting unit of this invention sets idle waiting to avoid frequent checks of the priority queue occupying CPU resources, saving hardware resources; it adopts a combination of timeout checks and priority checks to maximize the inference request throughput per unit of hardware while ensuring the algorithm logic response speed requirements; it sets waiting timeout times, taking the minimum waiting time of the batch-processing inference nodes as the waiting time, and waiting for as many inference nodes as possible to join while ensuring the algorithm logic response speed requirements are met, thereby improving throughput and reducing the overall number of inference calculations and energy consumption.

[0095] Example 6

[0096] like Figure 6 As shown, based on Example 5, step S501 provided in this embodiment of the invention specifically includes:

[0097] For a single algorithm logic that needs to be loaded, load all deep learning algorithm models that can complete the algorithm logic;

[0098] For different algorithm logics, first determine whether there is a shared deep learning algorithm model among the different algorithm logics. If there is a shared deep learning algorithm model, then it is shared by multiple algorithm logics.

[0099] When uninstalling a certain algorithm logic, it is necessary to first determine whether the deep learning algorithm model called by the algorithm logic being uninstalled is shared by other algorithm logics. If they are shared, the algorithm cannot be uninstalled; otherwise, it can be uninstalled.

[0100] The working principle and beneficial effects of the above technical solution are as follows: According to the preset reasoning calculation purpose, the inference service loads the required deep learning algorithm models on a unit basis (algorithm logic). Simultaneously, it checks the commonality of deep learning algorithm models; when commonality exists, redundant loading is avoided, saving hardware resources. When unloading algorithms, a commonality check is performed on the deep learning algorithm models; if commonality exists, they are not unloaded, preventing errors in other algorithm logic and improving program fault tolerance. This achieves unified management of multiple deep learning algorithm models with different algorithm logics.

[0101] Example 7

[0102] likeFigure 7 As shown, the step S502 provided by the embodiment of the present application specifically includes:

[0103] The inference calculation of the deep learning algorithm model has a limit of once batch processing of maximum inference node quantity, referred to as single maximum batch inference quantity. For the priority queue set of a deep learning algorithm model, the processing logic is as follows:

[0104] 1) Idle waiting: if there is no inference node in the priority queue set corresponding to the deep learning algorithm model, the deep learning algorithm model enters a waiting state, and a set of inference nodes to be batch processed is set, which is initially empty; until the inference node is added to the priority queue set by the external caller and notified, the deep learning algorithm model enters a working state;

[0105] 2) Timeout check: first, according to the order of priority from high to low, the inference nodes in each priority queue are checked in a loop, and the waiting time of the inference node is calculated according to the timeout time and the join time stamp of the inference node. If the waiting time has exceeded the timeout time of the inference node, the timeout inference node is added to the set of inference nodes to be batch processed; if the number of inference nodes in the set of inference nodes to be batch processed has reached the single maximum batch processing quantity during the loop or after the loop, the set of inference nodes to be batch processed is handed over to the deep learning algorithm model for batch inference calculation;

[0106] 3) Priority check: if the number of inference nodes in the set of inference nodes to be batch processed has not reached the single maximum batch processing quantity after the timeout check is completed, the priority queue is checked in a loop according to the order of priority from high to low to check whether there is an inference node to be processed; if yes, the inference node to be processed is added to the set of inference nodes to be batch processed; if the number of inference nodes in the set of inference nodes to be batch processed has reached the single maximum batch processing quantity during the loop or after the loop, the set of inference nodes to be batch processed is handed over to the deep learning algorithm model for batch inference calculation;

[0107] 4) waiting timeout: if the number of inference nodes in the batch processing inference node set fails to reach the single maximum batch processing quantity after the timeout check and the priority check, all inference nodes in the batch processing inference node set are checked, the waiting time of the inference node is calculated according to the timeout time and the join timestamp of the inference node, if the waiting time exceeds the timeout time of the inference node, it is indicated that there is a timeout inference node in the batch processing inference node set, and the batch processing inference node set is handed over to the deep learning algorithm model for batch inference calculation; if there is no timeout inference node after one cycle, the minimum value of the waiting time is taken, and waiting is performed; if an external notification is received during the waiting process, steps 2) and 3) are repeated, if no notification of joining a new inference node is received after the waiting ends, the batch processing inference node set is handed over to the deep learning algorithm model for batch inference calculation;

[0108] 5) after the single inference calculation ends, it is first checked whether there is an inference node in the priority queue of the deep learning algorithm model, if yes, steps 2), 3) and 4) are repeated, if no, step 1) is entered.

[0109] The working principle and beneficial effects of the technical solution are as follows: the idle waiting is set to avoid frequent checking of the priority queue to occupy hardware resources, and hardware resources are saved for real inference calculation. The timeout check and the priority check are combined to improve the inference request throughput rate of the unit hardware as much as possible on the premise of meeting the response speed requirement; the waiting timeout is set, the minimum waiting time in the batch processing inference node set is taken for waiting, more inference nodes are waited as much as possible on the premise of meeting the response speed requirement, the throughput rate is improved, and the total inference calculation times and energy consumption are reduced.

[0110] Embodiment 8

[0111] As shown in the embodiment 6, the step S503 provided by the embodiment of the application specifically includes: Figure 8 The deep learning algorithm model called by the algorithm logic is scheduled, including the following cases:

[0112] The deep learning algorithm models have no dependency relationship with each other, and the deep learning algorithm model called by the algorithm logic is scheduled for concurrent inference calculation;

[0113]

[0114] ​The deep learning algorithm models have a dependency relationship, and a deep learning algorithm model needs to wait for the inference of a preceding deep learning algorithm model to be completed, utilize the inference result, and then analyze and infer. For example, algorithm logic 1 needs to call deep learning algorithm model A and deep learning algorithm model B, and deep learning algorithm model B depends on the result of deep learning algorithm model A for inference calculation. At this time, the dependency relationship needs to be checked, the inference node of deep learning algorithm model A is constructed first, added to the priority queue set of deep learning algorithm model A, after the inference result of deep learning algorithm model A is obtained, the inference node of deep learning algorithm model B is constructed, combined with the inference result of deep learning algorithm model A, added to the priority queue set of deep learning algorithm model B, and after the inference of deep learning algorithm model B is completed, the final result can be returned.

[0115] In more complex algorithm logic, multiple deep learning algorithm models are called, and both of the above two cases are included. At this time, the two cases are processed respectively according to the cases without a dependency relationship and the cases with a dependency relationship.

[0116] The working principle and beneficial effects of the above technical solution are as follows: the model scheduling logic of the algorithm logic is reasonably disassembled and sorted, and the calculation is performed concurrently or waited for, instead of all models being treated equally and sequentially calculated. On the premise of ensuring the correctness of the algorithm logic, the processing speed of the entire algorithm logic is accelerated as much as possible.

[0117] Embodiment 9

[0118] On the basis of embodiment 7, the timeout check provided by the embodiment of the application can adjust the set timeout time according to the number of inference nodes in the priority queue, so as to increase the processing efficiency of the inference nodes. The adjustment model of the timeout time is as follows:

[0119]

[0120] wherein t ori represents an initial timeout time, t mod is an adjusted timeout time, the capacity of the priority queue is represented as N max = N c +N e , N c is the number of inference nodes in the actual priority queue at time T, N e is the empty capacity of the priority queue at time T, represents the number of inference nodes entering the priority queue at the next time, when , the timeout time does not need to be adjusted, and when , it is indicated that the priority queue is about to be filled, and the initial timeout time needs to be adjusted by a dynamic parameter p.

[0121] The working principle and beneficial effects of the technical solution are as follows: when the reasoning node is added to the priority queue, the remaining capacity of the priority queue is considered, and then the timeout time is dynamically adjusted, the processing efficiency of the reasoning node in the timeout check is improved, and the accumulation of the reasoning node is avoided.

[0122] Embodiment 10

[0123] On the basis of Embodiment 7, the design method of the priority queue in the priority queue set provided by the embodiment of the application comprises:

[0124] obtaining a priority level sequence to be placed in the priority queue set of the deep learning algorithm model;

[0125] constructing a priority queue by using the obtained priority level sequence, each priority level in the priority level sequence corresponding to a priority level of a reasoning node in the constructed priority queue;

[0126] setting a corresponding priority level index for the reasoning node in the constructed priority queue, and mapping the set priority level index to a priority level linked list, each reasoning node mapping a priority level linked list;

[0127] selecting reasoning nodes with the same priority level from the reasoning nodes to be placed in the priority queue, and placing the reasoning nodes in the priority level linked list corresponding to the priority level, so that the priority level linked list stores all reasoning nodes with the priority level.

[0128] The working principle and beneficial effects of the technical solution are as follows: for a single type of deep learning algorithm model, a priority queue set corresponding to the single type of deep learning algorithm model is set; the priority queue set contains multiple queues, different queues have different priorities, and a single queue is processed in the order of first-in first-out principle, and between different priority queues, the queue with high priority is arranged for processing first, and the queue with low priority is waiting, the above scheme improves the priority processing efficiency of the deep learning algorithm model, reduces the load of hardware running, and provides a basis for scheduling of the reasoning node.

[0129] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A deep learning algorithm model inference scheduling engine system, characterized in that, include: The model management module enables unified management of loading, unloading, and maintenance of deep learning algorithm models, with the loading method based on algorithm logic as the basic unit. The inference computation module sets up a priority queue set for the deep learning algorithm model. The processing logic of the priority queue set is: idle waiting, timeout check, priority check, and waiting timeout time. The inference nodes are scheduled according to the priority in the priority queue set, and the inference nodes are set with timeout time and timestamp to perform inference computation. Among them, idle waiting: if there are no inference nodes in the priority queue set corresponding to the deep learning algorithm model, the deep learning algorithm model enters the waiting state and sets the inference node set to be processed in batches, which is initially empty; until the external caller adds the inference node to the priority queue set and notifies it, the deep learning algorithm model enters the working state. Timeout Check: First, check the inference nodes in each priority queue in descending order of priority. Calculate the waiting time of the inference node based on its timeout time and addition timestamp. If the waiting time exceeds the timeout time, add the timed-out inference node to the batch processing inference node set. If, during the loop or after one loop, the number of inference nodes in the batch processing inference node set has reached the maximum batch processing quantity, then hand the batch processing inference node set over to the deep learning algorithm model for batch inference computation. Priority check: If, after the timeout check, the number of inference nodes in the set of inference nodes to be processed in batches has not reached the maximum batch processing quantity in a single run, then the queues are checked in descending order of priority to see if there are any inference nodes to be processed. If so, the inference nodes to be processed are added to the set of inference nodes to be processed in batches. If, during the loop or after one loop, the number of inference nodes in the set of inference nodes to be processed in batches has reached the maximum batch processing quantity in a single run, then the set of inference nodes to be processed in batches is handed over to the deep learning algorithm model for batch inference computation. Waiting timeout: If, after timeout and priority checks, the number of inference nodes in the set of inference nodes to be processed still fails to reach the maximum batch processing quantity, then all inference nodes in the set of inference nodes to be processed are checked. Based on the timeout time and addition timestamp of the inference node, the waiting time of the inference node is calculated. If the waiting time exceeds the timeout time of the inference node, it means that there are timeout inference nodes in the set of inference nodes to be processed. The set of inference nodes to be processed is then handed over to the deep learning algorithm model for batch inference calculation. If no timeout inference nodes are found after one loop, the minimum waiting time is taken and the process continues. If an external notification is received during the waiting process, the timeout check and priority check are repeated. If no notification of adding a new inference node is received after the waiting period ends, the set of inference nodes to be processed is handed over to the deep learning algorithm model for batch inference calculation. The scheduling module dynamically manages and schedules the deep learning algorithm models called by the algorithm logic, and asynchronously schedules inference nodes.

2. The deep learning algorithm model inference scheduling engine system as described in claim 1, characterized in that, The model management module includes: The first algorithm logic unit, for a single algorithm logic, loads all deep learning algorithm models capable of completing the algorithm logic; The second algorithm logic unit performs a commonality check on the models used by all algorithm logics for different categories of algorithm logics. If a model is shared by more than one algorithm logic, then the model is used to serve multiple inference and computation purposes; if the model is used by only a single algorithm logic, then the model is only responsible for inferring the request of the single algorithm logic and completing the inference and computation purpose of the single algorithm logic. The algorithm logic unloading unit checks whether the deep learning algorithm model under the algorithm logic is shared by other algorithm logic when it needs to unload a certain algorithm logic. If so, the deep learning algorithm model cannot be unloaded until no algorithm logic uses it, and then it can be unloaded to release resources.

3. The deep learning algorithm model inference scheduling engine system as described in claim 1, characterized in that, The inference computation module includes: The priority queue setting unit sets a priority queue set corresponding to a single type of deep learning algorithm model. The inference node setting unit sets a timeout time and timestamp that match the priority for each inference node. It determines whether an inference node has timed out based on the timestamp of its addition and the timeout time. If it has timed out, it is processed first. The inference node setting unit sets a timeout time and an addition timestamp for each inference node. If at a certain moment, it is determined that an inference node has timed out based on the timestamp of its addition and the timeout time, then regardless of which priority queue it is in, the inference node that has timed out is processed with the highest priority.

4. The deep learning algorithm model inference scheduling engine system as described in claim 3, characterized in that, The priority queue setting unit contains multiple queues, each with different priorities. Within a single queue, processing is performed sequentially according to the first-in, first-out principle. Between queues with different priorities, the queue with the higher priority is processed first, while the queue with the lower priority waits.

5. The deep learning algorithm model inference scheduling engine system as described in claim 3, characterized in that, In the inference node setting unit, for each inference node, a timeout time matching its priority is set. Inference nodes with algorithm logic requiring fast response are placed in a high-priority queue with a shorter timeout time to ensure that the inference nodes in this queue can be computed and processed as soon as possible. Inference nodes with algorithm logic that does not require high response speed are placed in a low-priority queue with a higher timeout time to ensure the minimum throughput requirement.

6. The deep learning algorithm model inference scheduling engine system as described in claim 1, characterized in that, The scheduling module includes: The algorithm logic model loading / unloading unit, when the algorithm logic is loaded into the inference service request, determines the loading process of the deep learning algorithm model based on the preset types and quantities of deep learning algorithm models required by the algorithm logic, combined with the types and quantities of deep learning algorithm models currently existing in the inference service. Simultaneously, it dynamically manages the loading and unloading of deep learning algorithm models based on the backlog of inference nodes in the priority queue. The model scheduling logic unit schedules the deep learning algorithm models called by the algorithm logic. After the multi-threaded asynchronous call unit adds the inference node to the priority queue of the corresponding deep learning algorithm model, the scheduling thread of the scheduling module does not block and wait, but continues to process the scheduling of other algorithm logic. After the inference node completes the calculation, the deep learning algorithm model thread sends a callback notification. After receiving the notification, the scheduling thread returns the final result of the algorithm logic to the requester.

7. An inference scheduling method for a deep learning algorithm model inference scheduling engine system, characterized in that, Includes the following steps: Implement unified management of loading, unloading, and maintenance of deep learning algorithm models; Set up a priority queue set for the deep learning algorithm model. The processing logic of the priority queue set is: idle waiting, timeout check, priority check, and waiting timeout. Inference nodes are scheduled according to their priority in the priority queue set. Inference nodes are set with timeout and timestamps and then perform inference calculations. The processing logic for the priority queue set of deep learning algorithm models in the deep learning algorithm model inference scheduling engine system includes: 1) Idle waiting: If there are no inference nodes in the priority queue set corresponding to the deep learning algorithm model, the deep learning algorithm model enters the waiting state and sets the set of inference nodes to be processed in batches, which is initially empty; until the external caller adds the inference nodes to the priority queue set and notifies it, the deep learning algorithm model enters the working state. 2) Timeout Check: First, check the inference nodes in each priority queue in descending order of priority. Calculate the waiting time of the inference node based on its timeout time and addition timestamp. If the waiting time exceeds the timeout time, add the timed-out inference node to the batch processing inference node set. If, during the loop or after one loop, the number of inference nodes in the batch processing inference node set has reached the maximum batch processing quantity, then hand the batch processing inference node set over to the deep learning algorithm model for batch inference computation. 3) Priority check: If the number of inference nodes in the set of inference nodes to be processed in batches fails to reach the maximum batch processing quantity in a single run after the timeout check, the queues are checked in descending order of priority to see if there are any inference nodes to be processed. If so, the inference nodes to be processed are added to the set of inference nodes to be processed in batches. If the number of inference nodes in the set of inference nodes to be processed in batches has reached the maximum batch processing quantity in a single run during the loop or after one loop, the set of inference nodes to be processed in batches is handed over to the deep learning algorithm model for batch inference computation. 4) Waiting timeout: If, after timeout and priority checks, the number of inference nodes in the set of inference nodes to be processed still fails to reach the maximum batch processing quantity, then all inference nodes in the set of inference nodes to be processed are checked. Based on the timeout time and addition timestamp of the inference node, the waiting time of the inference node is calculated. If the waiting time exceeds the timeout time of the inference node, it means that there are timeout inference nodes in the set of inference nodes to be processed. The set of inference nodes to be processed is then handed over to the deep learning algorithm model for batch inference calculation. If no timeout inference nodes are found after one loop, the minimum waiting time is taken and the process continues. If an external notification is received during the waiting process, steps 2) and 3) are repeated. If no notification of adding a new inference node is received after the waiting period ends, the set of inference nodes to be processed is handed over to the deep learning algorithm model for batch inference calculation. 5) After a single inference calculation is completed, first check if there is an inference node in the priority queue set of the deep learning algorithm model. If there is, repeat 2), 3), and 4); if not, proceed to 1). The deep learning algorithm model that calls the algorithm logic is dynamically managed and scheduled, and inference nodes are scheduled asynchronously.

8. The inference scheduling method of the deep learning algorithm model inference scheduling engine system as described in claim 7, characterized in that, Loading deep learning algorithm models includes: For a single algorithm logic that needs to be loaded, load all deep learning algorithm models that can complete the algorithm logic; For different algorithm logics, first determine whether there is a shared deep learning algorithm model among the different algorithm logics. If there is a shared deep learning algorithm model, then it is shared by multiple algorithm logics. Unloading deep learning algorithm models includes: When uninstalling a certain algorithm logic, it is necessary to first determine whether the deep learning algorithm model under the algorithm logic is shared by other algorithm logics. If they are shared, the algorithm cannot be uninstalled; otherwise, it can be uninstalled.

9. The inference scheduling method of the deep learning algorithm model inference scheduling engine system as described in claim 7, characterized in that, Scheduling deep learning algorithm models that call algorithmic logic includes the following cases: The deep learning algorithm models have no dependencies on each other, and concurrent inference computation scheduling is performed on the deep learning algorithm models contained in the algorithm logic; Deep learning algorithm models are interdependent. A certain deep learning algorithm model needs to wait for the previous deep learning algorithm model to complete its inference before it can use the inference results to perform analysis and inference. In more complex algorithmic logic, there are calls to multiple deep learning algorithm models, and there are two cases: those deep learning algorithm models have no dependencies on each other and those deep learning algorithm models have dependencies on each other. In this case, they are handled separately according to whether they have no dependencies or have dependencies.

Images