Method and device for qos-aware resource allocation in deep learning multi-model deployment scenarios

Abstract

The application relates to a QoS-aware resource allocation method and device in a deep learning multi-model deployment scenario. The method comprises the following steps: a deep learning model is split into multiple serially dependent sub-models, and corresponding target tasks are split into multiple sub-tasks; according to the overall response ratio of all sub-tasks in the global task queue when the queue changes, the sub-tasks are inserted into the global task queue; when the sub-tasks are to be operated, tokens are issued to the sub-tasks according to the number of current tasks of each type and the attributes of the sub-tasks, so that the running results of the target tasks are obtained based on the deep learning model or the multiple serially dependent sub-models. The application can effectively solve the problem that a short task has to wait for a long time due to a long task, can realize adjustment of a resource allocation strategy of a task with extremely low scheduling overhead, and can reduce the jitter of a task service level.

Description

Technical Field

[0001] This invention belongs to the field of software technology, specifically relating to a QoS (Quality of Service)-aware resource allocation method and apparatus in a deep learning multi-model deployment scenario. Background Technology

[0002] Deep learning, as a crucial technology in fields such as lane detection, pedestrian recognition, object recognition, object tracking, and image segmentation, is widely applied in edge computing scenarios including transportation, healthcare, education, and smart cities. Considering communication overhead, privacy and security, and the application's need for low latency, edge computing platforms have become an important way to deploy deep learning applications. The task requests in these scenarios typically originate from specific areas. Due to the complexity of the service scenarios, the simultaneous processing of multiple deep learning inference tasks is showing an inevitable growth trend. On the other hand, the resource utilization of specific types of tasks is usually very limited, while simultaneously needing to support numerous service scenarios. To reduce deployment costs, sharing computing resources among multiple deep learning models has become a common solution, such as edge computing boxes like the TW-T906 / TW-T906G and UBoxAI. These are based on a single GPU, providing inference services for numerous applications in specific scenarios. Because the inference latency of deep learning inference tasks varies significantly and task requests arrive randomly, long tasks can cause subsequent short tasks to wait excessively, resulting in poor QoS for the short tasks. Ensuring the QoS of each task request based on an effective resource allocation strategy remains a challenge.

[0003] Existing work has discussed the above resource allocation problem from the perspectives of computational unit level, computation graph level, and task level.

[0004] Resource allocation at the computational unit level leverages the worker thread context switching mechanism of computational resources (such as GPUs) to improve resource utilization when multiple deep learning model inference tasks share computational resources through thread blocking, looping, and other methods. A widely adopted example is MPS (Multi-Process Parallel Acceleration Service), a multi-process parallel acceleration service developed by NVIDIA for its Kepler architecture GPUs. On the one hand, deep learning inference resources are becoming increasingly diverse, including ARM's Mali-GPUs and traditional low-power devices such as FPGAs and DSPs, making the applicable scenarios for MPS technology very limited. On the other hand, the core of this technology lies in timely utilization of idle GPU cores to improve resource utilization. Existing research indicates that while this method increases the overall system throughput, it cannot predict the QoS of the perception task. Random resource allocation from the task's perspective introduces serious unfairness, resulting in the inability to effectively guarantee the QoS of the task. Although some research efforts have been made to address the shortcomings of this technology based on the idea of ​​task preemption, the recent published results (REEF: Han M, Zhang H, Chen R, et al. Microsecond-scale Preemption for Concurrent{GPU-accelerated}{DNN}Inferences[C] / / 16th USENIX Symposium on OperatingSystems Design and Implementation(OSDI 22).2022:539-558.) are limited to preemptive scheduling based on task preset priorities. That is, it provides services to the scene on the premise of ensuring the QoS of specific tasks, but it actually ignores the QoS requirements of other tasks.

[0005] Resource allocation at the computational graph level (Fuxun Yu, Shawn Bray, DiWang, Longfei Shangguan, Xulong Tang, Chenchen Liu, and Xiang Chen. 2021. Automated Runtime-Aware Scheduling for Multi-Tenant DNN Inference on GPU. In IEEE / ACM International Conference On Computer Aided Design, ICCAD 2021, Munich, Germany, November 1-4, 2021. IEEE, 1–9. https: / / doi.org / 10.1109 / ICCAD51958.2021.9643501) combines multiple deep learning models into a large model with multiple parallel branches. By rationally ordering and fusing the internal computational units, the overall computational efficiency is accelerated. This type of work is mainly applicable to scenarios where multiple deep learning tasks are strongly correlated, and these task requests are always generated and invoked synchronously. It is not suitable for scenarios where multiple tasks occur independently and randomly.

[0006] Task-level resource allocation treats tasks as resource management units, which are insensitive to the structural details of deep learning models, thus simplifying scheduling. For example, (Arpan Gujarati, Reza Karimi, Safya Alzayat, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace. 2020. Serving DNNslike Clockwork: Performance Predictability from the Bottom Up. CoRR abs / 2006.02464(2020). arXiv:2006.02464 https: / / arxiv.org / abs / 2006.02464) uses FCFS task-level scheduling, discarding tasks that are predicted to have unreliable QoS upon arrival. (Yujeong Choi and Minsoo Rhu. 2020. PREMA: A Predictive Multi-Task Scheduling Algorithm For Preemptible Neural Processing Units. In IEEE International Symposium on HighPerformance Computer Architecture, HPCA 2020, San Diego, CA, USA, February 22-26, 2020. IEEE, 220–233. https: / / doi.org / 10.1109 / HPCA47549.2020.00027) This paper presents a token-based online task scheduling algorithm based on QoS-aware prediction of task completion time, achieving overall optimization. However, while this approach can predict task QoS in advance, it cannot effectively address the problem of long tasks consuming computational resources and causing subsequent short tasks to wait for excessively long periods. This results in short tasks facing poor QoS, which, although predictable, cannot be effectively adjusted. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies, the purpose of this invention is to provide a QoS-aware resource allocation method and apparatus in deep learning multi-model deployment scenarios. First, by analyzing the computational graph structure of the deep learning model and automatically and uniformly partitioning it while ensuring the sequential dependencies of the partitioned sub-models, long tasks are uniformly divided into multiple sequential sub-tasks. This effectively solves the problem of excessively long waiting times for short tasks caused by long tasks while achieving predictable QoS awareness for tasks. Then, a microsecond-level task scheduling and sorting strategy is proposed. Based on continuous monitoring of task QoS, the resource allocation strategy for tasks is adjusted with extremely low scheduling overhead. Finally, a dynamic decision-making mechanism is provided to adaptively adjust the working mode of tasks, further reducing the jitter of task service levels.

[0008] To achieve the above objectives, the technical content of the present invention includes:

[0009] A QoS-aware resource allocation method for a deep learning multi-model deployment scenario, the method comprising:

[0010] By splitting a deep learning model into multiple sequentially dependent sub-models, its corresponding target task is also split into multiple sub-tasks.

[0011] The subtask is inserted into the global task queue based on the overall response ratio of all subtasks in the global task queue when the queue changes.

[0012] When the subtask is to be computed, a token is issued to the subtask based on the number of tasks of each type and the attributes of the subtask, so as to obtain the running result of the target task based on the deep learning model or multiple serially dependent sub-models.

[0013] Furthermore, the deep learning model is split into multiple sequentially dependent sub-models based on the sub-model splitting cost and splitting uniformity, including:

[0014] Convert deep learning models to ONNX format;

[0015] The ONNX format model is split based on the set splitting points to generate an initial splitting strategy population.

[0016] The effect prediction is performed on a contemporary splitting strategy population, and the fitness of a single individual in the splitting strategy population is obtained based on the effect prediction results; wherein the effect prediction results include: splitting cost and splitting uniformity;

[0017] Determine whether the partitioning cost and partitioning uniformity of the maximum fitness strategy meet the criteria.

[0018] If the criteria are not met, the next generation of the splitting strategy population is obtained by performing individual selection, crossover and mutation on individuals in the splitting strategy population, and then returning to the current splitting strategy population to perform effect prediction in order to obtain the fitness of individual individuals in the splitting strategy population.

[0019] Under the condition of meeting the standard, multiple serially dependent sub-models are obtained.

[0020] Furthermore, the step of splitting the ONNX format model based on the set splitting points to generate an initial splitting strategy population includes:

[0021] Based on the execution order of the operators, the operator numbers are sequentially numbered from 1 to n;

[0022] The ONNX format model is divided into N+1 serial sub-models to obtain the initial segmentation strategy population, where N≥0. Sub-model M1 consists of operators with operator indices from 1 to x1. i Operator number (x) i-1 +1)~x i The operator structure, sub-model M N+1 Operator number (x) N The operators from +1 to n are constructed, where 2≤i≤N.

[0023] Furthermore, the prediction of the performance of the contemporary splitting strategy population includes:

[0024] Obtain the inference latency of the deep learning model respectively. M And the inference latency of each sub-model i ;

[0025] Based on the inference latency M and the inference delay i The cost of partitioning is obtained;

[0026] Based on the inference latency i The standard deviation is used to obtain the uniformity of the cut.

[0027] Furthermore, the adaptability of the individual Cost raw is the execution time of the original model, std is the standard deviation, N+1 is the number of sub-models, k1 is the weight coefficient of uniformity, k2 is the weight coefficient of splitting cost, p is the first expectation coefficient, q is the second expectation coefficient, and overhead is the splitting cost.

[0028] Further, the step of inserting the subtask into the global task queue based on the overall response ratio of all subtasks in the global task queue when the queue changes includes:

[0029] Get the queued time, estimated queued time, and inference latency of the deep learning model corresponding to each subtask in the global task queue. M ;

[0030] When inserting a subtask into the global task queue at any position, obtain the estimated inference latency for each subtask in the global task queue;

[0031] Based on the existing queue time, the estimated queue time, and the inference latency M Based on the expected inference delay, the response ratio of each subtask in the global task queue is obtained;

[0032] The subtasks are inserted into the global task queue to minimize the overall response ratio of all subtasks in the global task queue.

[0033] Further, the step of issuing tokens to the subtasks based on the number of tasks of each type and the attributes of the subtasks, in order to obtain the running result of the target task based on the deep learning model or multiple sequentially dependent sub-models, includes:

[0034] When the current task execution scenario is either the first scenario or the second scenario, the subtask is handed over to the deep learning model for execution to obtain the running result of the target task; wherein, the first scenario is that the subtask is the first part of the target task and the total number of task types of the target task exceeds a set threshold, and the second scenario is that the subtask is the first part of the target task and the total number of tasks corresponding to all task types other than the task type of the target task is greater than the set threshold.

[0035] If the current task execution scenario is not the first or second scenario, the sub-task is handed over to the corresponding sub-model, and the execution result of the target task is obtained based on the execution results of each sub-model.

[0036] A QoS-aware resource allocation device for a deep learning multi-model deployment scenario, characterized in that the device comprises:

[0037] The uniform splitting module is used to split the target task into multiple sub-tasks by splitting the deep learning model into multiple sequentially dependent sub-models.

[0038] The task management module is used to insert the subtask into the global task queue based on the overall response ratio of all subtasks in the global task queue when the queue changes.

[0039] The token management module is used to issue tokens to the subtasks based on the number of tasks of each type and the attributes of the subtasks when the subtasks are about to be computed.

[0040] The task processing module is used to obtain the running result of the target task based on the deep learning model or multiple serially dependent sub-models.

[0041] An electronic device is characterized by comprising a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing any of the methods described above.

[0042] A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method described in any of the preceding claims.

[0043] Compared with the prior art, the present invention has at least the following advantages.

[0044] (1) The system continuously senses the QoS of each task request, taking into account the low latency requirements of each task.

[0045] (2) The model-based segmentation technology effectively solves the problem of excessively long waiting time for short tasks caused by long tasks, and provides strong scheduling guarantee for improving the QoS of tasks;

[0046] (3) Each task has strong service stability and less jitter. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the system structure of the resource allocation system of the present invention.

[0048] Figure 2 This is a schematic diagram of the execution flow of the resource allocation system of the present invention.

[0049] Figure 3 This describes the abstract execution process of the gene algorithm in the model segmentation of the resource allocation system of this invention. Implementation

[0050] The technical solution of the present invention will be further described below with reference to the accompanying drawings. The described embodiments are some embodiments of the present invention, but do not represent all embodiments.

[0051] The resource allocation device for QoS awareness in deep learning multi-model deployment scenarios of the present invention, such as Figure 1 As shown, it includes a uniform segmentation module, a request responder, a task management module, a task processing module, and a token management module.

[0052] Uniform Splitting Tool: Called during the offline phase, it analyzes the deep learning model to be deployed and automatically generates a uniform splitting strategy to ensure consistent execution time for each sub-model. It also automatically adds redundant inputs and outputs to the deep learning model computation graph to achieve a completely serial dependency relationship between the sub-models. The tool's ultimate goal is to automatically generate the final deployment strategy and corresponding deployment configuration file. It consists of three parts: a model splitter, an execution predictor, and a deployment decision maker. First, the specific meanings of the following concepts are given: (1) splitting the deep learning model into multiple consecutive sub-models; (2) splitting cost; (3) splitting uniformity.

[0053] 1) Divide the deep learning model into multiple consecutive sub-models: Analyze the computation graph G of the original model M and divide it into k sub-graphs, numbered sub_G1 to sub_G2. k Then, the computation graph structure is restored to the model structure and stored, corresponding sequentially to sub_M1 to sub_M. k This partitioning requires that for any data D, it be used as the input to Sub_M1, Sub_M p The output is used as Sub_M p+1 The input is used to sequentially execute Sub_M1 to Sub_M. k The final result is consistent with the output obtained by directly using it as input to M;

[0054] 2) Segmentation Cost: Assume the inference delay of model M is latency. M Model Sub_M i The latency of inference i The overhead of partitioning can then be expressed as:

[0055]

[0056] 3) Uniformity of segmentation: Assuming model Sub_M i The latency of inference i Then uniformity refers to latency i The numerical values ​​are similar, specifically expressed as their standard deviations.

[0057] The three core components that make up this tool are described in detail below:

[0058] 1) Model splitter: Based on a given splitting scheme, it splits a deep learning model into multiple sequentially dependent sub-models;

[0059] 2) Execution Predictor: Calculates the inference latency prediction value for each sub-model generated for a specific partitioning scheme and evaluates the partitioning overhead corresponding to that scheme. The prediction method involves executing the predictor n times in a task-exclusive manner on the resource device. Due to the cold start problem of model loading, the prediction model is as follows:

[0060]

[0061] Among them latency m This is the model inference delay measured in the m-th iteration.

[0062] 3) Deployment Decision Maker: It comprehensively evaluates the splitting cost and splitting uniformity of the current splitting scheme. When the expected target value is reached, it is used as the final output strategy and its corresponding deployment configuration file is created. This file contains the structural information of each sub-model and the relationship between them. Otherwise, it generates the next set of splitting schemes based on heuristic strategies and hands them over to the model splitter.

[0063] Request Response: In the online phase, it receives and registers task requests from users registered with the system based on protocols such as RPC, and promptly responds to the user with the calculation results after the system has completed the processing;

[0064] Task Management Module: Invoked during online phases, it performs tasks such as data format encapsulation / decapsulation, monitoring, logging, priority sorting, and push notifications for user-requested tasks. It consists of five components: a task encapsulator, a task decapsulator, a task manager, a priority sorter, and a log collector.

[0065] 1) Task Wrapper: Deep learning runtime deployment frameworks typically have their own data representation format. To decouple from the underlying deep learning deployment runtime framework, this functional device first converts the data format upon receiving a task request, transforming the user request data into runtime-specific input format data. It then associates this data with an internally defined data structure. This structure is used to more completely represent and record various characteristics of a task request for subsequent processing. This data structure includes the input / output data in the format specified by the deep learning model, the start and end times of each stage of task processing, the deep learning model runtime to be called by the task and the estimated inference time, the actual inference time of calling the deep learning runtime at each stage of task processing, the task processing progress, the task reception and completion times, and the task number. If any data is missing, it is temporarily filled with a null value.

[0066] 2) Task Unwrapper: It is the reverse process of the task wrapper. It mainly parses the various characteristics of task processing recorded by the system from the internally defined data structure and transforms the runtime framework-specific output format data into a user-readable general format.

[0067] 3) Task Manager: Manages all pending task requests in the current system and maintains a global priority queue. Upon completion of the previous computation task, it promptly pushes the highest-priority subtask to the computation module, while simultaneously monitoring the status of each task (waiting to process, processing, completed). When a task's status changes, it writes log information to the data structure.

[0068] 4) Priority Sorter: Newly added tasks are broken down into multiple sequential subtasks according to the model deployment strategy. A quicksort algorithm based on response ratio is used to calculate their respective priority numbers, and finally, these subtasks are inserted into the system priority queue. Specifically, the sorting algorithm inserts the corresponding subtasks of the new task into the queue without changing the existing priority comparison relationships of the subtasks in the system. A reasonable insertion strategy is calculated to ensure that all recorded tasks achieve the lowest overall response ratio under the above conditions. The response ratio is calculated as follows:

[0069]

[0070] 5) Log Collector: As a unified logging tool, it records all log information generated by tasks for later review.

[0071] Task processing module: In the online phase, it actually loads the task request data and the deep learning inference runtime framework, and calculates the output results. It consists of two parts: a model executor and a controller.

[0072] 1) Model Executor: Performs the specific calculations for a task / subtask. There are two working modes: "serial model execution" and "original model execution". In the "serial model execution" mode, only the pushed subtask is completed; in the "original model execution" mode, the entire task corresponding to the subtask is processed at once, which requires that the pushed subtask must be the first part of the entire task.

[0073] 2) Controller: Launches independent model executors for different types of tasks in a multi-threaded manner, controls their lifecycle, determines when they enter the execution state from the blocked state, and monitors their status information.

[0074] Token Management Module: In the online phase, it issues specific types of tokens to tasks pushed by the Task Management Module, specifies different task calculation methods based on the token type, and provides a token verification mechanism to ensure that only the model executor corresponding to the task can pass the verification. It consists of three parts: a split switch decision unit, a token distributor, and a token verifier.

[0075] 1) Task Splitting Decision Maker: Based on the number of each type of task in the system and the attributes of the currently pushed subtask, it issues two different types of tokens to determine which computation method the system should use to process the data. The specific algorithm decision is as follows:

[0076] a) When the current subtask is the first part of the user's original task (assuming task type T), and the total number of unprocessed tasks of type T exceeds 1, the current subtask is executed by directly processing the original model.

[0077] b) If the current subtask is the first part of the user's original task (assuming the task type is T), and the total number of tasks corresponding to all types of tasks other than type T is greater than 1, the current subtask will be executed by directly processing the original model.

[0078] c) Except as described above, the current subtask should be executed in a serial sub-model processing manner.

[0079] 2) Token Distributor: Binds the token to the model executor ID corresponding to the subtask, so that only the specified model executor can pass the token verification;

[0080] 3) Token Validator: Provides the function of validating the token and the ID of the model executor. It can only be verified when the ID is bound to the token.

[0081] In one embodiment, the QoS-aware resource allocation method for deep learning multi-model deployment scenarios of the present invention, such as... Figure 2 As shown, it includes steps 001-006 in the offline decision-making stage and steps 101-121 in the online service stage.

[0082] 1. Offline decision-making stage

[0083] Step 001: The user pre-exports the deep learning model to be deployed as an ONNX format. This step is usually directly supported by development frameworks such as PyTorch and TensorFlow, serving as input for the automated uniform model splitting tool. The tool will analyze different model files independently.

[0084] Step 002: The model splitter reads the model;

[0085] Step 003: The model segmenter generates sub-models based on the individual information in the segmentation strategy population and hands them over to the execution predictor. The initial segmentation strategy population is randomly generated.

[0086] Step 004: The predictor performs prediction and evaluation on the sub-models submitted by the model splitter, and gives the split uniformity and split cost corresponding to each strategy in the population.

[0087] Step 005: The deployment decision maker analyzes the prediction results fed back by the execution predictor. When the optimal solution reaches the expected value (i.e., the splitting is sufficiently uniform and the splitting cost meets the threshold), the algorithm ends; otherwise, the above genetic algorithm continues to be executed to generate the next generation of splitting strategy populations, which are submitted to the model splitter, and steps 003 to 005 are repeated;

[0088] Step 006: When the deployment decision maker ends the algorithm, it records the optimal splitting strategy and generates the corresponding sub-model files and their corresponding description information on the disk, so that when the online system starts, they can be read and called by the model executor.

[0089] Among them, the operation of the offline decision-making stage utilizes the classic genetic algorithm in the heuristic algorithm. First, the technical details of the genetic algorithm are introduced. The specific process can be referred to Figure 3 :

[0090] The processing object of the genetic algorithm is the ONNX model. Each time the genetic algorithm runs, it will have the number of expected sub-models input by the user. Assume that the ONNX model has a total of M operators, and the user expects to split it into N + 1 (N ≥ 0) serial sub-models. At this time, according to the execution order of the operators, the operator numbers are sequentially numbered from 1 to M.

[0091] Coding rule of the strategy: [x1, x2, …, x N represents that after splitting, the sub-models M1 to M N+1 in which M i (2 ≤ i ≤ N) is composed of the operators with operator numbers from (x i-1 + 1) to x i , M1 is composed of the operators with operator numbers from 1 to x1, and M N+1 is composed of the operators with operator numbers from (x N + 1) to M. It should satisfy x i < M (1 ≤ i ≤ N), and for any 1 ≤ i < j ≤ N, there is x i ≠ x j .

[0092] Adaptability calculation: For the adaptability evaluation of a single individual in the splitting strategy population, it is obtained based on the standard deviation std of the execution time of each sub-model corresponding to the strategy and the calculation of the overhead overhead. The specific calculation formula is as follows:

[0093]

[0094] Among them, Cost raw is the execution time of the original model, std is the standard deviation of the execution time of the sub-models of a specific individual (splitting strategy), N + 1 is the number of sub-models; the weight coefficient k1 of uniformity, the weight coefficient k2 of splitting cost, the expected coefficient p, and the expected coefficient q are all empirical values.

[0095] When determining whether a population meets the threshold, it is only necessary to check whether the overhead and std of the strategy corresponding to the minimum fitness (optimal individual) meet the criteria. In fact, if the optimal individual produced for 5 consecutive generations remains unchanged, this method considers that the optimal solution of the model has been found, meaning that the set threshold cannot be reached.

[0096] Individual Selection: Assuming there are m individuals in the population, the probability of individual g being selected is:

[0097]

[0098] Crossover rule: For codes of [a1, a2, ..., a N Individual A and [b1,b2,...,b] N Individual B, whose crossover produces offspring encoded as [a1*r1+b1*(1-r1),a2*r2+b2*(1-r2),...,a N *r N +b N *(1-r N )], where r1~r N It is a random floating-point number between 0 and 1.

[0099] Mutation rule: For each individual offspring produced, there is a certain probability of mutation. During mutation, x1 to x... are randomly selected... N Adjust left and right, but still satisfy the coding rules of the strategy.

[0100] In addition, when generating new offspring populations, it should always be ensured that the best individuals from the previous generation are fully preserved in the next generation.

[0101] 2. Online service phase

[0102] Step 101: The user sends a task request to the request responder, containing the task data to be processed;

[0103] Step 102: Task encapsulation receives a new task request submitted by the task responder;

[0104] Step 103: The task wrapper breaks down the task into multiple subtasks and encapsulates them according to the internally defined data structure.

[0105] Step 104: The priority sorter calculates the priority of the subtasks generated by the task wrapper and inserts them into the global task queue;

[0106] Step 105: Register the task information with Task Manager for subsequent management;

[0107] Step 106: Task Manager continues to log the task.

[0108] Step 107: When the system is in the initial state or receives an end signal from the model executor, the task manager pushes the highest priority subtask at the current moment to the token distributor. If the subtask is marked as completed, the subtask will be automatically deleted and the next highest priority subtask will be pushed.

[0109] Step 108: The token distributor calls the split switch decision-maker to request the token category to be written;

[0110] Step 109: The split switch decision-maker writes the category to the token based on the current system task status;

[0111] Step 110: The token distributor writes data to the token, binding the token to the model executor corresponding to the subtask;

[0112] Step 111: After the token is generated, the token control module pushes it to the controller;

[0113] Step 112: The controller sends the model executor ID and the token to the token verifier for comparison until the verification is successful, thereby confirming the model executor thread to be woken up and the working mode.

[0114] Step 113: The token validator refreshes the token content in preparation for the next round of token generation;

[0115] Step 114: The controller wakes up the model executor thread;

[0116] Step 115: The model executor loads the subtask input data from the task data structure;

[0117] Step 116: The model executor performs calculations according to the specified working mode to obtain the calculation results;

[0118] Step 117: The model executor records the calculation results into the task data structure;

[0119] Step 118: The model executor sends a signal to the task manager that the subtask calculation has ended, so that the next subtask can be successfully pushed.

[0120] Step 119: When all subtasks corresponding to a task have been processed, the task unsealer will complete the unsealing. At this time, the task manager will send its complete log information to the log collector.

[0121] Step 120: The request responder receives the processing result from the task unsealer and responds to the user;

[0122] Step 121: The user receives the returned data from the request responder.

[0123] The above embodiments are provided for the purpose of describing the present invention and are not intended to limit the scope of the invention. The scope of the invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the invention should be covered within the scope of the invention.

Claims

1. A QoS-aware resource allocation method for deep learning multi-model deployment scenarios, characterized in that, The method includes: By splitting a deep learning model into multiple sequentially dependent sub-models, its corresponding target task is also split into multiple sub-tasks. The subtask is inserted into the global task queue based on the overall response ratio of all subtasks in the global task queue when the queue changes. When the subtask is to be computed, a token is issued to the subtask based on the number of tasks of each type and the attributes of the subtask, so as to obtain the running result of the target task based on the deep learning model or multiple serially dependent sub-models. This involves breaking down the deep learning model into multiple sequentially dependent sub-models, including: Convert deep learning models to ONNX format; The ONNX format model is split based on the set splitting points to generate an initial splitting strategy population. The effect prediction is performed on a contemporary splitting strategy population, and the fitness of a single individual in the splitting strategy population is obtained based on the effect prediction results; wherein the effect prediction results include: splitting cost and splitting uniformity; Determine whether the partitioning cost and partitioning uniformity of the maximum fitness strategy meet the criteria. If the criteria are not met, the next generation of the splitting strategy population is obtained by performing individual selection, crossover and mutation on individuals in the splitting strategy population, and then returning to the effect prediction of the current splitting strategy population to obtain the fitness of individual individuals in the splitting strategy population. Under the condition that the standard is met, multiple serially dependent sub-models are obtained; The step of issuing tokens to subtasks based on the number of tasks of each type and the attributes of the subtasks, in order to obtain the running result of the target task based on the deep learning model or multiple sequentially dependent sub-models, includes: When the current task execution scenario is either the first scenario or the second scenario, the subtask is handed over to the deep learning model for execution to obtain the running result of the target task; wherein, the first scenario is that the subtask is the first part of the target task and the total number of task types of the target task exceeds a set threshold, and the second scenario is that the subtask is the first part of the target task and the total number of tasks corresponding to all task types other than the task type of the target task is greater than the set threshold. If the current task execution scenario is not the first or second scenario, the sub-task is handed over to the corresponding sub-model, and the execution result of the target task is obtained based on the execution results of each sub-model.

2. The method as described in claim 1, characterized in that, The process of splitting the ONNX format model based on a set splitting point to generate an initial splitting strategy population includes: Based on the execution order of the operators, the operator numbers are sequentially numbered from 1 to n; The ONNX format model is divided into N+1 serial sub-models to obtain the initial segmentation strategy population, where N≥0. Sub-model M1 consists of operators with operator indices from 1 to x1. i Operator number (x) i-1 +1)~x i The operator structure, sub-model M N+1 Operator number (x) N The operators from +1 to n are constructed, where 2≤i≤N.

3. The method as described in claim 1, characterized in that, The prediction of the population performance of the contemporary splitting strategy includes: Obtain the inference latency of the deep learning model respectively. M And the inference latency of each sub-model i ; Based on the inference latency M and the inference delay i The cost of partitioning is obtained; Based on the inference latency i The standard deviation is used to obtain the uniformity of the cut.

4. The method as described in claim 3, characterized in that, The adaptability of the individual Cost raw is the execution time of the original model, std is the standard deviation, N+1 is the number of sub-models, k1 is the weight coefficient of uniformity, k2 is the weight coefficient of splitting cost, p is the first expectation coefficient, q is the second expectation coefficient, and overhead is the splitting cost.

5. The method as described in claim 1, characterized in that, The step of inserting the subtask into the global task queue based on the overall response ratio of all subtasks in the global task queue when the queue changes includes: Get the queued time, estimated queued time, and inference latency of the deep learning model corresponding to each subtask in the global task queue. M ; When inserting a subtask into the global task queue at any position, obtain the estimated inference latency for each subtask in the global task queue; Based on the existing queue time, the estimated queue time, and the inference latency M Based on the expected inference delay, the response ratio of each subtask in the global task queue is obtained; The subtasks are inserted into the global task queue to minimize the overall response ratio of all subtasks in the global task queue.

6. A resource allocation device for QoS awareness in a deep learning multi-model deployment scenario, characterized in that, The device includes: The uniform splitting module is used to split the target task into multiple sub-tasks by splitting the deep learning model into multiple sequentially dependent sub-models. The task management module is used to insert the subtask into the global task queue based on the overall response ratio of all subtasks in the global task queue when the queue changes. The token management module is used to issue tokens to the subtasks based on the number of tasks of each type and the attributes of the subtasks when the subtasks are about to be computed. The task processing module is used to obtain the running result of the target task based on the deep learning model or multiple serially dependent sub-models; This involves breaking down the deep learning model into multiple sequentially dependent sub-models, including: Convert deep learning models to ONNX format; The ONNX format model is split based on the set splitting points to generate an initial splitting strategy population. The effect prediction is performed on a contemporary splitting strategy population, and the fitness of a single individual in the splitting strategy population is obtained based on the effect prediction results; wherein the effect prediction results include: splitting cost and splitting uniformity; Determine whether the partitioning cost and partitioning uniformity of the maximum fitness strategy meet the criteria. If the criteria are not met, the next generation of the splitting strategy population is obtained by performing individual selection, crossover and mutation on individuals in the splitting strategy population, and then returning to the effect prediction of the current splitting strategy population to obtain the fitness of individual individuals in the splitting strategy population. Under the condition that the standard is met, multiple serially dependent sub-models are obtained; The step of issuing tokens to subtasks based on the number of tasks of each type and the attributes of the subtasks, in order to obtain the running result of the target task based on the deep learning model or multiple sequentially dependent sub-models, includes: When the current task execution scenario is either the first scenario or the second scenario, the subtask is handed over to the deep learning model for execution to obtain the running result of the target task; wherein, the first scenario is that the subtask is the first part of the target task and the total number of task types of the target task exceeds a set threshold, and the second scenario is that the subtask is the first part of the target task and the total number of tasks corresponding to all task types other than the task type of the target task is greater than the set threshold. If the current task execution scenario is not the first or second scenario, the sub-task is handed over to the corresponding sub-model, and the execution result of the target task is obtained based on the execution results of each sub-model.

7. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 5.

Images