Method and system for scheduling and deploying deep learning models under limited hardware resources

Abstract

The present invention provides a deep learning model scheduling and deployment method and system under limited hardware resource conditions, including: Step M1: Obtain multiple phase tasks with dependencies in deep learning reasoning tasks; Step M2: Perform multiple phase tasks Resource consumption is evaluated separately; Step M3: Evaluate the currently available deployment hardware resources; Step M4: Select the currently runnable stage tasks from multiple dependent stage tasks; schedule the currently runnable stage tasks according to the currently available deployment hardware resources Stage tasks and update the evaluation of available deployment hardware resources, and repeat step M4 until all stage tasks in the deep learning reasoning task are completed. The invention realizes efficient adaptation of computing resources in different deployment environments and flexible adaptation of subsequent reasoning model changes.

Description

Technical field [0001] The present invention relates to the scheduling and deployment of depth learning models, and in particular, to the deployment method and system scheduling deployment method and system for depth learning models under limited hardware resource conditions. Background technique [0002] With the rapid development of computer hardware and the rapid development of deep learning, various neural network models are applied to various fields such as life health, retail, industrial. The successful application of depth learning models is in business sectors depending on multiple links. In addition to model training, it is often necessary to optimize and deploy training well-trained models for use. The user passed to the deployed model to the data, and after the input data passes the model reasoning, the user gets the corresponding output result. [0003] In order to optimize the model reasoning process, the model reasoning engine that is convenient for industrial deplo...

AI Technical Summary

Problems solved by technology

The optimization direction of these engines and tools is more focused on their own hardware or the framework itself, and it is impossible to optimize the effective automatic loading and unloading and reasonable scheduling of the deployed multi-model according to the characteristics of the inference task under the condition of limited computing resources.

In the existing medical image recognition scenario, the overall business process is jointly completed by multiple large-scale deep model reasoning tasks with complex dependencies, including multiple modules such as pre-processing, model reasoning, and post-processing. complex

Because the GPU memory is limited and difficult to expand, the overall reasoning task is limited by the memory, and the reasoning process takes a long time

Although most of the existing inference deployment engines can deploy multiple model services at the same time, under the condition of video memory limitation, they cannot reasonably schedule model inference tasks according to the resource consumption of model inference runtime, and cannot finely control computing resources.

Images