Multi-model reasoning service deployment method and device based on k8s cluster

Abstract

The invention discloses a multi-model reasoning service deployment method and device based on a k8s cluster. The method comprises the following steps: deploying a scheduling service in a minimum scheduling unit of a k8s cluster, and configuring a memory, computing resources and a scheduling strategy for the scheduling service; deploying a plurality of model inference services according to a memoryof a scheduling service, and configuring each model inference service as a computing resource for using the scheduling service and to be associated with the scheduling service; and the scheduling service calls the plurality of model reasoning services according to the scheduling strategy so as to process the reasoning task. According to the scheme of the invention, the capability of sharing the minimum scheduling unit by multiple model inference services is realized, the multi-model inference service can be elastically expanded and contracted along with the service load, and the deployment operation is relatively simple.

Description

technical field [0001] The invention belongs to the field of cloud computing, and in particular relates to a k8s cluster-based multi-model reasoning service deployment method, device, computer equipment and storage medium. Background technique [0002] As machine learning methods are more and more widely used in actual production, the number of models that need to be deployed in the production system is also increasing. For example, machine learning applications to provide a personalized experience often require the training of many models; for example, a news classification service will train a custom model on news categories, and a recommendation model can train each user's usage history to personalize its recommendations; respectively The main reason for training so many models is to protect the user's model and data privacy. [0003] In the K8S cluster, the number of POD resources is limited (by default, each Node can start 110 POD instances), and by default, in a clust...

AI Technical Summary

Problems solved by technology

Currently, the main way to deploy multi-models is to deploy services that support multi-model loading in the system, such as Tensor Flow Serving, Trion Serving, and AWS Multi-Model Serving. However, such services are traditional services and do not support elastic scaling in clusters. , and the operation is complex

Images