Model virtualization deployment method and device, storage medium and computer equipment

Abstract

According to the model virtualization deployment method and device, the storage medium and the computer equipment provided by the invention, after the target model selected by the user under the at least one model version of the at least one type created in advance is obtained, the reasoning framework and the at least one resource used when the target model is deployed can be determined firstly; creating a corresponding number of virtual instances according to the number of resources used when the target model is deployed, binding the virtual instances with a specified pod in the k8s service configured under the resources of the corresponding specification, and deploying the target model to the specified pod; in the process, the resources required by the model reasoning service are divided into different specifications in advance, the resources of each specification are configured with one k8s service, and each k8s service comprises a plurality of pods, so that a user only needs to select the resources of the corresponding specification according to the size of the model when deploying the model, and then the model can be deployed to the specified pods; and automatic deployment of the model is realized.

Description

Technical Field

[0001] The present application relates to the field of model deployment technology, and in particular to a model virtualization deployment method, device, storage medium and computer equipment. Background Art

[0002] In current artificial intelligence (AI) applications, large language models have become the focus of research and industry due to their excellent performance in natural language processing tasks. These models have demonstrated powerful capabilities in processing tasks such as text generation, machine translation, and sentiment analysis. However, designing and training these models is only part of the entire process, and model management and deployment are also crucial. In existing technologies, the deployment of most models requires a lot of manual operations, which places high demands on people without software engineering backgrounds and seriously restricts the ability to deploy on a large scale and quickly.

[0003] In the existing technology, the deployment of most models requires a lot of manual operations and requires users to have high software engineering skills. For example, users are required to manually configure the environment, adjust parameters, and perform model online operations, which is not only time-consuming and labor-intensive, but also prone to errors, which seriously restricts the ability of large-scale and rapid deployment of models. At the same time, this method has limited ability to dynamically adjust resources and is difficult to fully utilize the advantages of hardware resources (such as GPUs), resulting in waste of computing resources and low operating efficiency. Summary of the invention

[0004] The purpose of this application is to solve at least one of the above-mentioned technical defects, especially the technical defects that manual deployment of models in the prior art is not only time-consuming and labor-intensive, but also prone to errors, which seriously restricts the ability of large-scale and rapid deployment of models, and makes it difficult to fully utilize the advantages of hardware resources, resulting in waste of computing resources and low operating efficiency.

[0005] The present application provides a model virtualization deployment method, the method comprising:

[0006] Obtain a target model selected by a user under at least one model version of at least one pre-created type, and determine an inference framework and at least one resource used when deploying the target model, wherein each resource corresponds to a different resource specification, each resource specification is configured with a k8s service, each k8s service contains multiple pods, and all pods under the same k8s service have the same specifications;

[0007] Create a corresponding number of virtual instances according to the number of resources used when deploying the target model, bind the virtual instances to the specified pods in the k8s service configured under the corresponding specification resources, and deploy the target model to the specified pods;

[0008] Receive the model loading result sent by the designated pod through the information storage component, and when it is determined that the model loading result is successfully loaded, register the routing information corresponding to the virtual instance of the target model into the model service routing table.

[0009] Optionally, determining an inference framework used when deploying the target model includes:

[0010] Recommending a list of inference frameworks to the user according to the model type of the target model, wherein each inference framework in the list of inference frameworks is sorted according to framework performance;

[0011] The inference framework selected by the user in the inference framework list is obtained, and the inference framework is used as the inference framework used when deploying the target model.

[0012] Optionally, the determining at least one resource used when deploying the target model includes:

[0013] recommending a resource specification list to the user according to the model size of the target model;

[0014] Acquire at least one resource selected by a user in the resource specification list, and use the at least one resource as at least one resource used when deploying the target model;

[0015] When the user selects more than two resources, the resources may be from the same cloud vendor or different cloud vendors.

[0016] Optionally, the recommending a resource specification list to the user according to the model size of the target model includes:

[0017] Determining the parameter amount and optimizer parameters of the target model;

[0018] Calculate the size of the video memory occupied by the parameters of the target model according to the parameter amount and the optimizer parameters;

[0019] The video memory size is compared with pre-divided resources of different specifications, and a resource specification list is recommended to the user according to the comparison result.

[0020] Optionally, the process of dividing the resources of different specifications includes:

[0021] Determine the resources of at least one cloud vendor required for the model inference service;

[0022] The resources of the at least one cloud vendor are divided into multiple specifications according to preset resource division rules.

[0023] Optionally, when the model inference service requires two cloud vendors, the client is deployed on one of the cloud vendors and stores the model files uploaded by users, and the server is deployed on the other cloud vendor, and the model files are synchronized from the cloud storage of one of the cloud vendors to the cloud storage of the other cloud vendor through unison two-way synchronization.

[0024] Optionally, the specified pod includes a control container and an inference framework main container;

[0025] The deploying the target model to the specified pod includes:

[0026] A model loading instruction is sent to the control container so that the control container can record the model information of the target model, and the model loading instruction is forwarded to the reasoning framework main container. The reasoning framework main container loads the target model according to the model loading instruction and sends the model loading result to the control container. The control container sends the model loading result to the information storage component.

[0027] Optionally, the method further comprises:

[0028] When it is detected that a version update is performed using the deployed target model, a health check is performed on the routing information of the deployed target model;

[0029] When the health check of the deployed target model is successful, the routing information of the previous version corresponding to the deployed target model is deleted from the model service routing table, and the previous version of the model inference service is offline.

[0030] Optionally, the method further comprises:

[0031] During the model inference process, if it is detected that the user has enabled scheduled scaling, the scheduled scaling period and the expected number of replicas configured when the user enabled scheduled scaling are obtained;

[0032] When it is detected that the scheduled expansion and contraction period has arrived, the original number of copies of the model inference instance is saved, and the original number of copies is adjusted to the expected number of copies;

[0033] When it is detected that the scheduled expansion and contraction period has been exceeded, the expected number of replicas is adjusted to the original number of replicas.

[0034] Optionally, the method further comprises:

[0035] During the model reasoning process, monitor the resource usage of the virtual instance and determine whether the resource usage meets the preset expansion conditions;

[0036] If satisfied, determine whether the number of virtual instances is the maximum number of copies;

[0037] If so, then end;

[0038] If not, create a new virtual instance, bind the new virtual instance to the specified pod in the k8s service configured under the corresponding specification resources, deploy the target model to the specified pod and its subsequent steps;

[0039] If not, determining whether the number of virtual instances is the minimum number of copies;

[0040] If so, then end;

[0041] If not, after removing the route with the largest virtual instance number, uninstall the model in the corresponding resource, and terminate after the corresponding virtual instance is offline.

[0042] The present application also provides a model virtualization deployment device, including:

[0043] An information acquisition module is used to acquire a target model selected by a user under at least one model version of at least one type created in advance, and determine an inference framework and at least one resource used when deploying the target model, wherein each resource corresponds to a different resource specification, each resource specification is configured with a k8s service, each k8s service contains multiple pods, and all pods under the same k8s service have the same specifications;

[0044] A model deployment module is used to create a corresponding number of virtual instances according to the number of resources used when deploying the target model, bind the virtual instances to the specified pods in the k8s service configured under the corresponding specification resources, and then deploy the target model to the specified pods;

[0045] The routing registration module is used to receive the model loading result sent by the specified pod through the information storage component, and when it is determined that the model loading result is successfully loaded, register the routing information corresponding to the virtual instance of the target model into the model service routing table.

[0046] The present application also provides a computer-readable storage medium, in which computer-readable instructions are stored. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the service expansion method of the collection system as described in any of the above embodiments.

[0047] The present application also provides a computer device, comprising: one or more processors, and a memory;

[0048] The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the service expansion method of the collection system as described in any one of the above embodiments are performed.

[0049] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0050] The model virtualization deployment method, device, storage medium and computer equipment provided by the present application, after obtaining the target model selected by the user under at least one model version of at least one type created in advance, can first determine the inference framework and at least one resource used when deploying the target model, and then create a corresponding number of virtual instances according to the number of resources used when deploying the target model, and bind the virtual instances to the specified pods in the k8s service configured under the corresponding specification resources, and then deploy the target model to the specified pod; in this process, since the present application divides the resources required for the model inference service into different specifications in advance, and each specification of resource configuration has a k8s service, each k8s service contains Multiple pods, all pods under the same k8s service have the same specifications, so when deploying the model, the user only needs to select resources of corresponding specifications according to the model size to deploy the model to the specified pod, thereby realizing automatic deployment of the model. In addition, after the model is deployed, the application can also receive the model loading result sent by the specified pod through the information storage component, and when it is determined that the model loading result is successfully loaded, register the routing information corresponding to the virtual instance of the target model to the model service routing table, so that the corresponding virtual instance can be found directly through the routing information registered in the model service routing table for reasoning, thereby saving a lot of manpower, avoiding waste of resources, and improving the deployment efficiency and accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0052] Figure 1 A flow chart of a model virtualization deployment method provided in an embodiment of the present application;

[0053] Figure 2 A schematic diagram of a resource management architecture provided for an embodiment of the present application;

[0054] Figure 3 A schematic diagram of the virtual instance creation and pod binding process of the reasoning service provided in an embodiment of the present application;

[0055] Figure 4 A schematic diagram of the file synchronization process during multi-cloud deployment provided in an embodiment of the present application;

[0056] Figure 5 A schematic diagram of the deployment process of the model deployment service provided in the embodiment of the present application;

[0057] Figure 6 A schematic diagram of the expansion and contraction process after configuring the index expansion strategy according to the embodiment of the present application;

[0058] Figure 7 A schematic diagram of the structure of a model virtualization deployment device provided in an embodiment of the present application;

[0059] Figure 8 A schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0061] In one embodiment, Figure 1 As shown, Figure 1 A flow chart of a model virtualization deployment method provided in an embodiment of the present application; the present application provides a model virtualization deployment method, which may include:

[0062] S110: Acquire a target model selected by a user under at least one model version of at least one pre-created type, and determine a reasoning framework and at least one resource used when deploying the target model.

[0063] In this embodiment, when deploying a model, the application can provide a model management page, which supports users to create model types, create model versions, upload models, etc., so that each model can be recorded through version control, thereby supporting the backtracking of different versions of models. In addition, the model management page of the application also supports an automated model update and rollback mechanism to ensure the stability and reliability of the model during the update process.

[0064] Specifically, when the user creates different types of models in the model management page and creates different model versions under different types of models, the corresponding version of the model file can be uploaded to the storage service. The storage service of this application can be the cloud storage of the cloud vendor. Through the cloud storage, the model files uploaded by the user can be classified and stored according to different model versions, thereby effectively improving the efficiency of model management.

[0065] Furthermore, after the user uploads the model file of the corresponding version under the corresponding type on the model management page, if the model inference service is to be performed, the inference service can also be created on the inference service creation page so as to select the target model from at least one model version of at least one type created in advance. After the user selects the target model, the present application can determine the model type, model version and model size of the target model, so that the corresponding inference framework list can be recommended to the user based on the model type, so that the user can select the inference framework from the corresponding inference framework list. Of course, the present application can also directly determine the corresponding inference framework based on the model type, which can further reduce manual operations.

[0066] Similarly, when determining the resources used to deploy the target model, this application can also determine a list of resource specifications based on the model size of the target model and recommend them to the user so that the user can select the corresponding resources. It can also directly determine the resource specifications based on the model size and then select the corresponding resources. The specific settings can be made according to the actual situation and are not restricted here.

[0067] It is understandable that the present application divides the resources required for the model inference service into multiple specifications in advance. Figure 2 As shown, Figure 2 A schematic diagram of a resource management architecture provided for an embodiment of the present application; Figure 2 In this application, a k8s service is configured for each resource specification. Each k8s service contains multiple pods. Resource management uses the k8s API interface to deploy the k8s service. The k8s service is configured according to the resource specification size. All pods under the same k8s service have the same specifications.

[0068] Among them, the above-mentioned k8s (Kubernetes) service refers to an open source system for automating the deployment, expansion and management of containerized applications. Pod is the smallest deployable and manageable unit in Kubernetes. A pod can contain one or more containers. The containers in the pod share the network namespace and storage, and the pod can be deployed, scheduled and managed as a whole. In addition, when the load of the service is monitored to increase, such as CPU usage and memory usage exceeding the set threshold, or the number of requests received increases, the K8s service will automatically start more pods to share the workload to ensure the performance and responsiveness of the service. Of course, you can also explicitly specify the number of Pod replicas to run when creating a replica set or deployment. In addition, when some pods fail or are unavailable, the K8s service will automatically create new pods to replace the failed pods to maintain the desired service status.

[0069] Therefore, this application divides the resources required for the model inference service into different specifications in advance, and after configuring a k8s service for each specification of resources, when deploying the model, the user only needs to select the corresponding specification of resources according to the model size to deploy the model to the specified pod, thereby realizing the automatic deployment of the model.

[0070] In addition, in this application, users can select one or more resources according to business needs. When this application is deployed in multiple clouds, resources from different cloud vendors can also be selected. The specific selection can be made based on actual conditions and is not restricted here.

[0071] S120: Create a corresponding number of virtual instances according to the number of resources used when deploying the target model, bind the virtual instances to the specified pods in the k8s service configured under the corresponding specification resources, and then deploy the target model to the specified pod.

[0072] In this embodiment, after obtaining the target model through S110 and determining the inference framework and at least one resource used when deploying the target model, the present application can create a corresponding number of virtual instances according to the number of resources used when deploying the target model, and bind the virtual instances to the specified pods in the k8s service configured under the corresponding specification resources, and then deploy the target model to the specified pod.

[0073] Specifically, in order to record the correspondence between the inference service, model, resources, and pod, the present application can create a corresponding number of virtual instances according to the number of resources used when deploying the target model, so that the system can know which pods the model inference service is running on, so as to facilitate subsequent monitoring and offline virtual instances. In addition, since the present application configures a k8s service for each specification of resources, and each k8s service contains multiple pods, once the present application determines at least one resource used by the target model, it can determine at least one resource specification based on the at least one resource, and then determine at least one k8s service based on the at least one resource specification, so that the virtual instance can be bound to the specified pod in the k8s service configured under the corresponding specification resource, and the target model can be deployed to the specified pod.

[0074] Among them, when the present application binds the virtual instance to the specified pod, the specified pod can be a pre-agreed pod or a pod selected by the k8s service according to the current service status. The specific setting can be made according to the actual situation and is not restricted here.

[0075] Furthermore, if Figure 3 As shown, Figure 3 A schematic diagram of the virtual instance creation and pod binding process of the reasoning service provided in an embodiment of the present application; Figure 3 In the present application, after a virtualized reasoning service is created on the platform, a corresponding number of reasoning service instances, i.e., virtual instances, can be simulated according to the number of resources selected by the user, and then the virtual instances are bound to the specified pods in the k8s service configured under the corresponding specification resources, such as reasoning service instance 1 is bound to k8s pod1, and reasoning service instance 2 is bound to k8s pod2, and k8s pod1 and k8s pod2 are run on the same k8s node.

[0076] It is understandable that in this application, a k8s service can start multiple pods, a pod can contain multiple containers (such as sidecar (control container), reasoning framework main container, istio, etc.), and pods under the same k8s service can be started under different k8s nodes.

[0077] S130: receiving the model loading result sent by the designated pod through the information storage component, and when determining that the model loading result is successfully loaded, registering the routing information corresponding to the virtual instance of the target model into the model service routing table.

[0078] In this step, after S120 binds the virtual instance to the specified pod in the k8s service configured under the corresponding specification resources and deploys the target model to the specified pod, the model loading result sent by the specified pod through the information storage component can be received, so that subsequent steps can be performed according to the model loading result.

[0079] For example, the designated pod of this application can send the model loading result through an information storage component, such as etcd, which is equivalent to a stateful notification pipeline. Since the model deployment service in this application is often upgraded and restarted in daily iterations, if the model loading result is sent directly to the model deployment service, the designated pod needs to do additional retry work. After using etcd to send the message, the designated pod only needs to push the message to etcd to complete its work. The model deployment service can monitor the messages in etcd in real time and receive messages from etcd every time it restarts, thereby ensuring that the information of the model going online and offline is not lost.

[0080] When this application receives the model loading result sent by the specified pod through the information storage component, since the model loading result may be the result of successful loading or the result of failed loading, after receiving the model loading result, you can first determine whether the model loading result is successful loading or failed loading. If the loading is successful, you can register the routing information corresponding to the virtual instance of the target model to the model service routing table. When the model is called for reasoning, you can find the corresponding virtual instance for reasoning through the routing information registered in the model service routing table. If the loading fails, it means that the model service deployment failed. At this time, you can modify the model service status to failure and record the reason for the failure. The user can view it through the page.

[0081] In the above embodiment, after obtaining the target model selected by the user under at least one model version of at least one type created in advance, the inference framework and at least one resource used when deploying the target model can be determined first, and then a corresponding number of virtual instances are created according to the number of resources used when deploying the target model, and the virtual instances are bound to the specified pods in the k8s service configured under the corresponding specification resources, and the target model is deployed to the specified pod; in this process, since the application divides the resources required for the model inference service into different specifications in advance, and each specification of resource configuration has a k8s service, each k8s service contains multiple pods, and the same k8s service All pods under the service have the same specifications, so when deploying the model, the user only needs to select resources of corresponding specifications according to the model size to deploy the model to the specified pod, thereby realizing automatic deployment of the model. In addition, after the model is deployed, the application can also receive the model loading result sent by the specified pod through the information storage component, and when it is determined that the model loading result is successfully loaded, register the routing information corresponding to the virtual instance of the target model to the model service routing table, so that the corresponding virtual instance can be found directly through the routing information registered in the model service routing table for reasoning, thereby saving a lot of manpower, avoiding waste of resources, and improving the deployment efficiency and accuracy of the model.

[0082] In one embodiment, determining the reasoning framework used when deploying the target model in S110 may include:

[0083] S111: recommending a list of inference frameworks to the user according to the model type of the target model, wherein each inference framework in the list of inference frameworks is sorted according to framework performance.

[0084] S112: Acquire the inference framework selected by the user in the inference framework list, and use the inference framework as the inference framework used when deploying the target model.

[0085] In this embodiment, after the user uploads the model file of the corresponding version under the corresponding type on the model management page, if the model reasoning service is to be performed, the reasoning service can also be created on the reasoning service creation page to select the target model from at least one model version of at least one type created in advance. After the user selects the target model, the application can determine the model type, model version and model size of the target model, and recommend the corresponding reasoning framework list to the user based on the model type, so that the user can select the reasoning framework in the corresponding reasoning framework list. Of course, the application can also directly determine the corresponding reasoning framework according to the model type, which can further reduce manual operations.

[0086] It is understandable that the reasoning framework of this application is mainly used to load the corresponding model and provide reasoning services to the outside through the API interface. For example, the large language model has a large language model-related framework, which includes but is not limited to VLLM, Qwen, Llama, sglang, etc. The reasoning framework of the painting model includes but is not limited to sd-webui, comfyui, etc. These framework systems will support and store, and some frameworks can also be compatible. Therefore, the system will also record which frameworks are compatible with each other, and then sort them according to the framework performance to form a list of reasoning frameworks for users to choose.

[0087] When the user receives the inference framework list, he can select an inference framework from the inference framework list according to actual business needs. After the application obtains the inference framework selected by the user, the inference framework can be used as the inference framework when deploying the target model, thereby effectively improving the success rate of model loading.

[0088] In one embodiment, determining at least one resource used when deploying the target model in S110 may include:

[0089] S113: Recommending a resource specification list to the user according to the model size of the target model.

[0090] S114: Acquire at least one resource selected by the user in the resource specification list, and use the at least one resource as at least one resource used when deploying the target model; wherein, when the user selects more than two resources, the resources are from the same cloud vendor or different cloud vendors.

[0091] In this embodiment, when determining at least one resource used by the target model for deployment, a resource specification list can be determined according to the model size of the target model, and recommended to the user so that the user can select the corresponding resource. Of course, this application can also directly determine the resource specification according to the model size, and then select the corresponding resource. The specific setting can be based on the actual situation and is not limited here.

[0092] Furthermore, since the GPU resources of a cloud vendor are limited and not available at all times, the GPU resource models, prices, and performance of each cloud vendor are also different. Therefore, this application can adopt a multi-cloud deployment approach to provide model inference services. This can solve both the resource limitation problem and the price problem, as well as the system stability problem.

[0093] When this application adopts a multi-cloud deployment approach to provide model inference services, the resources of different cloud vendors can be orchestrated in advance, so that users can choose the resources of the same or different cloud vendors to directly deploy model inference services according to business needs. The specific choice can be made based on actual conditions and is not restricted here.

[0094] In one embodiment, in S113, recommending a resource specification list to the user according to the model size of the target model may include:

[0095] S1131: Determine the parameter quantity and optimizer parameters of the target model.

[0096] S1132: Calculate the size of the video memory occupied by the parameters of the target model according to the parameter quantity and the optimizer parameters.

[0097] S1133: Compare the video memory size with pre-divided resources of different specifications, and recommend a resource specification list to the user based on the comparison result.

[0098] In this embodiment, when recommending a list of resource specifications to a user based on the model size of a target model, the parameter quantity and optimizer parameters of the target model can be determined first. In this way, the size of video memory occupied by the parameters of the target model can be calculated based on the parameter quantity and the optimizer parameters. The video memory size can then be compared with pre-divided resources of different specifications. In this way, a list of resource specifications can be recommended to the user based on the comparison results.

[0099] For example, the size of video memory occupied by the parameters of the target model in this application = parameter quantity × N (float32: N=4; float16: N=2; double64: N=8) + optimizer parameters (generally 1G). After calculation according to this calculation formula, the size of video memory occupied by the parameters of the target model can be obtained. The resource sizes of different specifications pre-divided in this application generally include 25% computing power * 5G video memory, 50% computing power * 10G video memory, 75% computing power * 15G video memory, 100% computing power * 20G video memory, 200% computing power * 40G video memory, etc. This application can match the size of video memory occupied by the parameters of the target model with resources of different specifications, and select resource specifications that are not less than the video memory size to construct a resource specification list.

[0100] In one embodiment, the process of dividing resources of different specifications may include:

[0101] S310: Determine resources of at least one cloud vendor required for the model reasoning service.

[0102] S311: Divide the resources of the at least one cloud vendor into multiple specifications according to a preset resource division rule.

[0103] In this embodiment, before performing the model reasoning service, the present application may also first determine the resources of at least one cloud vendor required for the model reasoning service, and divide the resources of at least one cloud vendor into multiple specifications according to preset resource division rules.

[0104] For example, when this application only has resources from one cloud vendor that need to be divided, since the resources required for the model inference service are mainly GPU computing power and video memory, this application can divide the resources of this cloud vendor according to the preset resource division rules: 25% computing power * 5G video memory, 50% computing power * 10G video memory, 75% computing power * 15G video memory, 100% computing power * 20G video memory, 200% computing power * 40G video memory. Each specification of resources corresponds to a k8s service. Resource management uses the k8s API interface to deploy the k8s service, and the k8s service configuration is configured according to the specification size.

[0105] When this application is a multi-cloud deployment architecture, for each cloud vendor, the GPU computing power and video memory required for the model inference service can be divided into resources of different specifications according to the above-mentioned resource division rules. In this way, when recommending a resource specification list to the user, the video memory size occupied by the parameters of the target model can be compared with the resources of different specifications corresponding to each cloud vendor, and then the final resource specification list can be determined.

[0106] In one embodiment, Figure 4 As shown, Figure 4 A schematic diagram of the file synchronization process during multi-cloud deployment provided for an embodiment of the present application; when the model inference service requires two cloud vendors, a client is deployed on one of the cloud vendors and the model files uploaded by users are stored, and a server is deployed on the other cloud vendor, and the model files are synchronized from the cloud storage of one of the cloud vendors to the cloud storage of the other cloud vendor through unison two-way synchronization.

[0107] In this embodiment, Figure 4 As shown, the present application can deploy the model management service in the cluster of cloud vendor A, store the uploaded model files in the cloud storage of cloud vendor A, and deploy unison as a client on cloud vendor A and mount the cloud storage of cloud vendor A. The present application can also deploy unison as a server on cloud vendor B and mount the cloud storage of cloud vendor B. Then, the present application can synchronize the model files from the cloud storage of cloud vendor A to the cloud storage of cloud vendor B through unison two-way synchronization, thereby solving the problem of multi-cloud synchronization of model files.

[0108] In one embodiment, Figure 5 As shown, Figure 5 A schematic diagram of the deployment process of the model deployment service provided in an embodiment of the present application; the specified pod includes a control container and an inference framework main container.

[0109] Deploying the target model to the specified pod in S120 may include:

[0110] S121: Send a model loading instruction to the control container so that the control container can record the model information of the target model, and forward the model loading instruction to the reasoning framework main container. The reasoning framework main container loads the target model according to the model loading instruction and sends the model loading result to the control container. The control container sends the model loading result to the information storage component.

[0111] In this embodiment, Figure 5 As shown, after receiving the creation service instruction triggered by the user on the front-end page, this application can perform the following operations:

[0112] 1. Create a virtual reasoning service model_service;

[0113] 2. Create a corresponding number of virtual instances based on the number of resources selected by the user and bind them to the specified pod;

[0114] 3. Send a loading model instruction to the sidecar of the specified resource pod;

[0115] 4. After receiving the model loading instruction, Sidecar records the model information, including but not limited to the model name, model version, model file path, etc. This information will be recorded in Redis. In addition, Sidecar can also record the model list loaded by this pod. When the virtual instance is restarted, the model can be automatically loaded through this recorded information;

[0116] 5. Sidecar forwards the loading model instruction to the main container of the inference framework;

[0117] 6. The main container of the inference framework loads the model;

[0118] 7. Determine whether the model is loaded successfully;

[0119] 8. Notify the sidecar model of the loading result;

[0120] 9. Sidecar writes the model loading results to etcd;

[0121] 10. The model deployment service receives etcd messages;

[0122] 11. When the model is loaded successfully, the routing information corresponding to the virtual instance of the model is registered to the model service routing table.

[0123] Through the above deployment process, the target model can be deployed to the specified pod, thereby realizing the automatic deployment of the model. In addition, the corresponding virtual instance can be found directly through the routing information registered in the model service routing table for reasoning, thereby saving a lot of manpower, avoiding waste of resources, and improving the deployment efficiency and accuracy of the model.

[0124] In one embodiment, the method may further include:

[0125] S141: When it is detected that a version update is performed using a deployed target model, a health check is performed on the routing information of the deployed target model.

[0126] S142: When the health check of the deployed target model is successful, the routing information of the previous version corresponding to the deployed target model is deleted from the model service routing table, and the previous version of the model inference service is offline.

[0127] In this embodiment, after the target model is deployed, since the target model is the latest version of the model, the present application can also perform version update or model rollback on the model.

[0128] Specifically, the present application can first detect whether to use the deployed target model for version update. If necessary, a health check is performed on the routing information of the deployed target model. After the health check is successful, the routing information of the previous version corresponding to the deployed target model is deleted from the model service routing table, and the previous version of the model inference service is taken offline. In this way, the version update or model rollback can be completed.

[0129] In one embodiment, the method may further include:

[0130] S151: During the model inference process, if it is detected that the user turns on scheduled scaling, the scheduled scaling period and the expected number of replicas configured when the user turns on the scheduled scaling are obtained.

[0131] S152: When it is detected that the scheduled expansion and contraction period has arrived, the original number of copies of the model inference instance is saved, and the original number of copies is adjusted to the expected number of copies.

[0132] S153: When it is detected that the scheduled expansion and contraction period has been exceeded, the expected number of replicas is adjusted to the original number of replicas.

[0133] In this embodiment, during the model inference process, the incoming inference requests can be distributed more evenly to multiple instances by increasing the number of replicas, thereby improving the overall ability and response speed of the system to process requests, and when some replicas fail, other normal replicas can continue to process requests, thereby ensuring service continuity. When there are fewer inference requests, the number of replicas can be reduced to save resources.

[0134] For example, the present application can provide a timed scaling configuration entry in the front-end inference service configuration. After configuring the entry, the user can set whether to start the timed scaling service according to their own needs on the front-end page. If started, the back-end service can check whether the current time is within the timed scaling period according to the timed configuration. If so, the original number of replicas is saved and the number of replicas is adjusted to the configured number. After the timed scaling period has passed, the number of replicas is adjusted back to the original number of replicas, thereby achieving timed scaling, thereby improving the system's responsiveness while reducing manual operations and lowering labor costs.

[0135] In one embodiment, the method may further include:

[0136] S161: During the model reasoning process, monitor the resource usage of the virtual instance and determine whether the resource usage meets the preset expansion conditions.

[0137] S162: If satisfied, determine whether the number of virtual instances is the maximum number of copies.

[0138] S163: If yes, then end.

[0139] S164: If not, create a new virtual instance, bind the new virtual instance to the specified pod in the k8s service configured under the corresponding specification resources, deploy the target model to the specified pod and subsequent steps.

[0140] S165: If not satisfied, determine whether the number of virtual instances is the minimum number of copies.

[0141] S166: If yes, then end.

[0142] S167: If not, remove the route with the largest virtual instance number, uninstall the model in the corresponding resource, and terminate the corresponding virtual instance after going offline.

[0143] In this embodiment, the front-end page can also provide scaling policy configuration by indicators, including but not limited to CPU, memory, GPU, video memory, etc. Since the current model inference service uses a large amount of GPU computing power during the inference process, the main configuration is scaling by GPU.

[0144] Indicatively, Figure 6 As shown, Figure 6 A schematic diagram of the scaling process after configuring the index scaling policy provided in the embodiment of the present application; when the user in the present application configures the scaling policy according to the index scaling policy on the front-end page for scaling, the scaling process of the present application is as follows:

[0145] 1. Monitor resource usage of virtual instances;

[0146] 2. Determine whether the resource usage of the virtual instance meets the expansion conditions. If so, proceed to step 3. If not, determine whether the number of virtual instances is the minimum number of replicas. If so, end. If not, proceed to step 5.

[0147] 3. Determine whether the maximum number of copies has been reached. If so, end the process. If not, proceed to step 4.

[0148] 4. Create a virtual instance, deploy it according to the model inference service deployment process, and then jump to step 8;

[0149] 5. Remove the route with the largest virtual instance number;

[0150] 6. Uninstall the model and release resources;

[0151] 7. The virtual instance goes offline;

[0152] 8. End.

[0153] The automatic scaling operation of this application can be implemented according to the above scaling process, thereby improving the overall ability and response speed of the system to process requests.

[0154] The model virtualization deployment device provided in an embodiment of the present application is described below. The model virtualization deployment device described below and the model virtualization deployment method described above can be referenced to each other.

[0155] In one embodiment, Figure 7 As shown, Figure 7 A schematic diagram of the structure of a model virtualization deployment device provided in an embodiment of the present application; the present application also provides a model virtualization deployment device, which may include an information acquisition module 210, a model deployment module 220, and a routing registration module 230, specifically including the following:

[0156] An information acquisition module is used to obtain a target model selected by a user under at least one model version of at least one pre-created type, and determine an inference framework and at least one resource used when deploying the target model, wherein each resource corresponds to a different resource specification, each resource specification is configured with a k8s service, each k8s service contains multiple pods, and all pods under the same k8s service have the same specification.

[0157] The model deployment module is used to create a corresponding number of virtual instances according to the number of resources used when deploying the target model, bind the virtual instances to the specified pods in the k8s service configured under the corresponding specification resources, and then deploy the target model to the specified pods.

[0158] The routing registration module is used to receive the model loading result sent by the specified pod through the information storage component, and when it is determined that the model loading result is successfully loaded, register the routing information corresponding to the virtual instance of the target model into the model service routing table.

[0159] In the above embodiment, after obtaining the target model selected by the user under at least one model version of at least one type created in advance, the inference framework and at least one resource used when deploying the target model can be determined first, and then a corresponding number of virtual instances are created according to the number of resources used when deploying the target model, and the virtual instances are bound to the specified pods in the k8s service configured under the corresponding specification resources, and the target model is deployed to the specified pod; in this process, since the application divides the resources required for the model inference service into different specifications in advance, and each specification of resource configuration has a k8s service, each k8s service contains multiple pods, and the same k8s service All pods under the service have the same specifications, so when deploying the model, the user only needs to select resources of corresponding specifications according to the model size to deploy the model to the specified pod, thereby realizing automatic deployment of the model. In addition, after the model is deployed, the application can also receive the model loading result sent by the specified pod through the information storage component, and when it is determined that the model loading result is successfully loaded, register the routing information corresponding to the virtual instance of the target model to the model service routing table, so that the corresponding virtual instance can be found directly through the routing information registered in the model service routing table for reasoning, thereby saving a lot of manpower, avoiding waste of resources, and improving the deployment efficiency and accuracy of the model.

[0160] In one embodiment, the present application also provides a computer-readable storage medium, which stores computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the service expansion method of the collection system as described in any of the above embodiments.

[0161] In one embodiment, the present application also provides a computer device, including: one or more processors, and a memory.

[0162] The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the service expansion method of the collection system as described in any one of the above embodiments are performed.

[0163] Indicatively, Figure 8 As shown, Figure 8 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. The computer device 300 may be provided as a server. Figure 8The computer device 300 includes a processing component 302, which further includes one or more processors, and a memory resource represented by a memory 301, for storing instructions executable by the processing component 302, such as an application. The application stored in the memory 301 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 302 is configured to execute instructions to perform the model virtualization deployment method of any of the above embodiments.

[0164] The computer device 300 may further include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate based on an operating system stored in the memory 301, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.

[0165] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0166] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0167] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0168] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A model virtualization deployment method, characterized in that: The method comprises: Obtain a target model selected by a user under at least one model version of at least one pre-created type, and determine an inference framework and at least one resource used when deploying the target model, wherein each resource corresponds to a different resource specification, each resource specification is configured with a k8s service, each k8s service contains multiple pods, and all pods under the same k8s service have the same specifications; Create a corresponding number of virtual instances according to the number of resources used when deploying the target model, bind the virtual instances to the specified pods in the k8s service configured under the corresponding specification resources, and deploy the target model to the specified pods; Receive the model loading result sent by the designated pod through the information storage component, and when it is determined that the model loading result is successfully loaded, register the routing information corresponding to the virtual instance of the target model into the model service routing table.

2. The model virtualization deployment method according to claim 1, characterized in that: The determining of the reasoning framework used when deploying the target model includes: Recommending a list of inference frameworks to the user according to the model type of the target model, wherein each inference framework in the list of inference frameworks is sorted according to framework performance; The inference framework selected by the user in the inference framework list is obtained, and the inference framework is used as the inference framework used when deploying the target model.

3. The model virtualization deployment method according to claim 1, characterized in that: The determining at least one resource used when deploying the target model includes: recommending a resource specification list to the user according to the model size of the target model; Acquire at least one resource selected by a user in the resource specification list, and use the at least one resource as at least one resource used when deploying the target model; When the user selects more than two resources, the resources may be from the same cloud vendor or different cloud vendors.

4. The model virtualization deployment method according to claim 3, characterized in that: The recommending a resource specification list to the user according to the model size of the target model includes: Determining the parameter amount and optimizer parameters of the target model; Calculate the size of the video memory occupied by the parameters of the target model according to the parameter amount and the optimizer parameters; The video memory size is compared with pre-divided resources of different specifications, and a resource specification list is recommended to the user according to the comparison result.

5. The model virtualization deployment method according to claim 4, characterized in that: The process of dividing resources of different specifications includes: Determine the resources of at least one cloud vendor required for the model inference service; The resources of the at least one cloud vendor are divided into multiple specifications according to preset resource division rules.

6. The model virtualization deployment method according to claim 5, characterized in that: When the model inference service requires two cloud vendors, the client is deployed on one of the cloud vendors and the model files uploaded by the user are stored. The server is deployed on the other cloud vendor and the model files are synchronized from the cloud storage of one cloud vendor to the cloud storage of the other cloud vendor through unison two-way synchronization.

7. The model virtualization deployment method according to claim 1, characterized in that: The specified pod includes a control container and an inference framework main container; The deploying the target model to the specified pod includes: A model loading instruction is sent to the control container so that the control container can record the model information of the target model, and the model loading instruction is forwarded to the reasoning framework main container. The reasoning framework main container loads the target model according to the model loading instruction and sends the model loading result to the control container. The control container sends the model loading result to the information storage component.

8. The model virtualization deployment method according to any one of claims 1 to 7, characterized in that: The method further comprises: When it is detected that a version update is performed using the deployed target model, a health check is performed on the routing information of the deployed target model; When the health check of the deployed target model is successful, the routing information of the previous version corresponding to the deployed target model is deleted from the model service routing table, and the previous version of the model inference service is offline.

9. The model virtualization deployment method according to any one of claims 1 to 7, characterized in that: The method further comprises: During the model inference process, if it is detected that the user has enabled scheduled scaling, the scheduled scaling period and the expected number of replicas configured when the user enabled scheduled scaling are obtained; When it is detected that the scheduled expansion and contraction period has arrived, the original number of copies of the model inference instance is saved, and the original number of copies is adjusted to the expected number of copies; When it is detected that the scheduled expansion and contraction period has been exceeded, the expected number of replicas is adjusted to the original number of replicas.

10. The model virtualization deployment method according to any one of claims 1 to 7, characterized in that: The method further comprises: During the model reasoning process, monitor the resource usage of the virtual instance and determine whether the resource usage meets the preset expansion conditions; If satisfied, determine whether the number of virtual instances is the maximum number of copies; If so, then end; If not, create a new virtual instance, bind the new virtual instance to the specified pod in the k8s service configured under the corresponding specification resources, deploy the target model to the specified pod and its subsequent steps; If not, determining whether the number of virtual instances is the minimum number of copies; If so, then end; If not, after removing the route with the largest virtual instance number, uninstall the model in the corresponding resource, and terminate after the corresponding virtual instance is offline.

11. A model virtualization deployment device, characterized in that: include: An information acquisition module is used to acquire a target model selected by a user under at least one model version of at least one type created in advance, and determine an inference framework and at least one resource used when deploying the target model, wherein each resource corresponds to a different resource specification, each resource specification is configured with a k8s service, each k8s service contains multiple pods, and all pods under the same k8s service have the same specifications; A model deployment module is used to create a corresponding number of virtual instances according to the number of resources used when deploying the target model, bind the virtual instances to the specified pods in the k8s service configured under the corresponding specification resources, and then deploy the target model to the specified pods; The routing registration module is used to receive the model loading result sent by the specified pod through the information storage component, and when it is determined that the model loading result is successfully loaded, register the routing information corresponding to the virtual instance of the target model into the model service routing table.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the service expansion method of the collection system as described in any one of claims 1 to 10.

13. A computer device, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the service expansion method of the collection system as described in any one of claims 1 to 10 are performed.

Images