Model deployment method, device, electronic device and storage medium

By obtaining model configuration files, creating and deploying model services to container clusters, and using Kubernetes to manage deep learning models, it solves the efficiency, reliability and ease of use of deep learning model deployment, achieving high-performance and easy-to-manage deployment results.

CN117785224BActive Publication Date: 2024-11-26DARK MATTER ARTIFICIAL INTELLIGENT (BEIJING) TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311702833.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-12
Publication Date
2024-11-26
Estimated Expiration
2043-12-12

AI Technical Summary

Technical Problem

The prior art is difficult to efficiently and reliably deploy deep learning models in production environments, and is complex in management, affecting performance and ease of use.

Method used

By obtaining model configuration files, creating model services, and deploying them into target containers in pre-created container clusters, automated management is provided with open source container orchestration tools such as Kubernetes to provide model access interfaces for external services to access.

Benefits of technology

It realizes the deployment of deep learning models with high reliability, high performance and easy-to-management, improves the efficiency and reliability of model deployment, and provides convenient and efficient deployment services.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117785224B_ABST
    Figure CN117785224B_ABST
Patent Text Reader

Abstract

The present application provides a model deployment method, device, electronic device and storage medium, wherein the method includes: obtaining a model configuration file for model deployment; creating a model service based on the model configuration file; deploying the model service to at least one target container in a pre-created container cluster, and starting the model service in the target container, so that external services can access the model service through the pre-deployed service application architecture and the model access interface on the target container pre-assigned to the model service. The present application provides a more convenient and efficient model deployment service through containerization technology, ensuring high reliability, high performance and easy management of model deployment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model deployment method, device, electronic device and storage medium. Background Art

[0002] With the continuous development of artificial intelligence and deep learning technology, deep learning models have been widely used in many fields. Among them, deploying the trained deep learning model to the production environment is a key step in the actual application process.

[0003] Since model deployment needs to consider many factors, such as whether the deployment method is reliable, whether the performance is superior, and whether the model deployment is easy to manage, how to deploy deep learning models to ensure high reliability, high performance, and easy management is a problem that needs to be solved. Summary of the invention

[0004] The purpose of this application is to provide a model deployment method, device, electronic device and storage medium to address the deficiencies in the above-mentioned prior art, so as to solve the problem of how to deploy deep learning models to ensure high reliability, high performance and easy management.

[0005] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0006] In a first aspect, an embodiment of the present application provides a model deployment method, the method comprising:

[0007] Get the model configuration file for model deployment;

[0008] Creating a model service based on the model configuration file;

[0009] The model service is deployed to at least one target container in a pre-created container cluster, and the model service is started and run in the target container, so that external services can access the model service through the pre-deployed service application architecture and the model access interface on the target container pre-assigned to the model service.

[0010] As a possible implementation manner, before deploying the model service to at least one target container in a pre-created container cluster, the method further includes:

[0011] Obtain a predefined container configuration file, wherein the container configuration file includes image parameter information, container definition and parameters;

[0012] The container cluster is created according to the container configuration file.

[0013] As a possible implementation manner, deploying the model service to at least one target container in a pre-created container cluster includes:

[0014] Get predefined service node information;

[0015] Determine the number of nodes corresponding to the model service and the node type required by the model service according to the service node information;

[0016] At least one target container is selected from the container cluster according to the number of nodes and the node type, and the model service is deployed to each target container respectively.

[0017] As a possible implementation manner, after deploying the model service to at least one target container in a pre-created container cluster, the method further includes:

[0018] The external service obtains the service access address allocated to the model service in the process of creating the model service, and pushes the service access address to the service application architecture by using a message queue;

[0019] A model access interface is allocated to the model service, and the model access interface is pushed to the service application architecture.

[0020] As a possible implementation manner, before obtaining the service access address allocated to the model service in the process of creating the model service, the method further includes:

[0021] Obtaining predefined service mapping rules and routing rules, wherein the service mapping rules are used to represent the corresponding relationship between the gateway domain name and the model service;

[0022] Determine the gateway domain name corresponding to the model service based on the service mapping rule, and determine the routing address corresponding to the model service based on the routing rule;

[0023] The service access address of the model service is determined according to the gateway domain name and the routing address.

[0024] As a possible implementation manner, after starting and running the model service in the target container, the method further includes:

[0025] The external service sends a service inference request to the service application architecture, wherein the service inference request includes a service access address of the model service to be accessed;

[0026] The service application architecture accesses the model service to be accessed through the model access interface and based on the service access address.

[0027] As a possible implementation manner, the method further includes:

[0028] Acquire multiple probes, and add each probe to each container in the container cluster;

[0029] The target probe corresponding to the target container is used to detect the port of the target container to monitor the working status of the target container.

[0030] In a second aspect, an embodiment of the present application provides a model deployment device, the device comprising:

[0031] An acquisition module is used to obtain a model configuration file for model deployment;

[0032] A creation module, used to create a model service based on the model configuration file;

[0033] A deployment module is used to deploy the model service to at least one target container in a pre-created container cluster, and start running the model service in the target container, so that external services can access the model service through the pre-deployed service application architecture and the model access interface on the target container pre-assigned to the model service.

[0034] As a possible implementation, the creation module is further used to:

[0035] Obtain a predefined container configuration file, wherein the container configuration file includes image parameter information, container definition and parameters;

[0036] The container cluster is created according to the container configuration file.

[0037] As a possible implementation manner, the deployment module is specifically used to:

[0038] Get predefined service node information;

[0039] Determine the number of nodes corresponding to the model service and the node type required by the model service according to the service node information;

[0040] At least one target container is selected from the container cluster according to the number of nodes and the node type, and the model service is deployed to each target container respectively.

[0041] As a possible implementation manner, the deployment module is further used to:

[0042] The external service obtains the service access address allocated to the model service in the process of creating the model service, and pushes the service access address to the service application architecture by using a message queue;

[0043] A model access interface is allocated to the model service, and the model access interface is pushed to the service application architecture.

[0044] As a possible implementation manner, the deployment module is further used to:

[0045] Obtaining predefined service mapping rules and routing rules, wherein the service mapping rules are used to represent the corresponding relationship between the gateway domain name and the model service;

[0046] Determine the gateway domain name corresponding to the model service based on the service mapping rule, and determine the routing address corresponding to the model service based on the routing rule;

[0047] The service access address of the model service is determined according to the gateway domain name and the routing address.

[0048] As a possible implementation manner, the deployment module is further used to:

[0049] The external service sends a service inference request to the service application architecture, wherein the service inference request includes a service access address of the model service to be accessed;

[0050] The service application architecture accesses the model service to be accessed through the model access interface and based on the service access address.

[0051] As a possible implementation manner, the deployment module is further used to:

[0052] Acquire multiple probes, and add each probe to each container in the container cluster;

[0053] The target probe corresponding to the target container is used to detect the port of the target container to monitor the working status of the target container.

[0054] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the model deployment method as described in any one of the first aspects above.

[0055] In a fourth aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the model deployment method as described in any one of the first aspects above are executed.

[0056] According to the model deployment method, device, electronic device and storage medium of the embodiment of the present application, a model configuration file for model deployment is obtained, and a model service is created based on the model configuration file, and then the model service is deployed to at least one target container in the pre-created container cluster, and the model service is started in the target container, so that the external service accesses the model service through the pre-deployed service application architecture and the model access interface on the target container pre-assigned for the model service. Based on this, a model service is created according to the model configuration file for model deployment, and after the model service is created, a container cluster is created using an open source container orchestration tool, and the model service is deployed to the target container of the container cluster based on the containerization technology to start the model service in the target container. In addition, a model access interface is assigned to the model service so that the external service accesses the model service through the pre-deployed service application architecture and the model access interface. Thus, based on the open source container orchestration tool and containerization technology, a more convenient and efficient deep learning model deployment service is provided to users, and the high reliability, high performance and easy management of the model deployment are guaranteed, thereby also improving the efficiency and reliability of model deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 A schematic diagram of a process flow of a model deployment method provided in an embodiment of the present application is shown;

[0059] Figure 2 A schematic diagram of a process flow of a model service deployment method provided in an embodiment of the present application is shown;

[0060] Figure 3 A schematic diagram of a process flow of a push method provided by an embodiment of the present application is shown;

[0061] Figure 4 A schematic diagram of a model service access reasoning provided by an embodiment of the present application is shown;

[0062] Figure 5 A schematic diagram of a process flow of a container status monitoring method provided by an embodiment of the present application is shown;

[0063] Figure 6 A schematic diagram of the structure of a model deployment device provided in an embodiment of the present application is shown;

[0064] Figure 7A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0065] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0066] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0067] In order to enable those skilled in the art to use the content of this application, the following implementation is provided in conjunction with the specific application scenario "model service". For those skilled in the art, the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application. Although this application is mainly described around the model deployment method, it should be understood that this is only an exemplary embodiment.

[0068] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0069] Based on the containerization technology, applications and services can be packaged into lightweight and portable containers, thereby achieving more efficient and flexible deployment and management. The embodiment of the present application provides a method for model deployment based on an open source container orchestration tool (Kubernetes, k8s), wherein the open source container orchestration tool k8s can automatically manage operations such as container deployment, expansion, upgrade and fault recovery, thereby providing users with a more convenient and efficient deep learning model deployment service, and ensuring high reliability, high performance and easy management of model deployment, thereby also improving the efficiency and reliability of model deployment.

[0070] Figure 1 A schematic diagram of a model deployment method provided in an embodiment of the present application is shown. Figure 1 As shown, the method specifically comprises the following steps:

[0071] S101. Obtain a model configuration file for model deployment.

[0072] Optionally, for deep learning models, model deployment refers to the process of running a trained model in a specific environment. Its purpose is to transform the deep learning model from the research stage into practical applications, so that the deep learning model can provide services for users or solve practical problems. Deep learning models are usually written by some frameworks. Due to the limitations of framework scale and dependent environment, these frameworks are not suitable for installation in production environments. In addition, since the structure of deep learning models is usually large, a large amount of computing power is required to meet the needs of real-time operation. Therefore, the model structure and model parameters of the deep learning model are converted into a model configuration file to use containerization technology to provide a more convenient and efficient model deployment service based on the model configuration file.

[0073] Exemplarily, the model configuration file includes the user's configuration information for the deployed model, and the configuration information includes but is not limited to model configuration information, image configuration information, resource configuration information, and service configuration information.

[0074] S102. Create a model service based on the model configuration file.

[0075] Optionally, model serving is a technology that deploys machine learning or deep learning models into a production environment and provides services through an application programming interface (API) to provide intelligent prediction or decision support for users or applications.

[0076] Exemplarily, the model service is created according to the model configuration information, image configuration information, resource configuration information, and service configuration information of the configured deep learning model, so as to deploy the deep learning model to the production environment for operation. In addition, to achieve service access after the model service is deployed, the service address type in the open source container orchestration tool k8s, namely ClusterIP, is used when creating the model service, and the Ingress gateway is used for proxy. Ingress is an abstraction of the reverse proxy by the open source container orchestration tool k8s. Its function is to define the rules for how requests are forwarded to the model service, and in order to avoid multiple resolutions when using the domain name server, the Ingress domain name used to access the model service is the same.

[0077] S103. Deploy the model service to at least one target container in a pre-created container cluster, and start running the model service in the target container, so that external services can access the model service through the pre-deployed service application architecture and the model access interface on the target container pre-assigned for the model service.

[0078] Optionally, the embodiment of the present application selects the container service type to build a deployment environment for model service deployment. The container can run on any machine equipped with a container runtime environment and provide an independent resource space when the program is running. Therefore, after the created model service is deployed to at least one target container in the pre-created container cluster, it can be guaranteed that the model service can run normally in the target container after it is started.

[0079] In addition, when the number of containers in a container cluster reaches a certain scale, the open source container orchestration tool k8s is needed to schedule and manage the containers in the container cluster.

[0080] Therefore, according to a model deployment method provided in an embodiment of the present application, a model configuration file for model deployment is obtained, and a model service is created based on the model configuration file, and then the model service is deployed to at least one target container in a pre-created container cluster, and the model service is started in the target container, so that the external service accesses the model service through the pre-deployed service application architecture and the model access interface on the target container pre-assigned for the model service. Based on this, a model service is created according to the model configuration file for model deployment, and after the model service is created, a container cluster is created using an open source container orchestration tool, and the model service is deployed to the target container of the container cluster based on containerization technology to start running the model service in the target container. In addition, a model access interface is assigned to the model service so that the external service accesses the model service through the pre-deployed service application architecture and the model access interface. Thus, based on the open source container orchestration tool and containerization technology, a more convenient and efficient deep learning model deployment service is provided to users, and the high reliability, high performance and easy management of the model deployment are guaranteed, thereby also improving the efficiency and reliability of model deployment.

[0081] As a possible implementation manner, before the above step S103 deploys the model service to at least one target container in the pre-created container cluster, the method further includes: obtaining a pre-defined container configuration file, and creating a container cluster according to the container configuration file.

[0082] Exemplarily, the container configuration file includes but is not limited to image parameter information, container definition and parameters. After the model service is successfully created, click Go Online, and then create the container, where service launch refers to deploying the developed service to the formal environment through release, allowing users to access it through the specified service access portal. Deploy the open source container orchestration tool Kubernetes on the electronic device running the model deployment. The open source container orchestration tool Kubernetes can learn the container creation requirements by reading the container configuration file. And because the open source container orchestration tool Kubernetes has the function of automatic boxing, the resource configuration requirements of the container corresponding to the operating environment can be determined according to the pre-defined container configuration file, and then use the automatic boxing function of the open source container orchestration tool Kubernetes to automatically deploy containers based on the resource configuration requirements of the container for the application operating environment to create a container cluster.

[0083] Based on this, since each container in the container cluster is isolated from each other and the processes between containers will not affect each other, it is ensured that after the model service is deployed to at least one target container in the pre-created container cluster, the model services in each target container can run normally in the same production environment.

[0084] As a possible implementation, Figure 2 As shown, the above step S103 deploys the model service to at least one target container in the pre-created container cluster, and specifically includes the following steps:

[0085] S201: Obtain predefined service node information.

[0086] Exemplarily, in the process of creating a model service, in addition to setting the environment variable information of the image to facilitate the specification of some variable information required during the operation of the image, the service node information can also be pre-defined. The service node information may include the number of nodes of the service, the node type required for the service, etc.

[0087] S202: Determine the number of nodes corresponding to the model service and the node type required for the model service according to the service node information.

[0088] Exemplarily, the number of nodes of the created model service is determined based on predefined service node information, that is, whether the created model service corresponds to a single node or multiple nodes, wherein the number of nodes corresponds to the number of target containers, that is, whether the model service is deployed in one target container or multiple target containers.

[0089] Exemplarily, the node type required for the model service is, for example, a central processing unit (CPU) or a graphics processing unit (GPU). Determining the node type required for the created model service based on the pre-defined service node information is to select the deployment environment corresponding to the model service, that is, considering the selection and configuration of the container hardware device.

[0090] S203: Select at least one target container from the container cluster according to the number of nodes and the node type, and deploy the model service to each target container respectively.

[0091] Exemplarily, if the number of nodes corresponding to the determined model service is 1, that is, a single node, the model service is deployed to a target container in the container cluster; if the number of nodes corresponding to the determined model service is greater than 1, that is, multiple nodes, for example, 3, the model service is deployed to three target containers in the container cluster.

[0092] It should be noted that the choice of the relationship between containers and nodes affects the performance and stability of the container. There are also two different forms of the relationship between containers and nodes. One form is that container A is on node node1, and container B is on node node2. The two containers transmit data and information through the network. The other form is that container A and container B are both on node node1, and the containers transmit data and information through memory or non-physical networks. In the first form above, container A has the computing power of node node1, and container B has the computing power of node node2, so the overall computing power is too strong compared to the second form. In the second form, compared with the first form, the data exchange between container A and container B does not go through the physical network, and the operation efficiency is higher.

[0093] Based on this, the target container and the number of target containers for model service deployment are determined by the number of nodes and node types, and the deployment relationship between the appropriate container and node is selected according to actual needs, and then the model service is deployed to the corresponding target container according to the selected deployment relationship.

[0094] As a possible implementation, Figure 3 As shown, after the above step S103 deploys the model service to at least one target container in the pre-created container cluster, the method further includes the following steps:

[0095] S301. The external service obtains the service access address allocated to the model service during the process of creating the model service, and pushes the service access address to the service application architecture using a message queue.

[0096] For example, in the process of creating a model service, a unique service access address will be generated for the model service. Users can access the model service and implement reasoning based on the service access address. It should be noted that the service access address will not be affected by the deployment of the model service. In other words, if the same model service is deployed multiple times, the service access address corresponding to the model service will not change, so that it can be used stably after one creation. Among them, the service access address can be composed of the reasoning gateway address and the routing address.

[0097] S302: Allocate a model access interface for the model service, and push the model access interface to the service application architecture.

[0098] For example, the model access interface is the entry point for users to access the model service. After the container is successfully created, the routing information of the service access address will be sent to the Kafka message queue and pushed to the service application framework Nacos. At the same time, the model access interface assigned to the model service will also be pushed to the service application framework Nacos, so that external services can enter the access channel through the model access interface based on the service application framework Nacos and use the service access address to query the corresponding model service. Among them, the service application framework Nacos is a platform for dynamic service registration and discovery, configuration management, and service management that is easier to build cloud native applications.

[0099] Based on this, through real-time monitoring of the container, when the container is successfully created, the service access address and model access interface are pushed to the service application architecture, so as to access the model service based on the service access address and model access interface to achieve reasoning.

[0100] As a possible implementation, before obtaining the service access address allocated to the model service in the process of creating the model service in step S301, the method further includes:

[0101] Obtain predefined service mapping rules and routing rules, determine the gateway domain name corresponding to the model service based on the service mapping rules, determine the routing address corresponding to the model service based on the routing rules, and determine the service access address of the model service based on the gateway domain name and routing address.

[0102] For example, when creating the ingress resource of the open source container orchestration tool k8s, the configured service access address is "ingress domain name + UUID", where the ingress domain name is an object in the creation of the open source container orchestration tool k8s, and the model service can be accessed through the domain name, and the routing rule UUID is the universally unique identifier (UUID) corresponding to the model service. The service mapping rule is used to represent the correspondence between the gateway domain name and the model service. Since the service access address corresponding to the model service will not change, the routing universal unique identifier UUID is also fixed, so the corresponding model service can be determined through the gateway domain name ingress.

[0103] Based on this, by determining the gateway domain name and routing address corresponding to the model service, the service access address corresponding to the model service is determined. Since the service access address will not change and is not affected by the deployment of the model service, a service access address is created once for each model service and can be used stably.

[0104] As a possible implementation, after the above step S103 starts running the model service in the target container, the method further includes:

[0105] The external service sends a service inference request to the service application architecture, where the service inference request includes a service access address of the model service to be accessed; the service application architecture accesses the model service to be accessed through the model access interface and based on the service access address.

[0106] For example, Figure 4 As shown in the figure, based on the previous model service creation, deployment, scheduling, container cluster creation and container status monitoring, the model service deployment has been realized, and the service application architecture Nacos has also mastered the service access address and model access interface corresponding to each model service. On this basis, the external service can send a service reasoning request to the service application architecture Nacos. Specifically, the external service does not send a service reasoning request directly to the service application architecture Nacos, but pulls the routing address and service access address from the service application architecture Nacos in real time based on the reasoning gateway, so as to realize the forwarding of the service reasoning request, and then accesses the model service to be accessed through the model access interface and based on the service access address to perform service reasoning. Specifically, in the service reasoning process, the relevant information of each service reasoning request is recorded in real time, such as input parameters, output parameters, reasoning execution time, etc.

[0107] Based on this, by reasoning about the model service, we provide authentication, flow control, and recording functions for the model service to ensure the security and integrity of the model service.

[0108] As a possible implementation, Figure 5 As shown, the method also includes:

[0109] S501: Acquire multiple probes, and add each probe to each container in the container cluster.

[0110] For example, a container cluster is composed of multiple containers, each of which is equivalent to an application. During the operation of an application, the program may crash due to some unexpected situation. Therefore, it is particularly important to use probes to monitor the stability of the container state, ensure that the model service will not have problems during operation, and restart the model service after a problem occurs.

[0111] For example, in the open source container orchestration tool k8s, a large number of asynchronous mechanisms and decoupling of various object relationship designs are adopted. When the number of containers increases or decreases, or the application version changes, it is impossible to guarantee that the container configuration will always be refreshed in time. Usually, only the newly added containers can complete their own initialization, and some existing old containers will be deleted immediately, which eventually causes the model service to be temporarily unavailable. Therefore, some survival probes are added to the open source container orchestration tool k8s, and each probe is added to each container in the container cluster to monitor the working status of each container.

[0112] S502: Use a target probe corresponding to the target container to detect the port of the target container and monitor the working status of the target container.

[0113] For example, in the open source container orchestration tool k8s, Pod is the smallest deployable computing unit, a collection of containers, that is, a container cluster, sharing the same network namespace, storage resources, etc. Pods in the open source container orchestration tool k8s mainly have the following states: pending, running, successful, failed, and unknown. Among them, Pending means that the container image of the Pod has not been pulled, or the node resources required by the Pod have not been met. At this time, the Pod is not schedulable. Running means that the Pod has been scheduled to the node and all containers have been created, and at least one container is still running or in the process of starting. Succeeded means that all containers in the Pod have terminated normally and will not be restarted. Failed means that at least one container in the Pod has exited with a non-zero status. Unknown means that the status of the Pod cannot be obtained.

[0114] In addition to the above states, Pod also has some special states that are used to record some detailed information of Pod, such as whether Pod is being scheduled, whether the container image is pulled successfully, etc. Exemplarily, Pod also has the following special states: PodScheduled, Containers Ready, Initialized, Ready, Create Container Error, ContainerCreating, etc. Among them, Pod Scheduled indicates whether the Pod has been scheduled to the node, Containers Ready indicates whether all containers in the Pod are ready, Initialized indicates whether all containers in the Pod have been initialized, Ready indicates whether the Pod is ready, that is, all containers have been started and can receive traffic, CreateContainer Error indicates that the container creation failed, and Container Creating indicates that the container is being created.

[0115] It should be noted that the target probe corresponding to the target container is used here to detect the port of the target container and monitor the working status of the target container, mainly monitoring the Ready status of the container, that is, whether the container has been started and can receive traffic, so as to ensure that the model service is truly available after the successful launch.

[0116] Based on this, the working status of the container is monitored. During the monitoring process, by using probes to detect the port of the container and monitoring the Ready and other states of the container, while ensuring that the model service is truly available after the successful launch, the working status of the container can be grasped in real time, the stability of the container status can be monitored, and no problems will occur during the operation of the model service.

[0117] Therefore, the embodiment of the present application provides a method for deep learning model deployment based on Kubernetes, which realizes fast and efficient deployment and access to deep learning models by configuring model services, resource scheduling, container creation, dynamic routing and other operations, and realizes the efficiency, scalability, flexibility, security and ease of use of model deployment. Among them, the efficiency is reflected in the use of Kubernetes's efficient scheduling characteristics and containerization technology, which can quickly deploy and manage multiple deep learning model services, improving the efficiency and stability of model deployment. Scalability is reflected in the fact that the Kubernetes cluster has good horizontal expansion capabilities, and can dynamically expand or shrink services according to business needs to meet the needs of large-scale data processing and computing. Flexibility is reflected in the provision of rich configuration options and service deployment management tools, and users can flexibly configure and adjust according to their own business needs. Security is reflected in the fact that the inference service provides authentication, current limiting and recording functions to ensure the security and integrity of the service. Ease of use is reflected in the fact that the model deployment operation process is simple and easy to understand, allowing users to quickly get started and convenient for users to deploy and manage deep learning models.

[0118] Based on the same inventive concept, a model deployment device corresponding to the model deployment method is also provided in the embodiment of the present application. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned model deployment method in the embodiment of the present application, the implementation of the model deployment device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0119] Reference Figure 6 As shown, it is a structural diagram of a model deployment device provided in an embodiment of the present application, wherein the model deployment 600 includes: an acquisition module 601, a creation module 602, and a deployment module 603, wherein:

[0120] An acquisition module 601 is used to acquire a model configuration file for model deployment;

[0121] A creation module 602, used to create a model service based on a model configuration file;

[0122] The deployment module 603 is used to deploy the model service to at least one target container in a pre-created container cluster, and start running the model service in the target container, so that external services can access the model service through the pre-deployed service application architecture and the model access interface on the target container pre-assigned to the model service.

[0123] Thus, according to the model deployment device of the embodiment of the present application, a model configuration file for model deployment is obtained, and a model service is created based on the model configuration file, and then the model service is deployed to at least one target container in the pre-created container cluster, and the model service is started in the target container, so that the external service accesses the model service through the pre-deployed service application architecture and the model access interface on the target container pre-assigned for the model service. Based on this, a model service is created according to the model configuration file for model deployment, and after the model service is created, a container cluster is created using an open source container orchestration tool, and the model service is deployed to the target container of the container cluster based on the containerization technology to start the model service in the target container. In addition, a model access interface is assigned to the model service so that the external service accesses the model service through the pre-deployed service application architecture and the model access interface. Thus, based on the open source container orchestration tool and containerization technology, a more convenient and efficient deep learning model deployment service is provided to users, and the high reliability, high performance and easy management of the model deployment are guaranteed, thereby also improving the efficiency and reliability of the model deployment.

[0124] In a possible implementation manner, the creation module 602 is further used to:

[0125] Obtain a predefined container configuration file, which includes image parameter information, container definition and parameters;

[0126] Create a container cluster based on the container configuration file.

[0127] In a possible implementation manner, the deployment module 603 is specifically used to:

[0128] Get predefined service node information;

[0129] Determine the number of nodes corresponding to the model service and the node type required for the model service based on the service node information;

[0130] At least one target container is selected from the container cluster based on the number of nodes and node type, and the model service is deployed to each target container respectively.

[0131] In a possible implementation manner, the deployment module 603 is further used to:

[0132] Get the pre-defined service mapping rules and routing rules. The service mapping rules are used to indicate the correspondence between the gateway domain name and the model service.

[0133] Determine the gateway domain name corresponding to the model service based on the service mapping rules, and determine the routing address corresponding to the model service based on the routing rules;

[0134] Determine the service access address of the model service based on the gateway domain name and routing address.

[0135] In a possible implementation manner, the deployment module 603 is further used to:

[0136] The external service sends a service inference request to the service application architecture, where the service inference request includes the service access address of the model service to be accessed;

[0137] The service application architecture accesses the model service to be accessed through the model access interface and based on the service access address.

[0138] In a possible implementation manner, the deployment module 603 is further used to:

[0139] Get multiple probes and add each probe to each container in the container cluster;

[0140] The target probe corresponding to the target container is used to detect the port of the target container and monitor the working status of the target container.

[0141] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0142] The present application embodiment also provides an electronic device 700, such as Figure 7 As shown, it is a schematic diagram of the structure of an electronic device 700 provided in an embodiment of the present application, including: a processor 701, a memory 702, and optionally, a bus 703. The memory 702 stores machine-readable instructions executable by the processor 701. When the electronic device 700 is running, the processor 701 communicates with the memory 702 through the bus 703. When the machine-readable instructions are executed by the processor 701, the steps of the model deployment method described in any one of the above items are performed.

[0143] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the model deployment method as described in any one of the above items are executed.

[0144] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0145] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or part of the technical solution that contributes to the prior art or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.

[0146] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A model deployment method, characterized in that: include: Get the model configuration file for model deployment; Creating a model service based on the model configuration file; Deploy the model service to at least one target container in a pre-created container cluster, and start running the model service in the target container, so that external services can access the model service through the pre-deployed service application architecture and the model access interface on the target container pre-assigned to the model service; After starting and running the model service in the target container, the method further includes: An external service sends a service inference request to the service application architecture, wherein the service inference request includes a service access address of a model service to be accessed, wherein the service access address is determined based on a gateway domain name and a routing address, wherein the gateway domain name is determined based on a service mapping rule, wherein the routing address is determined based on a routing rule, wherein the service mapping rule is used to indicate a correspondence between a gateway domain name and a model service, and wherein the routing rule is a universal unique identification code corresponding to the model service; The service application architecture accesses the model service to be accessed through the model access interface and based on the service access address; The deploying the model service to at least one target container in a pre-created container cluster includes: According to the deployment relationship between the selected container and the node, the model service is deployed into the target container according to the deployment relationship.

2. The method according to claim 1, characterized in that Before deploying the model service to at least one target container in a pre-created container cluster, the method further includes: Obtain a predefined container configuration file, wherein the container configuration file includes image parameter information, container definition and parameters; The container cluster is created according to the container configuration file.

3. The method according to claim 1, characterized in that: The deploying the model service to at least one target container in a pre-created container cluster includes: Get predefined service node information; Determine the number of nodes corresponding to the model service and the node type required by the model service according to the service node information; At least one target container is selected from the container cluster according to the number of nodes and the node type, and the model service is deployed to each target container respectively.

4. The method according to claim 1, characterized in that: After deploying the model service to at least one target container in a pre-created container cluster, the method further includes: The external service obtains the service access address allocated to the model service in the process of creating the model service, and pushes the service access address to the service application architecture by using a message queue; A model access interface is allocated to the model service, and the model access interface is pushed to the service application architecture.

5. The method according to claim 4, characterized in that Before obtaining the service access address allocated to the model service in the process of creating the model service, the method further includes: Obtaining predefined service mapping rules and routing rules, wherein the service mapping rules are used to represent the corresponding relationship between the gateway domain name and the model service; Determine the gateway domain name corresponding to the model service based on the service mapping rule, and determine the routing address corresponding to the model service based on the routing rule; The service access address of the model service is determined according to the gateway domain name and the routing address.

6. The method according to claim 1, characterized in that The method further comprises: Acquire multiple probes, and add each probe to each container in the container cluster; The target probe corresponding to the target container is used to detect the port of the target container to monitor the working status of the target container.

7. A model deployment device, characterized in that: include: An acquisition module is used to obtain a model configuration file for model deployment; A creation module, used to create a model service based on the model configuration file; A deployment module, used to deploy the model service to at least one target container in a pre-created container cluster, and start running the model service in the target container, so that external services can access the model service through the pre-deployed service application architecture and the model access interface on the target container pre-assigned to the model service; The deployment module is further used for sending a service inference request from an external service to the service application architecture, wherein the service inference request includes a service access address of the model service to be accessed, and the service access address is determined according to a gateway domain name and a routing address, wherein the gateway domain name is determined based on a service mapping rule, and the routing address is determined based on a routing rule, wherein the service mapping rule is used to indicate a corresponding relationship between a gateway domain name and a model service, and the routing rule is a universal unique identification code corresponding to the model service; The service application architecture accesses the model service to be accessed through the model access interface and based on the service access address; The deployment module is specifically used to deploy the model service into the target container according to the deployment relationship between the selected container and the node.

8. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor executes the machine-readable instructions to perform the steps of the model deployment method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the model deployment method as described in any one of claims 1 to 6.