Model deployment method and device in multi-machine environment, equipment and storage medium
By combining the Kubernetes platform and MindIE inference engine with the StatefulSet controller, model services in multi-machine environments are deployed automatically, solving the problem of low efficiency in traditional deployment methods and achieving efficient and flexible model deployment and management.
Patent Information
- Application Number
- CN202511941293.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-22
- Publication Date
- 2026-02-13
AI Technical Summary
In a multi-machine environment, traditional model deployment methods have low automation and scalability. Manually configuring network and storage resources is time-consuming, labor-intensive, and difficult to guarantee consistency and reliability.
By leveraging the Kubernetes platform and the MindIE inference engine, a target model deployment package is built through the StatefulSet controller to automatically deploy model services. The Helm Chart tool is used to manage model deployment package templates, thereby achieving automated deployment, scaling, and management of model services.
It improves the efficiency of model deployment in multi-machine environments, reduces manual intervention, lowers operation and maintenance costs, supports dynamic adjustment of resource configuration, adapts to different scenario requirements, and maintains good horizontal scalability.
Smart Images
Figure CN121530864A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of cloud computing, and in particular to a method, apparatus, device and storage medium for deploying models in a multi-machine environment. Background Technology
[0002] Containerization has become the standard practice for deploying machine learning models, simplifying software distribution and deployment by encapsulating applications and their dependencies. However, when it comes to distributed deployments in multi-machine environments, traditional deployment methods are inefficient in terms of automation and scalability. For example, manually configuring network, storage, and other resources is not only time-consuming and labor-intensive, but also makes it difficult to guarantee consistency and reliability across different nodes.
[0003] In conclusion, improving the efficiency of model deployment in multi-machine environments is an urgent problem to be solved. Summary of the Invention
[0004] In view of this, the purpose of this invention is to provide a model deployment method, apparatus, device, and storage medium in a multi-machine environment, which can improve the model deployment efficiency in a multi-machine environment. The specific solution is as follows:
[0005] Firstly, this application discloses a model deployment method in a multi-machine environment, including:
[0006] Obtain the initial configuration file corresponding to each target deployment node, and construct the target model deployment package based on all the initial configuration files; the initial configuration file includes the container IP and RoCE network interface IP; the target model deployment package includes the common variables, container startup method and model service type in each of the initial configuration files;
[0007] The target model deployment platform utilizes a StatefulSet controller to deploy the target model in the target model deployment package to each of the target deployment nodes; the target model deployment platform is a Kubernetes platform.
[0008] The target model is configured on each target deployment node using the target model deployment package to obtain the target configuration file corresponding to each target deployment node, so as to use the MindIE inference engine to call the inference service of the target model on the corresponding target deployment node based on the target configuration file.
[0009] Optionally, the target configuration file is a node configuration file that meets the calling requirements of the MindIE inference engine. The target configuration file includes environment variable information, node device information, master node IP, container IP, and RoCE network port IP.
[0010] Optionally, the step of building the target model deployment package based on all the initial configuration files includes:
[0011] The target model deployment package is constructed using the target template based on all the initial configuration files.
[0012] The target template is a model deployment package template built using the Helm Chart tool of the target model deployment platform.
[0013] Optionally, the step of deploying the target model in the target model deployment package to each of the target deployment nodes using the StatefulSet controller through the target model deployment platform includes:
[0014] The target model deployment platform utilizes the StatefulSet controller to launch the corresponding target container on each target deployment node based on the container startup method in the target model deployment package, so as to deploy the target model in the target model deployment package on each target deployment node, and determine the first target deployment node among all the target deployment nodes as the master node to determine the master node IP.
[0015] Optionally, configuring the target model of each target deployment node using the target model deployment package to obtain the target configuration file corresponding to each target deployment node includes:
[0016] On each of the target deployment nodes, the environment variables corresponding to the target model, the node device information of the target deployment node, and the master node IP are configured using the ConfigMap in the target model deployment package;
[0017] The StatefulSet controller in the target model deployment package is used to name the target container of the target deployment node based on the target naming mechanism to determine the container name of the target container;
[0018] Obtain the corresponding container IP based on the container name;
[0019] The target configuration file corresponding to the target deployment node is obtained based on the container IP and RoCE network port IP corresponding to the target deployment node, the environment variables corresponding to the target model, the node device information of the target deployment node, and the master node IP.
[0020] Optionally, obtaining the corresponding container IP based on the container name includes:
[0021] Configure a headless service for each of the target deployment nodes so that the target containers on each target deployment node can resolve container IPs using the headless service and the container name.
[0022] Optionally, the step of launching the corresponding target container on each of the target deployment nodes through the target model deployment platform using the StatefulSet controller based on the container startup method in the target model deployment package includes:
[0023] The target model deployment platform utilizes the StatefulSet controller to launch the corresponding target containers on each target deployment node based on the container startup method and preset node anti-affinity strategy in the target model deployment package.
[0024] Secondly, this application discloses a model deployment device in a multi-machine environment, comprising:
[0025] The deployment package building module is used to obtain the initial configuration files corresponding to each target deployment node and build the target model deployment package based on all the initial configuration files; the initial configuration files include container IP and RoCE network interface IP; the target model deployment package includes common variables, container startup methods and model service types in each of the initial configuration files;
[0026] The model deployment module is used to deploy the target model in the target model deployment package to each of the target deployment nodes through the StatefulSet controller of the target model deployment platform; the target model deployment platform is the Kubernetes platform.
[0027] The service invocation module is used to configure the target model of each target deployment node using the target model deployment package to obtain the target configuration file corresponding to each target deployment node, so as to use the MindIE inference engine to invoke the inference service of the target model on the corresponding target deployment node based on the target configuration file.
[0028] Thirdly, this application discloses an electronic device, including:
[0029] Memory, used to store computer programs;
[0030] A processor is used to execute the computer program to implement the aforementioned model deployment method in a multi-machine environment.
[0031] Fourthly, this application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned model deployment method in a multi-machine environment.
[0032] In this application, when deploying a model in a multi-machine environment, the initial configuration files corresponding to each target deployment node are obtained, and a target model deployment package is constructed based on all the initial configuration files. The initial configuration files include container IPs and RoCE network interface IPs. The target model deployment package includes common variables, container startup methods, and model service types from each of the initial configuration files. The target model is deployed on each target deployment node using a StatefulSet controller through a target model deployment platform. The target model deployment platform is a Kubernetes platform. The target model on each target deployment node is configured using the target model deployment package to obtain the target configuration files corresponding to each target deployment node. The MindIE inference engine then calls the inference service of the target model on the corresponding target deployment node based on the target configuration files. Therefore, this application constructs a target model deployment package using the initial configuration files of each target deployment node and combines the Kubernetes platform with the MindIE inference framework. This transforms the key part of container deployment into Kubernetes deployment and utilizes the Kubernetes StatefulSet controller to achieve automated deployment, expansion, and management of model services, reducing manual intervention, simplifying operation processes, lowering maintenance costs, and improving model deployment efficiency in a multi-machine environment. Furthermore, the use of the target model deployment package supports dynamic adjustment of resource configuration and service scale according to actual business needs, adapting to application requirements in different scenarios, while maintaining good horizontal scalability, facilitating future function upgrades and technology evolution, and greatly improving the flexibility and scalability of model deployment in multi-machine environments. Attached Figure Description
[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0034] Figure 1 This is a flowchart of a model deployment method in a multi-machine environment disclosed in this application;
[0035] Figure 2 This is a schematic diagram of a specific target model deployment package disclosed in this application;
[0036] Figure 3 This is a schematic diagram of a specific deployment node architecture disclosed in this application;
[0037] Figure 4This is a schematic diagram of a model deployment device in a multi-machine environment disclosed in this application;
[0038] Figure 5 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation
[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0040] Containerization technology has become the standard practice for deploying machine learning models, simplifying software distribution and deployment processes by encapsulating applications and their dependencies. However, when it comes to distributed deployment in multi-machine environments, traditional deployment methods are inefficient in terms of automation and scalability. For example, manually configuring network, storage, and other resources is not only time-consuming and labor-intensive, but also makes it difficult to guarantee consistency and reliability across different nodes. To address these technical problems, this application discloses a model deployment method in a multi-machine environment, which can improve the efficiency of model deployment in such environments.
[0041] See Figure 1 As shown in the figure, an embodiment of the present invention discloses a model deployment method in a multi-machine environment, including:
[0042] Step S11: Obtain the initial configuration file corresponding to each target deployment node, and construct the target model deployment package based on all the initial configuration files; the initial configuration file includes the container IP and RoCE network interface IP; the target model deployment package includes the common variables, container startup method and model service type in each of the initial configuration files.
[0043] In this embodiment, when deploying Ascend 910B series GPU devices across multiple machines based on the Kubernetes platform and the MindIE inference framework, in order to convert the key part of container deployment to Kubernetes deployment, it is necessary to prepare initial configuration files in advance for the container IPs and RoCE network interface IPs (device part) of multiple nodes. The container IPs and RoCE IPs must correspond one-to-one, and the format is as follows:
[0044] {"server_count": "2",
[0045] "server_list": [
[0046] {"device": [
[0047] {"device_id": "0",
[0048] "device_ip": "...",
[0049] "rank_id": "0"},
[0050] { "device_id": "1",
[0051] "device_ip": "...",
[0052] "rank_id": "1"}, ...
[0054] { "device_id": "7",
[0055] "device_ip": "...",
[0056] "rank_id": "7"},],
[0057] "server_id": "...",
[0058] "container_ip": "..."},
[0059] { "device": [
[0060] { "device_id": "0",
[0061] "device_ip": "...",
[0062] "rank_id": "8"},
[0063] { "device_id": "1",
[0064] "device_ip": "...",
[0065] "rank_id": "9"}, ...
[0067] { "device_id": "7",
[0068] "device_ip": "...",
[0069] "rank_id": "15"},],
[0070] "server_id": "...",
[0071] "container_ip": "..."},],
[0072] "status": "completed",
[0073] "version": "1.0"};
[0074] The RoCE network interface IP information refers to the IP address of the physical machine's RoCE network interface, which can be written as a parameter to the k8sconfigmap. The container IP address changes with each pod restart and needs to be dynamically obtained after the container starts. In Kubernetes, the container name started by a StatefulSet can be set to `statefulsetname-{0..n}`. After the container starts, its IP address can be obtained by resolving the container name and then entered into the initial configuration file. It's understandable that using the Kubernetes platform to daemonize containers allows for automatic restart of failed instances, ensuring service continuity, and all operations can be implemented through the Kubernetes API, facilitating operation and maintenance management.
[0075] In this embodiment, the step of constructing the target model deployment package based on all the initial configuration files may specifically include: constructing the target model deployment package based on all the initial configuration files using a target template; wherein, the target template is a model deployment package template constructed using the Helm Chart tool of the target model deployment platform. In a specific implementation, the format of the target model deployment package is as follows: Figure 2 As shown, common variables are extracted from all initial configuration files into values.yaml, such as the RoCE network interface IP and container image version. templates / configmap.yaml defines two parts: MindIE's configuration settings and the startup script. The startup script contains configuration files that generate multi-node container IPs, RoCE network interface IPs (device part), etc. It resolves the pod IP based on the podname domain name and generates multi-node configuration files to start the MindIE service. templates / statefulset.yaml defines the container startup method, including: configuring node anti-affinity policies, such as preventing multiple containers from being deployed on the same node; mounting the configmap into the container and executing the startup script; mounting the model directory into the container; and defining the number of replicas, configured as 2 for dual-node deployment. Figure 3As shown, `templates / service.yaml` defines two types of services: headless services and ClusterIP services. Headless services are used for two pods (e.g., MindIE-Pod-0 and MindIE-Pod-1) to obtain their IP addresses via podname. ClusterIP services are used to redirect traffic to the master node, MindIE-Pod-0. It should be noted that this embodiment can use the HelmChart management model deployment package template, which includes definitions for Service, ConfigMap, and StatefulSet, to achieve one-click deployment and scaling across multiple nodes to adapt to different concurrency requirements.
[0076] Step S12: Deploy the target model in the target model deployment package to each of the target deployment nodes using the StatefulSet controller through the target model deployment platform; the target model deployment platform is the Kubernetes platform.
[0077] In this embodiment, as Figure 3 As shown, the target model deployment platform utilizes the StatefulSet controller to deploy the target model in the target model deployment package to each target deployment node. This includes: using the StatefulSet controller, the target model deployment platform starts the corresponding target container on each target deployment node based on the container startup method in the target model deployment package, thus deploying the target model in the target model deployment package to each target deployment node, and determining the first target deployment node among all target deployment nodes as the master node to determine the master node IP. Specifically, the Helm tool can be used to render the chart deployment directory, and the kubectl apply command can be executed to deploy the model on the Kubernetes cluster. Specifically, starting the corresponding target container on each target deployment node using the StatefulSet controller based on the container startup method in the target model deployment package can include: using the StatefulSet controller, the target model deployment platform starts the corresponding target container on each target deployment node based on the container startup method in the target model deployment package and a preset node anti-affinity strategy. The target model deployment platform is the Kubernetes platform, and the target model can be a large-scale language model service, such as the DeepSeek-R1 full-fledged model. Understandably, the use of a pre-defined node anti-affinity strategy ensures that only one Pod instance runs on each physical node, thereby improving resource isolation and stability.
[0078] Step S13: Configure the target model of each target deployment node using the target model deployment package to obtain the target configuration file corresponding to each target deployment node, so as to use the MindIE inference engine to call the inference service of the target model on the corresponding target deployment node based on the target configuration file.
[0079] In this embodiment, when the MindIE inference service starts, the configuration files of each node must be consistent with those of the master node, mainly including the master node IP configuration, model name, model path, etc. The target model deployment package is used to configure the target model of each target deployment node to obtain the target configuration file corresponding to each target deployment node. Specifically, this may include: configuring the environment variables corresponding to the target model, the node device information of the target deployment node, and the master node IP on each target deployment node using the ConfigMap in the target model deployment package; naming the target container of the target deployment node based on the target naming mechanism using the StatefulSet controller in the target model deployment package to determine the container name of the target container; obtaining the corresponding container IP based on the container name; and obtaining the target configuration file corresponding to the target deployment node based on the container IP and RoCE network interface IP corresponding to the target deployment node, the environment variables corresponding to the target model, the node device information of the target deployment node, and the master node IP. Specifically, the target configuration file is a node configuration file that meets the requirements of the MindIE inference engine, and the target configuration file includes environment variable information, node device information, master node IP, container IP, and RoCE network interface IP. The first container serves as the master node. The master node container IP can be obtained by resolving the `statefulsetname-0` container. Other information is relatively fixed and can be pre-configured in the configmap. Environment variables can be configured in the `statefulsetenv`. A specific example of environment variable configuration is as follows:
[0080] # Service-oriented multi-machine support variables
[0081] export MIES_CONTAINER_IP=container IP address
[0082] export RANK_TABLE_FILE=rank_table_file.json path
[0083] # Log printing (default Error level)
[0084] export MINDIE_LOG_TO_STDOUT=1
[0085] # Virtual Memory
[0086] export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
[0087] # ATB WorkSpace Enabled
[0088] export ATB_WORKSPACE_MEM_ALLOC_ALG_TYPE=3
[0089] export ATB_WORKSPACE_MEM_ALLOC_GLOBAL=1
[0090] # Single-threaded acceleration of weight loading
[0091] export OMP_NUM_THREADS=10
[0092] # Set the video memory percentage
[0093] export NPU_MEMORY_FRACTION=0.97
[0094] export INF_NAN_MODE_ENABLE=1
[0095] # Turn off deterministic computation
[0096] export HCCL_DETERMINISTIC=false
[0097] export HCCL_OP_EXPANSION_MODE="AIV"
[0098] export HCCL_CONNECT _TIMEOUT=7200
[0099] export HCCL_EXEC_TIMEOUT=0
[0100] export ATB_LLM_HCCL_ENABLE=1
[0101] export ATB_LLM_COMM_BACKEND="hccl"
[0102] # Enable memory reuse
[0103] export ATB_LAYER_INTERNAL_TENSOR_REUSE=1
[0104] export ATB_OPERATION_EXECUTE_ASYNC=1
[0105] export ATB_CONVERT_NCHW_TO_ND=1
[0106] export ATB_WORKSPACE_MEM_ALLOC_GLOBAL=1
[0107] export ATB_WORKSPACE_MEM_ALLOC_ALG_TYPE=3
[0108] export ATB_CONTEXT_WORKSPACE_SIZE=0
[0109] export ATB_LAUNCH_KERNEL_WITH_TILING=1
[0110] export ATB_LLM_ENABLE_AUTO_TRANSPOSE=0
[0111] export MINDIE_ASYNC_SCHEDULING_ENABLE=1
[0112] # CPU affinity scheduling
[0113] export CPU_AFFINITY_CONF=1
[0114] export TASK_QUEUE_ENABLE=2
[0115] unset ASCEND_LAUNCH_BLOCKING
[0116] It is understood that in this embodiment, obtaining the corresponding container IP based on the container name includes: configuring a headless service for each target deployment node so that the target containers on each target deployment node can resolve container IPs using the headless service and the container name. In other words, this embodiment ensures multi-node communication consistency by configuring a headless service to support IP address resolution between Pods via Pod names. Furthermore, by configuring persistent storage and a unique network identifier in the StatefulSet, the state consistency and access reliability of the model inference service under multi-node deployment are guaranteed. Through the above model deployment process, the complete deployment architecture proposed in this embodiment, combining Kubernetes orchestration capabilities and MindIE inference engine features, can achieve fully automated management of processes such as model image pulling, resource configuration, service startup, and health checks.
[0117] As can be seen, this application utilizes the initial configuration files of each target deployment node to construct the target model deployment package, and combines the Kubernetes platform with the MindIE inference framework to convert the key parts of container deployment into Kubernetes deployment. Leveraging Kubernetes' StatefulSet controller, it achieves automated deployment, scaling, and management of model services, reducing manual intervention, simplifying operational processes, lowering maintenance costs, and improving model deployment efficiency in multi-machine environments. Furthermore, the use of the target model deployment package supports dynamic adjustment of resource configuration and service scale according to actual business needs, adapting to application requirements in different scenarios while maintaining good horizontal scalability, facilitating future feature upgrades and technological evolution, and significantly improving the flexibility and scalability of model deployment in multi-machine environments.
[0118] See Figure 4 As shown, this application discloses a model deployment device in a multi-machine environment, comprising:
[0119] The deployment package building module 11 is used to obtain the initial configuration files corresponding to each target deployment node and build the target model deployment package based on all the initial configuration files; the initial configuration files include container IP and RoCE network interface IP; the target model deployment package includes common variables, container startup methods and model service types in each of the initial configuration files;
[0120] Model deployment module 12 is used to deploy the target model in the target model deployment package to each of the target deployment nodes through the StatefulSet controller of the target model deployment platform; the target model deployment platform is the Kubernetes platform;
[0121] Service invocation module 13 is used to configure the target model of each target deployment node using the target model deployment package to obtain the target configuration file corresponding to each target deployment node, so as to use the MindIE inference engine to invoke the inference service of the target model on the corresponding target deployment node based on the target configuration file.
[0122] As can be seen, this application utilizes the initial configuration files of each target deployment node to construct the target model deployment package, and combines the Kubernetes platform with the MindIE inference framework to convert the key parts of container deployment into Kubernetes deployment. Leveraging Kubernetes' StatefulSet controller, it achieves automated deployment, scaling, and management of model services, reducing manual intervention, simplifying operational processes, lowering maintenance costs, and improving model deployment efficiency in multi-machine environments. Furthermore, the use of the target model deployment package supports dynamic adjustment of resource configuration and service scale according to actual business needs, adapting to application requirements in different scenarios while maintaining good horizontal scalability, facilitating future feature upgrades and technological evolution, and significantly improving the flexibility and scalability of model deployment in multi-machine environments.
[0123] In one specific implementation, the deployment package building module 11 may include:
[0124] The deployment package building submodule is used to build the target model deployment package based on all the initial configuration files using the target template;
[0125] The target template is a model deployment package template built using the Helm Chart tool of the target model deployment platform.
[0126] In one specific embodiment, the model deployment module 12 may include:
[0127] The model deployment submodule is used to launch the corresponding target container on each of the target deployment nodes through the target model deployment platform using the StatefulSet controller based on the container startup method in the target model deployment package, so as to deploy the target model in the target model deployment package on each of the target deployment nodes, and determine the first target deployment node among all the target deployment nodes as the master node to determine the master node IP.
[0128] In one specific implementation, the service invocation module 13 may include:
[0129] The information configuration submodule is used to configure the environment variables corresponding to the target model, the node device information of the target deployment node, and the master node IP on each target deployment node using the ConfigMap in the target model deployment package;
[0130] The container naming submodule is used to name the target container of the target deployment node based on the target naming mechanism using the StatefulSet controller in the target model deployment package to determine the container name of the target container;
[0131] The container IP acquisition submodule is used to obtain the corresponding container IP based on the container name;
[0132] The configuration file acquisition submodule is used to acquire the target configuration file corresponding to the target deployment node based on the container IP and RoCE network interface IP corresponding to the target deployment node, the environment variables corresponding to the target model, the node device information of the target deployment node, and the master node IP.
[0133] In one specific implementation, the container IP acquisition submodule may include:
[0134] The container IP acquisition unit is used to configure a headless service for each of the target deployment nodes, so that the target containers on each target deployment node can resolve container IPs using the headless service and the container name.
[0135] In one specific implementation, the model deployment submodule may include:
[0136] The container startup unit is used to start the corresponding target container on each target deployment node through the target model deployment platform using the StatefulSet controller based on the container startup method in the target model deployment package and the preset node anti-affinity strategy.
[0137] Furthermore, embodiments of this application also disclose an electronic device, Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.
[0138] Figure 5 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the model deployment method in a multi-machine environment disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.
[0139] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0140] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored thereon can include an operating system 221, computer programs 222, etc., and the storage method can be temporary storage or permanent storage.
[0141] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the model deployment method in a multi-machine environment executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.
[0142] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned model deployment method in a multi-machine environment. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.
[0143] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0144] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0145] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0146] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0147] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for deploying a model in a multi-machine environment, characterized in that, include: Obtain the initial configuration file corresponding to each target deployment node, and construct the target model deployment package based on all the initial configuration files; the initial configuration file includes the container IP and RoCE network interface IP; the target model deployment package includes the common variables, container startup method and model service type in each of the initial configuration files; The target model deployment platform utilizes a StatefulSet controller to deploy the target model in the target model deployment package to each of the target deployment nodes; the target model deployment platform is a Kubernetes platform. The target model is configured on each target deployment node using the target model deployment package to obtain the target configuration file corresponding to each target deployment node, so as to use the MindIE inference engine to call the inference service of the target model on the corresponding target deployment node based on the target configuration file.
2. The model deployment method in a multi-machine environment according to claim 1, characterized in that, The target configuration file is a node configuration file that meets the calling requirements of the MindIE inference engine. The target configuration file includes environment variable information, node device information, master node IP, container IP, and RoCE network port IP.
3. The model deployment method in a multi-machine environment according to claim 1, characterized in that, The construction of the target model deployment package based on all the initial configuration files includes: The target model deployment package is constructed using the target template based on all the initial configuration files. The target template is a model deployment package template built using the Helm Chart tool of the target model deployment platform.
4. The model deployment method in a multi-machine environment according to claim 1, characterized in that, The step of deploying the target model in the target model deployment package to each of the target deployment nodes using the StatefulSet controller through the target model deployment platform includes: The target model deployment platform utilizes the StatefulSet controller to launch the corresponding target container on each target deployment node based on the container startup method in the target model deployment package, so as to deploy the target model in the target model deployment package on each target deployment node, and determine the first target deployment node among all the target deployment nodes as the master node to determine the master node IP.
5. The model deployment method in a multi-machine environment according to claim 4, characterized in that, The step of configuring the target model of each target deployment node using the target model deployment package to obtain the target configuration file corresponding to each target deployment node includes: On each of the target deployment nodes, the environment variables corresponding to the target model, the node device information of the target deployment node, and the master node IP are configured using the ConfigMap in the target model deployment package; The StatefulSet controller in the target model deployment package is used to name the target container of the target deployment node based on the target naming mechanism to determine the container name of the target container; Obtain the corresponding container IP based on the container name; The target configuration file corresponding to the target deployment node is obtained based on the container IP and RoCE network port IP corresponding to the target deployment node, the environment variables corresponding to the target model, the node device information of the target deployment node, and the master node IP.
6. The model deployment method in a multi-machine environment according to claim 5, characterized in that, The step of obtaining the corresponding container IP based on the container name includes: Configure a headless service for each of the target deployment nodes so that the target containers on each target deployment node can resolve container IPs using the headless service and the container name.
7. The model deployment method in a multi-machine environment according to claim 4, characterized in that, The step of launching the corresponding target container on each target deployment node using the StatefulSet controller based on the container startup method in the target model deployment package through the target model deployment platform includes: The target model deployment platform utilizes the StatefulSet controller to launch the corresponding target containers on each target deployment node based on the container startup method and preset node anti-affinity strategy in the target model deployment package.
8. A model deployment device in a multi-machine environment, characterized in that, include: The deployment package building module is used to obtain the initial configuration files corresponding to each target deployment node and build the target model deployment package based on all the initial configuration files; the initial configuration files include container IP and RoCE network interface IP; the target model deployment package includes the common variables, container startup method and model service type in each of the initial configuration files; The model deployment module is used to deploy the target model in the target model deployment package to each of the target deployment nodes through the StatefulSet controller of the target model deployment platform; the target model deployment platform is the Kubernetes platform. The service invocation module is used to configure the target model of each target deployment node using the target model deployment package to obtain the target configuration file corresponding to each target deployment node, so as to use the MindIE inference engine to invoke the inference service of the target model on the corresponding target deployment node based on the target configuration file.
9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the model deployment method in a multi-machine environment as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the model deployment method in a multi-machine environment as described in any one of claims 1 to 7.