Model deployment method and system, and electronic device and computer-readable storage medium

Abstract

The present disclosure relates to large model technology and the field of model deployment. Disclosed are a model deployment method and system, and an electronic device and a computer-readable storage medium. The method comprises: acquiring a model file of a model to be deployed; distributing the model file to a plurality of storage devices, wherein different storage devices are geographically deployed in different regions; and mounting the storage devices to a target server cluster, which is deployed in the same region as the storage devices, among a plurality of server clusters, and thus the model to be deployed is deployed to the plurality of server clusters. The present disclosure solves the technical problem in the related art of the efficiency of cross-region deployment of a model being relatively low.

Description

Model deployment method, system, electronic device, and computer-readable storage medium Technical Field

[0001] The present disclosure relates to the fields of large model technology and model deployment, and more specifically, to a model deployment method, system, electronic device, and computer-readable storage medium. Background Art

[0002] Current large models store a large number of parameters and computational graph structures, resulting in very large model files. Loading the model when starting the service takes a long time, and is limited by factors such as region, network, and hardware. Cross-region file transfers further extend the loading time, resulting in low model deployment efficiency.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far.

[0004] Summary of the Invention

[0005] The embodiments of the present disclosure provide a model deployment method, system, electronic device, and computer-readable storage medium to at least solve the technical problem of low efficiency in cross-regional model deployment in related technologies.

[0006] According to one aspect of an embodiment of the present disclosure, a model deployment method is provided, including: obtaining a model file of a model to be deployed; distributing the model file to multiple storage devices, wherein different storage devices are geographically deployed in different regions; mounting the storage device on a target server cluster in multiple server clusters that is deployed in the same region as the storage device, so that the model to be deployed is deployed to multiple server clusters.

[0007] According to another aspect of an embodiment of the present disclosure, a model deployment method is also provided, including: in response to receiving a model file of a model to be deployed, storing the model file in a central warehouse; in response to receiving a model distribution request, distributing the model file to multiple storage devices, wherein different storage devices are geographically deployed in different regions, and different server clusters are also deployed in different regions; in response to receiving a server deployment request, based on the server deployment request, mounting the storage device to a target server cluster in multiple server clusters that is deployed in the same region as the storage device, so that the model to be deployed is deployed to multiple server clusters.

[0008] According to another aspect of an embodiment of the present disclosure, a model deployment method is also provided, including: obtaining a model file of a model to be deployed by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the model file; distributing the model file to multiple storage devices, wherein different storage devices are geographically deployed in different regions; when deploying the inference service of the model to be deployed, mounting the storage device on a target server cluster in multiple server clusters that is deployed in the same region as the storage device, so that the model to be deployed is deployed to multiple server clusters to obtain a deployment result of the model to be deployed; outputting the deployment result by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the deployment result.

[0009] According to another aspect of an embodiment of the present disclosure, a model deployment system is also provided, including: multiple server clusters, different server clusters are geographically deployed in different regions; multiple storage devices, different storage devices are geographically deployed in different regions; a control device, connected to the storage device and the server cluster, for distributing the model file of the model to be deployed to the storage device, and when deploying the inference service of the model to be deployed, mounting the storage device to the target server cluster deployed in the same region as the storage device, so that the model to be deployed is deployed to multiple server clusters.

[0010] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes any one of the methods in the above embodiments when running.

[0011] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored executable program, wherein when the executable program runs, the device where the computer-readable storage medium is located is controlled to execute any one of the methods in the above embodiments.

[0012] In an embodiment of the present disclosure, a model file of a model to be deployed can be obtained; the model file can be distributed to multiple storage devices, wherein different storage devices are geographically deployed in different regions; the storage device can be mounted on a target server cluster in multiple server clusters that is deployed in the same region as the storage device, so that the model to be deployed is deployed to multiple server clusters, thereby improving the processing efficiency of the model in different regions; it is easy to notice that the model file can be distributed to storage devices deployed in different regions, so as to utilize the network between the storage device and the server cluster in the same region to achieve high-speed interconnection between the storage device and the server cluster, eliminate cross-regional file transmission, thereby shortening the loading time of the model file, thereby improving the processing efficiency of the model in different regions, and thus solving the technical problem of low efficiency of cross-regional deployment of models in related technologies.

[0013] It is easy to note that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present disclosure, and do not constitute a limitation of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation of the present disclosure. In the drawings:

[0015] FIG1 is a schematic diagram of a hardware environment of a virtual reality device according to a model deployment method according to an embodiment of the present disclosure;

[0016] FIG2 is a flow chart of a model deployment method according to Embodiment 1 of the present disclosure;

[0017] FIG3 is a structural diagram of a model deployment method according to an embodiment of the present disclosure;

[0018] FIG4 is a flowchart of another model deployment method according to an embodiment of the present disclosure;

[0019] FIG5 is a flowchart of another model deployment method according to an embodiment of the present disclosure;

[0020] FIG6 is a flow chart of a model deployment method according to Embodiment 2 of the present disclosure;

[0021] FIG7 is a flowchart of a model deployment method according to Embodiment 3 of the present disclosure;

[0022] FIG8 is a schematic diagram of a model deployment device according to Embodiment 4 of the present disclosure;

[0023] FIG9 is a schematic diagram of a model deployment device according to Embodiment 5 of the present disclosure;

[0024] FIG10 is a schematic diagram of a model deployment device according to Example 6 of the present disclosure;

[0025] FIG11 is a schematic diagram of a model deployment system according to Embodiment 7 of the present disclosure;

[0026] FIG12 is a structural block diagram of a computer terminal according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present disclosure.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] The technical solution provided by the present disclosure is mainly implemented using large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. The large model can also be called a cornerstone model / foundation model (Foundation Model). The large model is pre-trained by large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities, such as large-scale language models (LLMs) and multi-modal pre-training models.

[0030] It should be noted that when the large model is actually applied, the pre-trained model can be fine-tuned through a small number of samples, so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision and other fields, specifically in computer vision tasks such as visual question answering (VQA), image description (IC), image generation, etc., and can also be widely used in natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In the embodiment of the present disclosure, the deployment of the large model in the model deployment scenario is taken as an example for explanation.

[0031] First, some nouns or terms that appear in the description of the embodiments of the present disclosure are subject to the following explanations:

[0032] Network Attached Storage (NAS): provides file sharing services over the network and has other additional features, providing a convenient and reliable way to store and share data;

[0033] Cloud Object Storage Service (OSS): provides cloud storage services for storing and managing massive amounts of unstructured data.

[0034] Virtual Private Cloud (VPC): A virtual network environment created in a cloud computing environment that provides an isolated and secure way to deploy and manage cloud resources.

[0035] Elastic Algorithm Service (EAS): This is a platform for deploying and managing machine learning models. EAS provides users with a convenient way to deploy, run, and manage their own machine learning algorithms and models, and offers scalable computing resources and high-performance computing capabilities.

[0036] Container orchestration platform (Kubernetes, abbreviated as K8S): can organize and manage containerized applications, helping users simplify application deployment, management, and expansion.

[0037] In order to solve the above problems, the present disclosure proposes a method based on multi-regional storage of storage devices, which deploys storage devices and services in the same virtual private network within the same region to achieve high-speed interconnection, eliminate cross-regional file transfer, thereby shortening the loading time of model files and further improving the elasticity of services.

[0038] Example 1

[0039] According to an embodiment of the present disclosure, a model deployment method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] Taking into account the huge amount of model parameters of the large model and the limited computing resources of the mobile terminal, the above-mentioned model deployment method provided by the embodiment of the present disclosure can be applied to the application scenario shown in Figure 1, but is not limited thereto. Figure 1 is a schematic diagram of the hardware environment of a virtual reality device according to a model deployment method of an embodiment of the present disclosure. In the application scenario shown in Figure 1, the large model is deployed in a server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of networks. The client devices 20 here may include but are not limited to: smart phones, tablet computers, laptops, PDAs, personal computers, smart home devices, car-mounted devices, etc. The client device 20 can interact with the user through a graphical user interface to implement the call to the large model, thereby implementing the method provided by the embodiment of the present disclosure.

[0041] In an embodiment of the present disclosure, a system consisting of a client device and a server can perform the following steps: the client device generates a model file for a model to be deployed; the server obtains the model file for the model to be deployed and multiple server clusters; distributes the model file to multiple storage devices; and mounts the storage device to a target server cluster deployed in the same region as the storage device, thereby deploying the model to be deployed to multiple server clusters. It should be noted that if the operating resources of the client device can meet the deployment and operation requirements of a large model, the embodiment of the present disclosure can be performed in the client device.

[0042] In the above operating environment, the present disclosure provides a model deployment method as shown in Figure 2. Figure 2 is a flow chart of the model deployment method according to Example 1 of the present disclosure. As shown in Figure 2, the method may include the following steps:

[0043] Step S202: Obtain the model file of the model to be deployed.

[0044] The aforementioned model to be deployed can be any model to be deployed. The model to be deployed can be a large model, a neural network model, a language processing model, or an image processing model. A large model can refer to a machine learning or deep learning model with a large number of parameters and powerful computing capabilities. In the embodiments disclosed herein, a large model is used as an example for description. The model to be deployed can be determined based on actual needs.

[0045] The above-mentioned model file can be a model file of a trained model to be deployed, wherein the model file includes but is not limited to the parameters and structure of the model. This is only an example, and the specific model file can be defined according to actual usage.

[0046] In an optional embodiment, a model file of a to-be-deployed model may be obtained, and a plurality of server clusters may be determined, wherein different server clusters are geographically deployed in different regions.

[0047] The above-mentioned multiple server clusters can be used to provide services to users. The above-mentioned different regions can be different countries, different provinces, different cities, etc., and the different regions are not specifically limited here.

[0048] In an optional embodiment, after the training of the model to be deployed is completed and the model file is generated, the model file can be stored in a cloud server. When the model to be deployed is deployed, the model file can be retrieved from the cloud server and distributed to storage devices in multiple different regions.

[0049] Step S204: Distribute the model file to multiple storage devices.

[0050] Among them, different storage devices are geographically deployed in different regions.

[0051] The above-mentioned storage device may be a hardware storage device, for example, a NAS, but is not limited thereto.

[0052] In an optional embodiment, the model file can be distributed to multiple storage devices based on a distribution request initiated by a user. Alternatively, the model management and control platform can retrieve the model file from the cloud server based on the distribution request initiated by the user and distribute and store the model file to multiple storage devices.

[0053] Step S206 : Mount the storage device to a target server cluster in the multiple server clusters that is deployed in the same region as the storage device, so that the model to be deployed is deployed to the multiple server clusters.

[0054] The above-mentioned mounting refers to connecting the storage device to the server of the target server cluster so that the model files in the storage device can be visible and accessible on the servers of the target server cluster.

[0055] In an optional embodiment, the storage device can be mounted to a target server cluster deployed in the same region as the storage device, so that users can quickly access the model file through the mounted storage device on the target server cluster, thereby more efficiently using the services provided by the model.

[0056] Through the above steps, the model file of the model to be deployed can be obtained; the model file is distributed to multiple storage devices, wherein different storage devices are geographically deployed in different regions; the storage device is mounted on a target server cluster in multiple server clusters that is deployed in the same region as the storage device, so that the model to be deployed is deployed to multiple server clusters, thereby improving the processing efficiency of the model in different regions; it is easy to notice that the model file can be distributed to storage devices deployed in different regions, so as to utilize the network between the storage device and the server cluster in the same region to achieve high-speed interconnection between the storage device and the server cluster, eliminate cross-regional file transmission, thereby shortening the loading time of the model file, thereby improving the processing efficiency of the model in different regions, and thus solving the technical problem of low efficiency of cross-regional deployment of models in related technologies.

[0057] In the above embodiment of the present disclosure, the storage device includes: a network attached storage, wherein distributing the model file to multiple storage devices includes: sending the model file to the multiple network attached storages through a public network and storing the model file in the multiple network attached storages.

[0058] The aforementioned network attached storage may be a storage device that provides file sharing services over a network and has other additional functions. The network attached storage may also provide users with a convenient and reliable way to store and share data.

[0059] The above-mentioned public network may be an open, shared and widely covered network.

[0060] In an optional embodiment, since the public network has a wide coverage area, the model file can be sent to multiple network attached storages through the public network so that the multiple network attached storages can receive the model file. After receiving the model file, the multiple network attached storages can store the model file.

[0061] In another optional embodiment, in order to improve the security during the storage process of the model file, the model file can be encrypted before being sent to multiple network attached storages via a public network to obtain an encrypted model file, and then the encrypted model file can be sent to the multiple network attached storages via the public network. After receiving the encrypted model file, the multiple network attached storages can decrypt the encrypted model file to obtain the model file and store the model file.

[0062] In the above embodiment of the present disclosure, when deploying the inference service of the model to be deployed, the storage device is mounted to a target server cluster in multiple server clusters that is deployed in the same region as the storage device, including: determining the target virtual private network corresponding to the storage device based on the target region where the storage device is deployed; obtaining the server cluster corresponding to the target virtual private network to obtain the target server cluster, wherein different server clusters correspond to different virtual private networks; when deploying the inference service of the model to be deployed, the storage device is mounted to the server in the target server cluster.

[0063] The target region can be determined based on the geographical location of the storage device deployment, or based on the country, province, etc. where the storage device is deployed. There is no limitation on the method for determining the target region here, and the method for determining the target region for storage device deployment can be set according to actual conditions.

[0064] The target virtual private cloud (VPC) can provide an isolated and secure way to deploy and manage resources, and can also increase file transfer speeds. Optionally, the target virtual private network can be a virtual network environment created in a cloud computing environment.

[0065] In an optional embodiment, a target virtual private network corresponding to the storage device can be determined based on the target region where the storage device is deployed. A server cluster connected to the target virtual private network can be obtained, and one or more server clusters from the server clusters connected to the target virtual private network can be selected as the target server cluster. Optionally, a server cluster geographically closest to the storage device can be selected from the server clusters connected to the target virtual private network as the target server cluster. The method for determining the target server cluster herein is provided for illustrative purposes only and is not specifically limited. The method can be configured based on actual circumstances.

[0066] In another optional embodiment, after determining the target region for storage device deployment, a search can be conducted for server clusters within the target region based on the target virtual private network. One or more server clusters within the target region can be used as the target server cluster. Alternatively, the server cluster closest to the storage device can be used as the target server cluster. The specific method for determining the target server cluster is not limited herein.

[0067] In another optional embodiment, when deploying the inference service of the model to be deployed, the storage device can be connected to the server in the target server cluster so that the model file in the storage device can be read on the server through the intranet.

[0068] In the above embodiment of the present disclosure, after mounting the storage device to a target server cluster in multiple server clusters and deployed in the same region as the storage device, the method also includes: building an elastic scheduling cluster for the target server cluster; and determining the computing resources required to deploy the model to be deployed based on the elastic scheduling cluster.

[0069] The aforementioned elastic scheduling cluster is a cluster management method based on cloud computing and virtualization technologies. It automatically adjusts the computing resources in the cluster according to changes in system load to meet different needs. The elastic scheduling cluster can adjust the cluster size based on load conditions, including adding or removing computing nodes to cope with high or low load conditions. This cluster management method can improve system flexibility and availability while also saving resources and costs.

[0070] The above-mentioned elastic scaling capability can automatically adjust the resources and capacity of the target service cluster according to real-time demand and load conditions to meet user needs; the elastic scaling capability can automatically increase or decrease the number of server instances in the target server cluster according to different situations to achieve load balancing and high availability.

[0071] In an optional embodiment, an elastic scheduling cluster can be built based on an online service platform (Elastic Algorithm Service, abbreviated as EAS) or a container orchestration platform (Kubernetes, abbreviated as K8S). The elastic scheduling cluster can provide elastic scaling capabilities for the target service cluster so that the target service cluster can automatically adjust system resources and capacity to meet user needs based on actual needs and load conditions.

[0072] In another optional embodiment, when the load of the target server cluster is low, the number of servers can be reduced to save costs, and when the load of the target server cluster is high, the number of servers can be increased to improve performance and capacity; optionally, the number of servers in the target server cluster can be managed according to pre-set rules, and the number of servers in the target server cluster can also be dynamically adjusted based on real-time monitoring and analysis.

[0073] In another optional embodiment, after building an elastic scheduling cluster for the target server cluster, it is necessary to determine the computing resources required to deploy the model to be deployed, such as memory, storage and other resources, and configure and allocate them according to the computing requirements of the model. By determining the computing resources required for deployment, it can be ensured that the model to be deployed can run normally on the elastic scheduling cluster and meet the required performance and availability requirements. In addition, sufficient computing resources can be provided by building an elastic scheduling cluster to support the deployment and operation of the model to be deployed.

[0074] In the above embodiment of the present disclosure, the method further includes: distributing preset resources to multiple server clusters; distributing model files to multiple storage devices, including: distributing the model files to multiple storage devices when the preset resources are distributed.

[0075] The above-mentioned preset resources can be various resources shared and allocated in the server cluster, among which the preset resources can be computing resources, network resources, storage resources, software resources, and load balancing resources. This is only described as an example, and the specific preset resources can be set according to actual conditions.

[0076] The aforementioned preset resources may be pre-set resource types and resource quantities, and there is no limitation on the preset resources herein.

[0077] The above-mentioned preset resources can also determine the preset resources according to the computing resources required for the deployment of the model to be deployed, and distribute the preset resources to multiple server clusters, so that the model to be deployed can be deployed through multiple server clusters later.

[0078] In an optional embodiment, the model management and control platform can be used to provide preset resources, including but not limited to management cluster metadata, service deployment, automatic / scheduled scalers, monitoring data, model distribution, monitoring resource usage, and grayscale / rollback.

[0079] In another optional embodiment, the preset resources can be distributed to multiple server clusters, and after the preset resources are distributed, the model file can be distributed to multiple storage devices, so that multiple storage devices can subsequently use the preset resources of the target server in the server cluster to deploy the model according to the model file.

[0080] In the above embodiment of the present disclosure, obtaining the model file of the model to be deployed includes: obtaining the model file of the model to be deployed from a central warehouse, wherein the model file is pre-uploaded to the central warehouse.

[0081] The central warehouse can be, but is not limited to, a cloud object storage service (OSS). The central warehouse can be used to provide cloud storage services and can be used to store and manage massive amounts of unstructured data.

[0082] The above-mentioned central warehouse can communicate with the server cluster and transfer files with the storage devices on the server cluster. According to the user's model deployment instructions, the central warehouse can automatically distribute the model files of the model to be deployed to multiple storage devices mounted on multiple server clusters.

[0083] In an optional embodiment, since the central warehouse has a large memory, after the training of the model to be deployed is completed, the model file of the model to be deployed can be pre-uploaded to the central warehouse. When the model to be deployed needs to be deployed, the model file corresponding to the model to be deployed can be retrieved from the central warehouse according to the model deployment instruction, and the model file can be distributed to multiple storage devices.

[0084] In the above embodiments of the present disclosure, the model to be deployed is a large model.

[0085] The aforementioned large model may be a model with a large storage capacity and a strong computing capability.

[0086] Figure 3 is a structural diagram of a model deployment method according to an embodiment of the present disclosure. As shown in Figure 3, the model management and control platform can store the model files of the model to be deployed in the central warehouse, wherein the model management and control platform can be used to provide preset resources, for example, managing cluster metadata, service deployment, auto / cron scaler, monitoring data, model distribute, monitoring resource usage, gray / rollback, and managing multiple accounts. The preset resources can be distributed to multiple server clusters. When the preset resources have been distributed, the model files can be distributed to multiple storage devices. As shown in FIG3 , the region A, region B, and region C can be included, wherein region A includes server cluster A, server cluster B, and network attached storage, region B includes server cluster A, server cluster B, and network attached storage, and region C includes server cluster A, server cluster B, and network attached storage. The network attached storage in different regions can be mounted on the servers in the server cluster so that the agents in the server cluster can store the model files in the network attached storage, and the servers in the server cluster can load the model files in the network attached storage. An elastic scheduling cluster can be constructed based on an online service platform or a container orchestration platform. The elastic scheduling cluster can provide elastic scaling capabilities for the target service cluster so that the target service cluster can automatically adjust system resources and capacity to meet user needs based on actual demand and load conditions.

[0087] The present disclosure may include two model deployment scenarios. The first model deployment scenario may be a scenario where the model, program, data, and image are deployed separately. FIG4 is a flowchart of another model deployment method according to an embodiment of the present disclosure. As shown in FIG4 , the method includes:

[0088] Step S401, distributing preset resources to multiple server clusters;

[0089] The model management and control platform in Figure 3 can be used to provide the above-mentioned preset resources.

[0090] Step S402: When the preset resources have been distributed, the model file is distributed to multiple storage devices;

[0091] Optionally, if the preset resources are not distributed completely, if the model file is distributed to multiple storage devices, an error message will be returned until the preset resources are distributed completely, and then the step of distributing the model file to multiple storage devices will be executed.

[0092] Step S403: determining a target virtual private network corresponding to the storage device based on the target region where the storage device is deployed;

[0093] Step S404, obtaining a server cluster corresponding to the target virtual private network to obtain a target server cluster;

[0094] Step S405: When deploying the inference service of the model to be deployed, the storage device is mounted on a server in the target server cluster.

[0095] The second model deployment scenario can be a mirror deployment scenario. In this process, there is no need to execute the resource distribution process, and service deployment can be performed directly. FIG5 is a flowchart of another model deployment method according to an embodiment of the present disclosure. As shown in FIG5, the method includes:

[0096] Step S501, obtaining the model file of the model to be deployed;

[0097] Step S502, distributing the model file to multiple storage devices;

[0098] Step S503: determining a target virtual private network corresponding to the storage device based on the target region where the storage device is deployed;

[0099] Step S504, obtaining a server cluster corresponding to the target virtual private network to obtain a target server cluster;

[0100] Step S505 , when deploying the inference service of the model to be deployed, the storage device is mounted to the server in the target server cluster.

[0101] The following are the experimental results of a single-machine test. This paper mainly uses NAS mounting model files. The OSS mounting model file and the NAS mounting model file can be compared using a GPT model with 13B parameters (25GB size):

[0102] 1. Use the dd command to test, time dd if= / home / workspace / model / pytorch_model.bin of= / dev / null bs=100K count=100000000;

[0103] a. Use OSS to mount the model file. 25716611305 bytes (26GB, 24GiB) have been copied. 115.811s, 222MB / s.

[0104] b. Using NAS to mount the model file, 25716611305 bytes (26GB, 24GiB) were copied in 4.23957s, 6.1GB / s.

[0105] 2. Large Model Inference Service Startup Speed ​​Test

[0106] a. Using OSS to mount the model file, the total time it takes to start the service is: 5m28.259s;

[0107] b. Using NAS to mount the model file, the total time taken to start the service is: 4m2.845s;

[0108] By comparison, it can be seen that NAS storage startup time is shorter than OSS.

[0109] This disclosure aims to provide a system for improving the multi-region, large-scale elasticity of large-scale model services. Based on NAS storage, this system enables the distribution of model files across multiple regions. When deploying an inference service, the system automatically mounts the NAS in the region to load the model files. Because the NAS and inference service are within the same VPC, high-speed interconnection is achieved, shortening model loading time, accelerating service startup, and improving elasticity.

[0110] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0111] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0112] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present disclosure.

[0113] Example 2

[0114] According to an embodiment of the present disclosure, a model deployment method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0115] FIG6 is a flow chart of a model deployment method according to Embodiment 2 of the present disclosure. As shown in FIG6 , the method includes the following steps:

[0116] Step S602: In response to receiving the model file of the model to be deployed, the model file is stored in the central warehouse;

[0117] Step S604, in response to receiving the model distribution request, distributing the model file to multiple storage devices;

[0118] Among them, different storage devices are geographically deployed in different regions, and different server clusters are also deployed in different regions.

[0119] The above-mentioned model distribution request can be a request for pre-distributing the model to a storage device when deploying the model to be deployed remotely. Optionally, the above-mentioned model distribution request can be generated by touching the interactive interface. The specific method of generating the above-mentioned model distribution request is not limited here.

[0120] Step S606, in response to receiving the server deployment request, based on the server deployment request, the storage device is mounted to a target server cluster in the multiple server clusters that is deployed in the same region as the storage device, so that the to-be-deployed model is deployed to the multiple server clusters.

[0121] The above-mentioned server deployment request can be a request generated when the user needs to deploy the deployment model. Optionally, the above-mentioned server deployment request can be generated by touching the interactive interface. The specific method of generating the above-mentioned server deployment request is not limited here.

[0122] Through the above steps, in response to receiving the model file of the model to be deployed, the model file is stored in the central warehouse; in response to receiving the model distribution request, the model file is distributed to multiple storage devices, wherein different storage devices are geographically deployed in different regions, and different server clusters are also deployed in different regions; in response to receiving the server deployment request, based on the server deployment request, the storage device is mounted to the target server cluster deployed in the same region as the storage device in the multiple server clusters, so that the model to be deployed is deployed to multiple server clusters, thereby improving the processing efficiency of the model in different regions; it is easy to notice that the model file can be distributed to the storage devices deployed in different regions, so as to utilize the network between the storage devices and the server clusters in the same region to achieve high-speed interconnection between the storage devices and the server clusters, eliminate the transmission of cross-regional files, thereby shortening the loading time of the model file, thereby improving the processing efficiency of the model in different regions, and thus solving the technical problem of low efficiency of cross-regional deployment of models in related technologies.

[0123] It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the solution provided in Example 1, as well as the application scenario and implementation process, but is not limited to the solution provided in Example 1.

[0124] Example 3

[0125] According to an embodiment of the present disclosure, a model deployment method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0126] FIG7 is a flow chart of a model deployment method according to Embodiment 3 of the present disclosure. As shown in FIG7 , the method includes the following steps:

[0127] Step S702: Acquire the model file of the model to be deployed by calling the first interface.

[0128] The first interface includes a first parameter, and the parameter value of the first parameter is a model file.

[0129] The first interface in the above steps can be an interface for data interaction between the cloud server and the client. The client can pass the model file of the model to be deployed into the interface function as the first parameter of the interface function to achieve the purpose of uploading the model file of the model to be deployed to the cloud server.

[0130] Step S704: Distribute the model file to multiple storage devices.

[0131] Among them, different storage devices are geographically deployed in different regions.

[0132] Step S706, when deploying the inference service of the model to be deployed, the storage device is mounted to the target server cluster deployed in the same region as the storage device in multiple server clusters, so that the model to be deployed is deployed to multiple server clusters to obtain the deployment result of the model to be deployed.

[0133] Step S708: Output the deployment result by calling the second interface.

[0134] The second interface includes a second parameter, and the parameter value of the second parameter is the deployment result.

[0135] The above-mentioned second interface can be an interface for data interaction between the cloud server and the client. The cloud server can pass the deployment result into the interface function as the second parameter of the interface function to achieve the purpose of sending the deployment result to the client.

[0136] Through the above steps, the model file of the model to be deployed is obtained by calling the first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the model file; the model file is distributed to multiple storage devices, wherein different storage devices are geographically deployed in different regions; when deploying the inference service of the model to be deployed, the storage device is mounted to a target server cluster in the same region as the storage device in multiple server clusters, so that the model to be deployed is deployed to multiple server clusters, and the deployment result of the model to be deployed is obtained; the deployment result is output by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the deployment result, thereby improving the processing efficiency of the model in different regions; it is easy to notice that the model file can be distributed to storage devices deployed in different regions, so as to utilize the network between the storage device and the server cluster in the same region to achieve high-speed interconnection between the storage device and the server cluster, eliminate cross-regional file transmission, thereby shortening the loading time of the model file, thereby improving the processing efficiency of the model in different regions, and thus solving the technical problem of low efficiency of cross-regional deployment of models in related technologies.

[0137] It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the solution provided in Example 1, as well as the application scenario and implementation process, but is not limited to the solution provided in Example 1.

[0138] Example 4

[0139] According to an embodiment of the present disclosure, a model deployment device for implementing the above-mentioned model deployment method is also provided. Figure 8 is a schematic diagram of a model deployment device according to Example 4 of the present disclosure. As shown in Figure 8, the device 800 includes: an acquisition module 802, a distribution module 804, and a mounting module 806.

[0140] Among them, the acquisition module is configured to obtain the model file of the model to be deployed; the distribution module is configured to distribute the model file to multiple storage devices, wherein different storage devices are geographically deployed in different regions; the mounting module is configured to mount the storage device to a target server cluster in multiple server clusters that is deployed in the same region as the storage device, so that the model to be deployed is deployed to multiple server clusters.

[0141] It should be noted that the acquisition module 802, distribution module 804, and mounting module 806 correspond to steps S202 to S206 in Example 1. The examples and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The modules can also be part of the device and can run in the server 10 provided in Example 1.

[0142] In the above embodiment of the present disclosure, the storage device includes: a network attached storage, wherein the distribution module is further configured to send the model file to the plurality of network attached storages via a public network and store the model file in the plurality of network attached storages.

[0143] In the above-mentioned embodiment of the present disclosure, the mounting module is also used to determine the target virtual private network corresponding to the storage device based on the target region where the storage device is deployed, obtain the server cluster corresponding to the target virtual private network, and obtain the target server cluster, wherein different server clusters correspond to different virtual private networks. When deploying the inference service of the model to be deployed, the storage device is mounted to the server in the target server cluster.

[0144] In the above embodiment of the present disclosure, the mounting module is also used to determine the target virtual private network corresponding to the storage device based on the target region where the storage device is deployed, obtain the server cluster corresponding to the target virtual private network, and obtain the target server cluster, wherein different server clusters correspond to different virtual private networks.

[0145] In the above embodiments of the present disclosure, the device further includes: a building module and a providing module.

[0146] Among them, the construction module is configured to construct an elastic scheduling cluster; the provision module is configured to provide elastic scaling capabilities for the target server cluster based on the elastic scheduling cluster.

[0147] In the above embodiment of the present disclosure, the distribution module is further used to distribute the preset resources to multiple server clusters; the distribution module is further used to distribute the model file to multiple storage devices when the preset resources are distributed.

[0148] In the above embodiment of the present disclosure, the acquisition module is further used to obtain the model file of the to-be-deployed model from the central warehouse, wherein the model file is uploaded to the central warehouse in advance.

[0149] In the above embodiments of the present disclosure, the model to be deployed is a large model.

[0150] It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the solution provided in Example 1, as well as the application scenario and implementation process, but is not limited to the solution provided in Example 1.

[0151] Example 5

[0152] According to an embodiment of the present disclosure, a model deployment device for implementing the above-mentioned model deployment method is also provided. Figure 9 is a schematic diagram of a model deployment device according to embodiment 5 of the present disclosure. As shown in Figure 9, the device 900 includes: a storage module 902, a distribution module 904, and a mounting module 906.

[0153] Among them, the storage module is configured to store the model file of the model to be deployed in the central warehouse in response to receiving the model file; the distribution module is configured to distribute the model file stored in the central warehouse to multiple storage devices in response to receiving a model distribution request, wherein different storage devices are geographically deployed in different regions, and different server clusters are also deployed in different regions; the mounting module is configured to mount the storage device to the target server cluster deployed in the same region as the storage device in multiple server clusters in response to receiving a server deployment request based on the server deployment request, so that the model to be deployed is deployed to multiple server clusters.

[0154] It should be noted that the storage module 902, distribution module 904, and mounting module 906 correspond to steps S602 to S606 in Example 2. The examples and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The modules can also be part of the device and can run in the server 10 provided in Example 1.

[0155] It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the solution provided in Example 1, as well as the application scenario and implementation process, but is not limited to the solution provided in Example 1.

[0156] Example 6

[0157] According to an embodiment of the present disclosure, a model deployment device for implementing the above-mentioned model deployment method is also provided. Figure 10 is a schematic diagram of a model deployment device according to Example 6 of the present disclosure. As shown in Figure 10, the device 1000 includes: an acquisition module 1002, a distribution module 1004, a mounting module 1006, and an output module 1008.

[0158] Among them, the acquisition module is configured to obtain the model file of the model to be deployed by calling the first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the model file; the distribution module is configured to distribute the model file to multiple storage devices, wherein different storage devices are geographically deployed in different regions; the mounting module is configured to mount the storage device to a target server cluster in multiple server clusters that is deployed in the same region as the storage device when deploying the inference service of the model to be deployed, so that the model to be deployed is deployed to multiple server clusters to obtain the deployment result of the model to be deployed; the output module is configured to output the deployment result by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the deployment result.

[0159] It should be noted that the acquisition module 1002, distribution module 1004, mounting module 1006, and output module 1008 correspond to steps S702 to S708 in Example 3. The examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The above modules can also be run as part of the device in the server 10 provided in Example 1.

[0160] It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the solution provided in Example 1, as well as the application scenario and implementation process, but is not limited to the solution provided in Example 1.

[0161] Example 7

[0162] According to an embodiment of the present disclosure, a model deployment system for implementing the above-mentioned model deployment method is also provided. FIG11 is a schematic diagram of a model deployment system according to Embodiment 7 of the present disclosure. As shown in FIG11 , the system 1100 includes:

[0163] Multiple server clusters 1102, different server clusters are geographically deployed in different regions;

[0164] Multiple storage devices 1104, different storage devices are geographically deployed in different regions;

[0165] The control device 1106 is connected to the storage device and the server cluster, and is used to distribute the model files of the model to be deployed to the storage device, and mount the storage device to the target server cluster deployed in the same region as the storage device, so that the model to be deployed can be deployed to multiple server clusters.

[0166] In the above embodiments of the present disclosure, the storage device 1104 includes: a network attached storage, which is connected to the control device via a public network.

[0167] In the above embodiments of the present disclosure, the target server cluster is connected to the storage device via a target virtual private network.

[0168] In the above embodiment of the present disclosure, the system further includes: an elastic scheduling cluster, which is used to provide elastic scaling capabilities for the target server cluster.

[0169] It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the solution provided in Example 1, as well as the application scenario and implementation process, but is not limited to the solution provided in Example 1.

[0170] Example 8

[0171] The embodiment of the present disclosure may provide an electronic device, which may be any electronic device in a group of electronic devices. Optionally, in this embodiment, the electronic device may also be replaced by a terminal device such as a mobile terminal.

[0172] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0173] In this embodiment, the above-mentioned electronic device can execute the program code of the following steps in the model deployment method: obtaining the model file of the model to be deployed, and multiple server clusters, wherein different server clusters are geographically deployed in different regions; distributing the model file to multiple storage devices, wherein different storage devices are geographically deployed in different regions; mounting the storage device to the target server cluster deployed in the same region as the storage device, so that the model to be deployed is deployed to multiple server clusters.

[0174] Optionally, Figure 12 is a block diagram of a computer terminal according to an embodiment of the present disclosure. As shown in Figure 12, the computer terminal A may include: one or more (only one is shown in the figure) processors 102, a memory 104, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0175] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the model deployment method and device in the embodiment of the present disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned model deployment method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely located relative to the processor, and these remote memories can be connected to the terminal A via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0176] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain the model file of the model to be deployed; distribute the model file to multiple storage devices, where different storage devices are geographically deployed in different regions; mount the storage device to the target server cluster in multiple server clusters that is deployed in the same region as the storage device, so that the model to be deployed is deployed to multiple server clusters.

[0177] Optionally, the processor may further execute the program code of the following steps: sending the model file to a plurality of network attached storages via a public network, and storing the model file in the plurality of network attached storages.

[0178] Optionally, the processor may further execute program code of the following steps: determining a target server cluster deployed in the target region based on the target region where the storage device is deployed; and mounting the storage device to a server in the target server cluster.

[0179] Optionally, the above-mentioned processor can also execute the program code of the following steps: based on the target region where the storage device is deployed, determine the target virtual private network corresponding to the storage device; obtain the server cluster corresponding to the target virtual private network to obtain the target server cluster, where different server clusters correspond to different virtual private networks; when deploying the inference service of the model to be deployed, mount the storage device to the server in the target server cluster.

[0180] Optionally, the processor may further execute program codes of the following steps: constructing an elastic scheduling cluster; and providing elastic scaling capabilities for the target server cluster based on the elastic scheduling cluster.

[0181] Optionally, the processor may further execute program codes of the following steps: distributing the preset resources to multiple server clusters; and distributing the model file to multiple storage devices after the preset resources have been distributed.

[0182] Optionally, the processor may further execute the program code of the following steps: obtaining a model file of the model to be deployed from the central warehouse, wherein the model file is uploaded to the central warehouse in advance.

[0183] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: in response to receiving the model file of the model to be deployed, the model file is stored in the central warehouse; in response to receiving the model distribution request, the model file is distributed to multiple storage devices, wherein different storage devices are geographically deployed in different regions, and different server clusters are also deployed in different regions; in response to receiving the server deployment request, based on the server deployment request, the storage device is mounted to the target server cluster deployed in the same region as the storage device in the multiple server clusters, so that the model to be deployed is deployed to multiple server clusters.

[0184] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain the model file of the model to be deployed by calling the first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the model file; distribute the model file to multiple storage devices, wherein different storage devices are geographically deployed in different regions; when deploying the inference service of the model to be deployed, mount the storage device to a target server cluster in multiple server clusters that is deployed in the same region as the storage device, so that the model to be deployed is deployed to multiple server clusters to obtain the deployment result of the model to be deployed; output the deployment result by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the deployment result.

[0185] Using an embodiment of the present disclosure, a model deployment method is provided, including: obtaining a model file of a model to be deployed; distributing the model file to multiple storage devices, wherein different storage devices are geographically deployed in different regions; mounting the storage device to multiple server clusters and a target server cluster deployed in the same region as the storage device, so that the model to be deployed is deployed to multiple server clusters, thereby improving the processing efficiency of the model in different regions; it is easy to notice that the model file can be distributed to storage devices deployed in different regions, so as to utilize the network between the storage device and the server cluster in the same region to achieve high-speed interconnection between the storage device and the server cluster, eliminate cross-regional file transmission, thereby shortening the loading time of the model file, thereby improving the processing efficiency of the model in different regions, and thus solving the technical problem of low efficiency of cross-regional deployment of models in related technologies.

[0186] Those skilled in the art will appreciate that the structure shown in FIG12 is merely illustrative, and the computer terminal may also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile internet device (MID), a PAD, or other terminal device. FIG12 does not limit the structure of the above-mentioned electronic devices. For example, the computer terminal A may include more or fewer components (such as a network interface, a display device, etc.) than those shown in FIG12, or may have a configuration different from that shown in FIG12.

[0187] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0188] Example 6

[0189] The embodiment of the present disclosure further provides a computer-readable storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the model deployment method provided in the first embodiment.

[0190] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0191] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: obtaining a model file of the model to be deployed, wherein different server clusters are geographically deployed in different regions; distributing the model file to multiple storage devices, wherein different storage devices are geographically deployed in different regions; mounting the storage device on a target server cluster in multiple server clusters that is deployed in the same region as the storage device, so that the model to be deployed is deployed to multiple server clusters.

[0192] Optionally, the storage medium is further configured to store program codes for executing the following steps: sending the model file to a plurality of network attached storages via a public network, and storing the model file in the plurality of network attached storages.

[0193] Optionally, the above-mentioned storage medium is also configured to store program code for executing the following steps: determining the target virtual private network corresponding to the storage device based on the target region where the storage device is deployed; obtaining the server cluster corresponding to the target virtual private network to obtain the target server cluster, wherein different server clusters correspond to different virtual private networks; when deploying the inference service of the model to be deployed, mounting the storage device on the server in the target server cluster.

[0194] Optionally, the storage medium is further configured to store program codes for executing the following steps: constructing an elastic scheduling cluster; and providing elastic scaling capabilities for the target server cluster based on the elastic scheduling cluster.

[0195] Optionally, the storage medium is further configured to store program codes for executing the following steps: distributing preset resources to multiple server clusters; and distributing the model file to multiple storage devices after the preset resources have been distributed.

[0196] Optionally, the storage medium is further configured to store program codes for executing the following steps: obtaining a model file of a to-be-deployed model from a central warehouse, wherein the model file is pre-uploaded to the central warehouse.

[0197] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: in response to receiving a model file of a model to be deployed, storing the model file in a central warehouse; in response to receiving a model distribution request, distributing the model file to multiple storage devices, wherein different storage devices are geographically deployed in different regions, and different server clusters are also deployed in different regions; in response to receiving a server deployment request, based on the server deployment request, mounting the storage device to a target server cluster in multiple server clusters that is deployed in the same region as the storage device, so that the model to be deployed is deployed to multiple server clusters.

[0198] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining a model file of the model to be deployed by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the model file; distributing the model file to multiple storage devices, wherein different storage devices are geographically deployed in different regions; when deploying the inference service of the model to be deployed, mounting the storage device on a target server cluster in multiple server clusters that is deployed in the same region as the storage device, so that the model to be deployed is deployed to multiple server clusters to obtain the deployment result of the model to be deployed; outputting the deployment result by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the deployment result.

[0199] Using an embodiment of the present disclosure, a model deployment method is provided, including: obtaining a model file of a model to be deployed; distributing the model file to multiple storage devices, wherein different storage devices are geographically deployed in different regions; mounting the storage device to a target server cluster in multiple server clusters that is deployed in the same region as the storage device, so that the model to be deployed is deployed to multiple server clusters, thereby improving the processing efficiency of the model in different regions; it is easy to notice that the model file can be distributed to storage devices deployed in different regions, so as to utilize the network between the storage device and the server cluster in the same region to achieve high-speed interconnection between the storage device and the server cluster, eliminate cross-regional file transmission, thereby shortening the loading time of the model file, thereby improving the processing efficiency of the model in different regions, and thus solving the technical problem of low efficiency of cross-regional deployment of models in related technologies.

[0200] The serial numbers of the above-mentioned embodiments of the present disclosure are for description only and do not represent the advantages or disadvantages of the embodiments.

[0201] In the above embodiments of the present disclosure, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0202] In the several embodiments provided in the present disclosure, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0203] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0204] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0205] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0206] The above is only a preferred embodiment of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present disclosure. These improvements and modifications should also be regarded as within the scope of protection of the present disclosure.

Claims

1. A model deployment method, comprising: Get the model file of the model to be deployed; Distributing the model file to a plurality of storage devices, wherein different storage devices are geographically deployed in different regions; When deploying the inference service of the model to be deployed, the storage device is mounted on a target server cluster in multiple server clusters that is deployed in the same region as the storage device, so that the model to be deployed is deployed to the multiple server clusters.

2. The method according to claim 1, wherein: The storage device includes: a network attached storage, distributing the model file to multiple storage devices, including: The model file is sent to the plurality of network attached storages through a public network and stored in the plurality of network attached storages.

3. The method according to claim 1, wherein: When deploying the inference service of the model to be deployed, mounting the storage device to a target server cluster in the plurality of server clusters that is deployed in the same region as the storage device includes: Determining a target virtual private network corresponding to the storage device based on the target region where the storage device is deployed; Acquire a server cluster corresponding to the target virtual private network to obtain the target server cluster, wherein different server clusters correspond to different virtual private networks; When deploying the inference service of the model to be deployed, the storage device is mounted to a server in the target server cluster.

4. The method according to claim 1, wherein: After mounting the storage device to a target server cluster in the plurality of server clusters and deployed in the same region as the storage device, the method further includes: Building an elastic scheduling cluster of the target server cluster; Based on the elastic scheduling cluster, computing resources required to deploy the to-be-deployed model are determined.

5. The method according to claim 1, wherein: The method further comprises: Distributing the preset resources to the plurality of server clusters; The model file is distributed to multiple storage devices.

6. The method according to claim 5, wherein: Distributing the model file to multiple storage devices, including: When the preset resources are distributed, the model file is distributed to the plurality of storage devices.

7. The method according to claim 1, wherein: Get the model file of the model to be deployed, including: The model file of the model to be deployed is obtained from the central warehouse, wherein the model file is pre-uploaded to the central warehouse.

8. The method according to claim 7, wherein: The central warehouse may be a cloud object storage.

9. The method according to claim 1, wherein: The model to be deployed is a large language model.

10. A model deployment method, wherein: include: In response to receiving a model file of a model to be deployed, storing the model file in a central repository; In response to receiving a model distribution request, distributing the model file stored in the central warehouse to a plurality of storage devices, wherein different storage devices are geographically deployed in different regions, and different server clusters are also deployed in the different regions; In response to receiving a server deployment request, based on the server deployment request, the storage device is mounted to a target server cluster in multiple server clusters that is deployed in the same region as the storage device, so that the model to be deployed is deployed to the multiple server clusters.

11. The method according to claim 10, wherein: The storage device includes: a network attached storage, distributing the model file stored in the central warehouse to multiple storage devices, including: The model file stored in the central warehouse is sent to the plurality of network attached storages through a public network and stored in the plurality of network attached storages.

12. A model deployment method, wherein: include: Acquire a model file of the model to be deployed by calling a first interface, wherein the first interface includes a first parameter, and a parameter value of the first parameter is the model file; Distributing the model file to a plurality of storage devices, wherein different storage devices are geographically deployed in different regions; When deploying the inference service of the model to be deployed, the storage device is mounted on a target server cluster in multiple server clusters that is deployed in the same region as the storage device, so that the model to be deployed is deployed to the multiple server clusters, and a deployment result of the model to be deployed is obtained; The deployment result is output by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter is the deployment result.

13. The method according to claim 12, wherein: The storage device includes: a network attached storage, distributing the model file to multiple storage devices, including: The model file is sent to the plurality of network attached storages through a public network and stored in the plurality of network attached storages.

14. A model deployment system, wherein: include: A plurality of server clusters, wherein different server clusters are geographically deployed in different regions; A plurality of storage devices, different storage devices being geographically deployed in different regions; A control device connected to the plurality of storage devices and the plurality of server clusters, for storing The model file of the model is distributed to multiple storage devices, and when deploying the inference service of the model to be deployed, the storage device is mounted to the target server cluster deployed in the same region as the storage device, so that the model to be deployed is deployed to multiple server clusters.

15. The system of claim 14, wherein: The storage device comprises: The network attached storage is connected to the control device via a public network.

16. The system of claim 14, wherein: Server clusters and storage devices in the same region are connected through a virtual private network.

17. The system of claim 14, wherein: The target server cluster is connected to the storage device via a target virtual private network.

18. The system of claim 14, wherein: The system further comprises: The elastic scheduling cluster is used to provide elastic scaling capabilities for the target server cluster.

19. An electronic device, wherein: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 13 when running.

20. A computer-readable storage medium, wherein: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 13.

Images