A model deployment method, apparatus, device, and medium

Abstract

This paper relates to a model deployment method, apparatus, device, and medium. The method includes: in response to an input operation, obtaining the performance requirements of a first model; in response to a deployment operation, determining a first configuration based on the performance requirements of the first model and resource data; and deploying the first model based on the first configuration. By employing the above technical solution, the performance requirements of the model to be deployed can be obtained based on the user's input operation. When the user triggers a deployment operation, the optimal configuration at that time is determined based on the performance requirements and resource data. The model is then deployed based on this configuration, achieving fully managed model deployment according to the user's input performance requirements. This reduces the model deployment cycle and thus reduces costs. Furthermore, it considers resource data to achieve the optimal configuration in terms of performance, stability, and cost for model deployment.

Description

Technical Field

[0001] This article relates to the field of computer technology, and in particular to a model deployment method, apparatus, device and medium. Background Technology

[0002] The rapid development of models presents challenges for model deployment, with users demanding a comprehensive improvement in performance, stability, and cost. Summary of the Invention

[0003] To address the aforementioned technical issues, this paper provides a model deployment method, apparatus, device, and medium.

[0004] This article provides a model deployment method, which includes: In response to input operations, obtain the performance requirements of the first model; In response to the deployment operation, a first configuration is determined based on the performance requirements and resource data of the first model, and the first model is deployed based on the first configuration.

[0005] This article also provides a model deployment apparatus, the apparatus comprising: The acquisition module is used to obtain the performance requirements of the first model in response to input operations; The deployment module is used to respond to the deployment operation by determining a first configuration based on the performance requirements and resource data of the first model, and deploying the first model based on the first configuration.

[0006] This document also provides an electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the executable instructions to implement the model deployment method provided herein.

[0007] This document also provides a computer-readable storage medium storing a computer program for performing the model deployment method provided herein.

[0008] The technical solution provided in this paper has the following advantages compared with existing technologies: The model deployment scheme provided in this paper, in response to input operations, obtains the performance requirements of a first model; in response to deployment operations, it determines a first configuration based on the performance requirements of the first model and resource data, and deploys the first model based on the first configuration. Using the above technical solution, the performance requirements of the model to be deployed can be obtained based on user input operations, and when the user triggers a deployment operation, the optimal configuration at that time is determined based on the performance requirements and resource data. The model is then deployed based on this configuration, achieving fully managed model deployment according to the user-input performance requirements, reducing the model deployment cycle, thereby reducing costs, and considering resource data to deploy the model in the configuration that is optimal in terms of performance, stability, and cost. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments herein will become more apparent when taken in conjunction with the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0010] Figure 1 This is a schematic diagram of an interactive scenario provided in this article; Figure 2 This is a flowchart illustrating a model deployment method provided in this paper. Figure 3 A flowchart illustrating another model deployment method provided in this article; Figure 4 This is a schematic diagram illustrating a model deployment process provided in this paper; Figure 5 This paper provides a schematic diagram of the structure of a model deployment device; Figure 6 This is a schematic diagram of the structure of an electronic device provided in this article. Detailed Implementation

[0011] The embodiments described herein will now be described in more detail with reference to the accompanying drawings. While some embodiments are shown in the drawings, it should be understood that this document can be implemented in various forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided to provide a more thorough and complete understanding of the document. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of this document.

[0012] It should be understood that the steps described in the method embodiments herein may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this document is not limited in this respect.

[0013] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0014] It should be noted that the concepts of "first" and "second" mentioned in this article are only used to distinguish different devices, modules or units, and are not used to limit the order of the functions performed by these devices, modules or units or their interdependencies.

[0015] It should be noted that the terms "one" and "more" used in this document are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0016] The names of messages or information exchanged between multiple devices in the embodiments herein are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0017] In related technologies, the challenges of model deployment are reflected in the following aspects: rapid model iteration, a wide variety of models, and the continuous release of various new models and versions, resulting in long deployment cycles and high costs; and significant differences among various graphics processor models, making it very difficult to find a deployment method that comprehensively optimizes performance, stability, and cost.

[0018] To address the aforementioned issues, this paper presents a model deployment method, which will be described below with reference to specific embodiments.

[0019] For example, Figure 1 This article provides a schematic diagram of an interactive scenario, such as... Figure 1 As shown, the model deployment method is applied to a model deployment system, which is set in an electronic device 101 in the figure. The electronic device 101 responds to the user's input operation to obtain the performance requirements of the model, and responds to the deployment operation to determine the configuration based on the performance requirements and resource data, and then deploys the model through the configuration. The user can perform interactive operations on the electronic device 101, and the gestures of the electronic device 101 represent the interactive operations triggered by the user.

[0020] Figure 2 This is a flowchart illustrating a model deployment method provided in this paper. This method can be executed by a model deployment device, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. Figure 2 As shown, the method includes: Step 201: In response to the input operation, obtain the performance requirements of the first model.

[0021] The model deployment method presented in this paper can be applied to a model deployment system that deploys models on a graphics processor. Here, the model can be any functional machine learning model that needs to be deployed.

[0022] Input operations can be operations on input controls on the page, while output controls can include image input controls for inputting images, text input controls for inputting text, etc. The first model can be the model to be deployed, with no limit on its specific functions or number; for example, the first model could be a pre-trained question-answering model. Performance requirements can measure the speed of the first model's output, and can be user requirements for the first model. Optionally, the performance requirements of the first model include at least one of the following: Time Per Output Token (TPOT) and Tokens Per Minute (TPM). TPOT can be the average generation time for each output token, also known as data unit generation latency or per-to-total ...

[0023] In some embodiments, obtaining the performance requirements of the first model in response to an input operation may include at least one of the following: obtaining the data unit generation time of the first model in response to an editing operation on the time control; and obtaining the number of data units per unit time of the first model in response to an editing operation on the quantity control.

[0024] The time control can be a control for inputting the generation time of data units, and the quantity control can be a control for inputting the number of time units. The model deployment device can display a first page to the user, including the time control and the quantity control. The first page can provide the user with a page to select the first model to be deployed and input the performance requirements of the first model. When the user's editing operation on the time control is detected, the generation time of the data units entered by the user in the time control can be obtained, and / or. When the user's editing operation on the quantity control is detected, the number of data units per unit time entered in the quantity control can be obtained, thereby obtaining the performance requirements of the first model. By providing functional controls, users can quickly input the performance requirements of the model, which more efficiently guides model deployment and prepares for subsequent model deployment.

[0025] Step 202: In response to the deployment operation, determine the first configuration based on the performance requirements and resource data of the first model, and deploy the first model based on the first configuration.

[0026] Deployment operations can be interactive operations that trigger the deployment of the first model, such as clicking or double-clicking the deployment control; this is not limited to these actions in this document. Resource data can include the status of all graphics processing units (GPUs) in the system, specifically the first GPU that is idle. The first GPU can be an unassigned or unused GPU, while other GPUs besides the first GPU have been assigned.

[0027] The first configuration can be the configuration parameters required to deploy the model to a real resource environment. This first configuration adjusts the model's behavior, performance, and interaction with external systems within the deployment environment. It can be the optimal configuration selected from multiple simulation configurations, specifically chosen based on performance requirements and resource data. Optionally, the first configuration should include at least the following: the type of graphics processor, the number of graphics processors, the number of instances, inference framework parameters, and a resource queue. The type of graphics processor includes the supplier and model. The number of instances can be the maximum number of model instances deployed on the corresponding type of graphics processor in the current configuration. This number can be dynamically adjusted based on utilization and time, for example, more instances during the day and fewer at night. The inference framework parameters are parameters used to adjust the inference process, performance, and results when using the inference framework for model inference. These parameters can be understood as configuration parameters related to the inference process when deploying the model, such as data parallel processing and weights. The resource queue can include the status of the corresponding type of graphics processor in the current configuration. The simulation configuration includes the content of the first configuration and the corresponding simulation performance parameters.

[0028] When the model deployment device detects a user's deployment operation, it can select a first configuration from multiple simulation configurations based on the performance requirements and resource data of the first model, and deploy the first model through the first configuration. The user can then use the first model to perform corresponding functions, such as question answering.

[0029] For example, Figure 3 A flowchart illustrating another model deployment method provided in this article is shown below. Figure 3 As shown, in one feasible implementation, determining the first configuration based on the performance requirements of the first model and resource data may include: Step 301: Obtain multiple simulation configurations.

[0030] The simulation configuration can be the configuration parameters for model deployment when the simulator simulates the first model. There can be multiple simulation configurations, which can include the type of graphics processor, the number of graphics processors, the number of instances, inference framework parameters, resource queues, and performance parameters. At least one parameter of each simulation configuration is different. The simulator runs the first model under the simulation configuration and determines the corresponding performance parameters.

[0031] Step 302: Based on the performance requirements and resource data of the first model, multiple simulation configurations are filtered to obtain the first configuration.

[0032] Optionally, based on the performance requirements and resource data of the first model, multiple simulation configurations are screened to obtain a first configuration, which may include: extracting multiple simulation configurations based on the performance requirements of the first model to obtain multiple second configurations; extracting multiple third configurations based on the first graphics processor for the multiple second configurations; and determining the first configuration based on the evaluation results corresponding to the multiple third configurations.

[0033] The second configuration can be a configuration that meets the performance requirements input by the user from multiple simulated configurations. The third configuration can be a configuration from multiple second configurations where the graphics processor is in an idle state. The evaluation result can be a specific score obtained by quantifying multiple third configurations from the perspective of resource quantity and value. The higher the score, the greater the resource quantity and the lower the cost of this configuration, which means that this configuration is better in terms of performance, stability and cost.

[0034] When the model deployment device filters multiple simulation configurations, it can do so based on the performance requirements of the first model. If a simulation configuration's performance parameters meet the performance requirements (e.g., its data unit generation time is less than or equal to the required data unit generation time), it is determined to meet the performance requirements and designated as a second configuration. There are multiple second configurations. For each second configuration, it is determined whether its graphics processor type is the first graphics processor. If so, the second configuration is designated as a third configuration. There are multiple third configurations. The evaluation result of each third configuration is determined, and the first configuration is then selected based on the evaluation results. There are one or more first configurations. By filtering suitable configurations from multiple simulation configurations based on the user-input model performance requirements and resource data, and considering the evaluation results, the selected configuration achieves the optimal balance of performance, stability, and cost.

[0035] Optionally, determining the first configuration based on the evaluation results corresponding to multiple third configurations may include: determining the corresponding evaluation result based on the association information of the first graphics processor in the third configuration; in response to multiple third configurations corresponding to the same type of graphics processor, determining the third configuration with the highest evaluation result as the first configuration; in response to multiple third configurations corresponding to multiple types of graphics processors, extracting the third configuration with the highest evaluation result for each type, and combining the multiple third configurations to obtain the first configuration.

[0036] The associated information can include the number of graphics processors (GPUs) and resource value parameters. The resource value parameters can be the price of the GPUs, representing resource costs. When determining the first configuration based on the evaluation results of multiple first configurations, the model deployment device can obtain the associated information of the first GPUs in the third configuration. Specifically, it extracts the number of first GPUs in the third configuration and obtains the resource value parameters of the first GPUs. It then calculates the evaluation result of the third configuration based on the number and resource value parameters. For example, the calculation method can be a weighted summation. The weights of the number and resource value parameters can be the same or different depending on the situation. The evaluation result is positively correlated with the number, indicating that the larger the number of idle GPUs in the third configuration, the higher the evaluation result. The evaluation result is negatively correlated with the resource value parameters, indicating that the lower the cost of the idle GPUs in the third configuration, the higher the evaluation result. Afterward, it can be determined whether the types of GPUs in the multiple third configurations are the same. If so, the third configuration with the highest evaluation result can be determined as the first configuration, and the number of first configurations is one. If the types of GPUs in the multiple third configurations are different, the third configuration with the highest evaluation result for each type of GPU is determined as the first configuration, and thus the first configuration includes the third configuration with the highest evaluation result for each type, and the number of first configurations is multiple. In the above scheme, when selecting the optimal configuration, the evaluation results of resource quantity and resource value parameters can be considered to improve the accuracy and comprehensiveness of configuration selection. Furthermore, one optimal configuration is selected for each type of graphics processor, which helps to dynamically switch graphics processors after the model is deployed.

[0037] In some embodiments, deploying a first model based on a first configuration may include: obtaining deployment template and inference framework information; generating a model description file based on the first configuration, deployment template, and inference framework information; sending the model description file to the graphics processor corresponding to the first configuration, and deploying the first model in the graphics processor.

[0038] Deployment templates can be pre-built files or data structures with specific formats and content. They include a framework of various parameters, configuration information, and execution instructions required for deploying the model. Space is reserved for configuration. A deployment template can be called a Worker Template Group, comprising a collection of templates for different roles required in a specific deployment configuration. This deployment template is compatible with multiple types of graphics processors and provides a standardized and reusable solution for model deployment. Inference framework information can include all version numbers of the first model's inference framework. It can be a list or set of version numbers, with different version numbers corresponding to different update times. The model description file can be a configuration file generated using specific tools to store configuration information related to model deployment. This file can use a highly readable text format to organize various complex configuration information in an orderly manner. It includes a series of key information, from the model's parameter settings and runtime environment configuration to various operation instructions during deployment, providing detailed guidance for the accurate deployment and operation of the model in the target environment.

[0039] When deploying a first model based on a first configuration, the model deployment device can first obtain the deployment template and inference framework information, then input the first configuration, deployment template, and inference framework information into the ApplicationBuilder to generate a model description file for the first model, and then distribute the model description file to the graphics processor corresponding to the first configuration to deploy the first model. When deploying a model based on a configuration, a model description file can be generated based on the deployment template and inference version framework, and then the model description file can be distributed to the resource cluster for deployment. The deployment template provides a standardized deployment scheme for the model, so that different models follow the same template when deployed, thus improving deployment efficiency.

[0040] Optionally, based on the first configuration, deployment template, and inference framework information, a model description file is generated, including: extracting the major version number from the first configuration; extracting the first version number from the inference framework information based on the major version number, wherein the first version number is the latest among at least one second version number corresponding to the major version number, and the first version number includes the major version number and the minor version number; filling the inference framework parameters of the first configuration and the second version number into the deployment template to obtain deployment information; and generating the model description file based on the deployment information.

[0041] The major version number can be a significant component of the inference framework's version number, used to identify major, fundamental changes to the inference framework's functionality, architecture, or core features. For example, the major version number could be 1.35. The minor version number is a portion of the inference framework's version number, used to represent functional enhancements, improvements, and minor optimizations made to the inference framework. The major and minor version numbers together form a complete version number. The first version number can be the latest complete version number, including the major version number in the first configuration, or it can be the most recently updated version number based on the major version number in the first configuration, which includes both the major and minor version numbers. For example, if the major version number of the first configuration is 1.35 and the second version number is 1.35.3, then 1.35 is the major version number and 3 is the minor version number. The second version number can be a complete version number including the major version number in the first configuration. There must be at least one second version number. For example, if the major version number of the first configuration is 1.35, the second version numbers could be 1.35.1, 1.35.2, and 1.35.3. Different second version numbers have different update times; the larger the second version number, the more recent the update time. The first version number is the largest of the at least one second version number. Deployment information can be complete information obtained by filling in the filtered configuration based on the deployment template.

[0042] When the model deployment device generates a model description file based on the first configuration, deployment template, and inference framework information, it can extract the first version number from the inference framework parameters in the first configuration through the application builder, and extract at least one second version number corresponding to the major version number of the first configuration from the inference framework information. The second version number with the most recent update time among the at least one second version number is determined as the first version number. After replacing the original inference framework parameters in the first configuration with the inference framework parameters of the first version number, the deployment information is filled into the deployment template. The model description file of the first model is generated based on the deployment information. Optionally, the application builder can generate application-level configuration, worker set template, volume configuration, and framework configuration based on deployment information, and combine them to generate a model description file. The application-level configuration may include metadata information unrelated to model deployment details, such as resource queue names and application scheduling priorities. The worker set template may be a deployment definition template for a specific role during distributed deployment, such as single-instance deployment or PD (Prefill-Decode) separate deployment. There can be multiple worker set templates, and different worker set templates correspond to different roles, such as different worker set templates corresponding to different graphics processor types. The volume configuration may be configuration related to model file storage volumes, providing model files for model deployment. The inference framework configuration may be a common inference framework configuration, referring to the configuration that does not need to be simulated by the simulator.

[0043] In the above scheme, the inference framework parameters can be extracted when the model description file is generated, which optimizes inference performance, accelerates the inference process, adapts to different graphics processors while meeting user needs, and improves the stability and compatibility of the model.

[0044] The model deployment scheme presented in this paper responds to input operations by obtaining the performance requirements of a first model; responding to deployment operations, it determines a first configuration based on the performance requirements and resource data of the first model, and deploys the first model based on the first configuration. Using this technical solution, the performance requirements of the model to be deployed can be obtained based on user input operations. When the user triggers a deployment operation, the optimal configuration at that time is determined based on the performance requirements and resource data. The model is then deployed based on this configuration, achieving fully managed model deployment according to the user-input performance requirements. This reduces the model deployment cycle, thereby reducing costs, and considers resource data to achieve the optimal configuration in terms of performance, stability, and cost.

[0045] The following concrete example will further illustrate the model deployment scheme presented in this paper. For instance, Figure 4 This paper provides a schematic diagram of a model deployment process, such as... Figure 4 As shown, the model deployment process may include: responding to user input to obtain the performance requirements of the first model, and obtaining multiple simulation configurations and resource data; the simulator can simulate the first model on multiple simulation configurations to obtain corresponding performance parameters; through the filter in the scheduler, multiple second configurations can be extracted from multiple model configurations based on the performance requirements of the first model, and multiple third configurations can be extracted from multiple second configurations based on resource data; the scorer in the scheduler determines the evaluation result of each third configuration, and the first configuration is determined based on the evaluation result; obtaining deployment template and inference framework information; inputting the deployment template, first configuration, and inference framework information into the application creator to generate the model description file of the first model; distributing the model description file to the graphics processor corresponding to the first configuration through the resource manager to deploy the first model, and the first model can provide services to the outside world.

[0046] This paper presents a fully managed model deployment solution that supports efficient model deployment and rapid model updates. It obtains performance parameters under different simulation configurations through a simulator, and selects the optimal configuration in terms of performance, stability, and cost based on user-input performance requirements and resource data. This solution is then used for model deployment and is widely adaptable to various heterogeneous hardware resources.

[0047] Figure 5 This is a schematic diagram of a model deployment device provided in this paper. This device can be implemented by software and / or hardware, and is generally integrated into an electronic device. Figure 5 As shown, the device includes: The acquisition module 501 is used to acquire the performance requirements of the first model in response to the input operation; Deployment module 502 is used to respond to deployment operations, determine a first configuration based on the performance requirements and resource data of the first model, and deploy the first model based on the first configuration.

[0048] Optionally, the performance requirements of the first model include at least one of the following: data unit generation time and number of data units per unit time; Module 501 is used for: In response to the editing operation of the time control, the generation time of the data unit of the first model is obtained; In response to an edit operation on the quantity control, the number of data units per unit time for the first model is obtained.

[0049] Optionally, the deployment module 502 includes a configuration submodule, which is used for: The simulation unit is used to acquire multiple simulation configurations; The filtering unit is used to filter the plurality of simulation configurations based on the performance requirements of the first model and the resource data to obtain the first configuration.

[0050] Optionally, the resource data includes a first graphics processor in an idle state; The filtering unit includes: The first subunit is used to extract the multiple simulation configurations based on the performance requirements of the first model to obtain multiple second configurations; The second subunit is used to extract multiple third configurations based on the first graphics processor for the multiple second configurations; The third subunit is used to determine the first configuration based on the evaluation results corresponding to the plurality of third configurations.

[0051] Optionally, the third subunit is used for: The corresponding evaluation result is determined based on the association information of the first graphics processor in the third configuration; In response to the fact that the plurality of third configurations correspond to the same type of graphics processor, the third configuration with the highest evaluation result is determined as the first configuration; In response to the plurality of third configurations corresponding to the plurality of graphics processors of the plurality of types, the third configuration with the highest evaluation result is extracted from the plurality of types, and the plurality of third configurations are combined to obtain the first configuration.

[0052] Optionally, the deployment module 502 includes an execution submodule, which includes: The acquisition unit is used to acquire deployment templates and inference framework information; The generation unit is configured to generate a model description file based on the first configuration, the deployment template, and the inference framework information; An execution unit is configured to send the model description file to the graphics processor corresponding to the first configuration, and deploy the first model in the graphics processor.

[0053] Optionally, the generating unit is used for: Extract the major version number from the first configuration, and extract the first version number from the inference framework information based on the major version number. The first version number is the latest among at least one second version number corresponding to the major version number, and the first version number includes the major version number and the minor version number. The inference framework parameters of the first configuration and the second version number are filled into the deployment template to obtain deployment information; The model description file is generated based on the deployment information.

[0054] Optionally, the first configuration includes at least the following: the type of graphics processor, the number of graphics processors, the number of instances, inference framework parameters, and resource queues.

[0055] The model deployment apparatus provided herein can execute the model deployment method provided in any embodiment herein, and has the corresponding functional modules and beneficial effects of the execution method.

[0056] This document also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the model deployment method provided in any embodiment of this document and has the same beneficial effects as the implementation method embodiment.

[0057] Figure 6 This document provides a schematic diagram of the structure of an electronic device. The document also provides an electronic device comprising: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the executable instructions to implement the model deployment method provided in any embodiment of this document, achieving the same beneficial effects as the execution method embodiments.

[0058] The following is a detailed reference. Figure 6 The diagram illustrates a suitable structural schematic for implementing the electronic device 600 described herein. The electronic device 600 described herein may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not impose any limitations on the functionality and scope of this article.

[0059] like Figure 6As shown, electronic device 600 may include processing unit 601 (e.g., central processing unit, graphics processor, etc.), which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 602 or programs loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0060] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0061] In particular, according to embodiments herein, the processes described in the above-referenced flowcharts can be implemented as computer software programs. For example, embodiments herein include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 609, or installed from storage device 608, or installed from ROM 602. When the computer program is executed by processing device 601, it performs the functions defined in the model deployment method herein.

[0062] It should be noted that the computer-readable medium mentioned above can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an electrically erasable programmable read-only memory (EPROM), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this document, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, radio frequency (RF), etc., or any suitable combination thereof.

[0063] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol, such as Hypertext Transfer Protocol (HTTP), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include Local Area Networks (LANs), Wide Area Networks (WANs), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0064] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0065] The aforementioned computer-readable medium carries one or more programs. When these programs are executed by the electronic device, the electronic device: in response to an input operation, obtains the performance requirements of a first model; in response to a deployment operation, determines a first configuration based on the performance requirements of the first model and resource data, and deploys the first model based on the first configuration. The computer-readable storage medium implements the model deployment method provided in any embodiment of this document and has the same beneficial effects as the implementation method embodiments.

[0066] Computer program code for performing the operations described herein may be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including local area networks (LANs) or wide area networks (WANs), or it may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0067] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments herein. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0068] The units described herein can be implemented in software or hardware. The names of the units are not, in some cases, limiting to the unit itself.

[0069] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field-Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and so on.

[0070] In the context of this document, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, flash memory, optical fiber, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0071] It is understood that before using the technical solutions disclosed in the embodiments of this article, users should be informed of the type, scope of use, and usage scenarios of the information involved in this article in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0072] The above description is merely a preferred embodiment and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure herein is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed herein that have similar functions.

[0073] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order. Multitasking and parallel processing may be advantageous in certain environments. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this document. Certain features described in the context of a single embodiment may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0074] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A model deployment method, comprising: In response to input operations, obtain the performance requirements of the first model; In response to the deployment operation, a first configuration is determined based on the performance requirements and resource data of the first model, and the first model is deployed based on the first configuration.

2. The method according to claim 1, wherein the performance requirements of the first model include at least one of data unit generation time and the number of data units per unit time; In response to an input operation, obtain the performance requirements of the first model, including at least one of the following: In response to the editing operation of the time control, the generation time of the data unit of the first model is obtained; In response to an edit operation on the quantity control, the number of data units per unit time for the first model is obtained.

3. The method according to claim 1, wherein determining the first configuration based on the performance requirements of the first model and resource data includes: Obtain multiple simulated configurations; Based on the performance requirements of the first model and the resource data, the multiple simulation configurations are filtered to obtain the first configuration.

4. The method of claim 3, wherein the resource data includes a first graphics processor in an idle state; Based on the performance requirements of the first model and the resource data, the multiple simulation configurations are filtered to obtain the first configuration, including: Based on the performance requirements of the first model, multiple simulation configurations are extracted to obtain multiple second configurations; For the multiple second configurations, multiple third configurations are extracted based on the first graphics processor; The first configuration is determined based on the evaluation results corresponding to the multiple third configurations.

5. The method according to claim 4, wherein determining the first configuration based on the evaluation results corresponding to the plurality of third configurations includes: The corresponding evaluation result is determined based on the association information of the first graphics processor in the third configuration; In response to the fact that the plurality of third configurations correspond to the same type of graphics processor, the third configuration with the highest evaluation result is determined as the first configuration; In response to the plurality of third configurations corresponding to the plurality of graphics processors of the plurality of types, the third configuration with the highest evaluation result is extracted from the plurality of types, and the plurality of third configurations are combined to obtain the first configuration.

6. The method according to claim 1, wherein deploying the first model based on the first configuration comprises: Obtain deployment template and inference framework information; Based on the first configuration, the deployment template, and the inference framework information, a model description file is generated; The model description file is sent to the graphics processor corresponding to the first configuration, and the first model is deployed in the graphics processor.

7. The method according to claim 6, wherein a model description file is generated based on the first configuration, the deployment template, and the inference framework information, comprising: Extract the major version number from the first configuration, and extract the first version number from the inference framework information based on the major version number. The first version number is the latest among at least one second version number corresponding to the major version number, and the first version number includes the major version number and the minor version number. The inference framework parameters of the first configuration and the first version number are filled into the deployment template to obtain deployment information; The model description file is generated based on the deployment information.

8. The method according to claim 1, wherein the first configuration includes at least the following: the type of graphics processor, the number of graphics processors, the number of instances, inference framework parameters, and resource queues.

9. A model deployment apparatus, comprising: The acquisition module is used to obtain the performance requirements of the first model in response to input operations; The deployment module is used to respond to the deployment operation by determining a first configuration based on the performance requirements and resource data of the first model, and deploying the first model based on the first configuration.

10. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the model deployment method according to any one of claims 1-8.

11. A computer-readable storage medium storing a computer program for performing the model deployment method according to any one of claims 1-8.

Images