Deployment methods, devices, media, equipment, and program products for large model services

Abstract

A method, apparatus, medium, device, and program product for deploying a large-scale model service are disclosed. The deployment method includes: obtaining first configuration information, which includes configuration information of a first execution environment, wherein the large-scale model service is deployed in a second execution environment, and the security level of the first execution environment is higher than that of the second execution environment; generating a first policy based on the first configuration information, which includes policies to be executed to obtain first deployment information, and the first deployment information includes deployment information for deploying the large-scale model service in the first execution environment; executing the first policy to obtain the first deployment information; and deploying the large-scale model service in the first execution environment based on the first deployment information. This enables the large-scale model service to be deployed and reliably run in the more secure first execution environment, mitigating the problem of the large-scale model service failing to start or malfunctioning after deployment in the first execution environment due to differences in execution environments.

Description

Technical Field

[0001] This article relates to the field of computer technology, specifically to a method, apparatus, medium, device, and program product for deploying large model services. Background Technology

[0002] PaaS (Platform as a Service) provides a software development platform as a service to users. MaaS (Model as a Service) is a type of PaaS, which encapsulates machine learning models as cloud services through standardized interfaces for users to call. In scenarios where large model services are deployed through PaaS, users have certain requirements regarding the security of the services provided by the service provider. Summary of the Invention

[0003] This content section is provided to briefly introduce the ideas, which will be described in detail in the examples section later. This content section is not intended to identify key or essential features of the claimed content, nor is it intended to limit the scope of the claimed content.

[0004] Firstly, a deployment method for large model services is provided, including: Obtain first configuration information, which includes configuration information of a first execution environment. The large model service is deployed in a second execution environment, and the security level of the first execution environment is higher than that of the second execution environment. Based on the first configuration information, a first strategy is generated. The first strategy includes the strategy required to obtain the first deployment information, which includes deployment information for deploying the large model service in the first execution environment. Execute the first strategy to obtain the first deployment information; Based on the first deployment information, the large model service is deployed in the first execution environment.

[0005] Secondly, a deployment device for large model services is provided, including: The first acquisition module is configured to acquire first configuration information, which includes configuration information of a first execution environment. The large model service is deployed in a second execution environment, and the security level of the first execution environment is higher than that of the second execution environment. The first generation module is configured to generate a first strategy based on the first configuration information. The first strategy includes the strategy to be executed in order to obtain the first deployment information. The first deployment information includes deployment information for deploying the large model service in the first execution environment. The first execution module is configured to execute the first strategy to obtain the first deployment information; The first deployment module is configured to deploy the large model service in the first execution environment based on the first deployment information.

[0006] Thirdly, a computer-readable medium is provided having a computer program stored thereon, wherein the computer program, when executed by a processing device, implements the steps of the deployment method for the large model service provided in the first aspect.

[0007] Fourthly, an electronic device is provided, comprising: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the deployment method for the large model service provided in the first aspect.

[0008] Fifthly, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, it implements the steps of the deployment method for the large model service provided in the first aspect.

[0009] The above approach deploys the large model service in the second execution environment. To improve the security of the large model service, it needs to be deployed in a more secure first execution environment. Based on the configuration information of the first execution environment, a first policy is generated. This first policy can be understood as an adaptation policy, i.e., the policy to be executed to obtain first deployment information adapted to the first execution environment. The first policy is executed to obtain the first deployment information. Based on this information, the large model service is deployed in the first execution environment. This ensures that the large model service can be deployed and run reliably in the more secure first execution environment, mitigating the problem of the large model service failing to start or malfunctioning after deployment in the first execution environment due to differences in execution environments.

[0010] Other features and advantages will be described in detail in the following examples section. Attached Figure Description

[0011] The above and other features, advantages, and aspects of this document will become more apparent when viewed in conjunction with the accompanying drawings and the following examples. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings: Figure 1 This is a schematic diagram illustrating one application scenario.

[0012] Figure 2 This is a schematic diagram illustrating an example of a reasoning service employing a PD architecture.

[0013] Figure 3 This is a flowchart illustrating an exemplary method for deploying a large model service.

[0014] Figure 4 This is a flowchart illustrating an exemplary method for generating a first policy based on first configuration information.

[0015] Figure 5 This is a flowchart illustrating an exemplary method for deploying a large model service.

[0016] Figure 6 This is a block diagram illustrating an exemplary deployment apparatus for a large model service.

[0017] Figure 7 A schematic diagram of an electronic device suitable for implementing the above method is shown. Detailed Implementation

[0018] The following description will be given in more detail with reference to the accompanying drawings. While certain scenarios are shown in the drawings, it should be understood that this document can be implemented in various forms and should not be construed as limited to the scenarios described herein. Rather, these scenarios are provided to provide a more thorough and complete understanding of this document. It should be understood that the accompanying drawings and the scenarios depicted are for illustrative purposes only and are not intended to limit the scope of this document.

[0019] It should be understood that the steps described in the method may be performed in different orders and / or in parallel. Furthermore, the method may include additional steps and / or omit the steps shown. The scope of this document is not limited in this respect.

[0020] The term "comprising" and its variations can be open-ended, meaning "including but not limited to". The term "based on" can mean "at least partially based on". The term "one case" means "at least one case"; the term "another case" means "at least one additional case"; the term "some cases" means "at least some cases". Definitions of other terms will be given in the following description.

[0021] It should be noted that the concepts of "first" and "second" are only used to distinguish different devices, modules or units, and are not used to limit the order of the functions performed by these devices, modules or units or their interdependencies.

[0022] It should be noted that the modifiers “one” and “multiple” can be illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as “one or more”.

[0023] The names of messages or information exchanged between multiple devices are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0024] It is understandable that the data involved in the technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of relevant regulations.

[0025] MaaS-related tasks can involve three types of roles. The first type is the Cloud Service Provider (CSP), which is mainly used to provide cloud services to different model providers. The second type is the model provider, which can use the resources provided by the cloud service provider to build its own inference service. The third type is the client, which is the entity that can run the applications provided by the cloud service provider and the model provider.

[0026] Large model service refers to Large Language Model (LLM) service. LLM is a natural language processing model that can automatically learn the rules and structure of language, understand the meaning of language, and generate images, videos, and text with correct grammar and semantic coherence based on the understood meaning.

[0027] Large model services can refer to training tasks or inference services for large language models deployed by users. Taking inference services as an example, when the cloud receives an inference request, it can perform inference operations based on the large model service to obtain the inference result. Large model services can be deployed in container groups (Pods), and a large model service can consist of one or more Pods.

[0028] For machine learning platforms, customers can purchase corresponding IaaS (Infrastructure as a Service) resources (such as file systems, networks, ECS (Elastic Compute Service)) from cloud service providers and deploy large model services within these IaaS resources. Customers have certain security requirements for large model services when using application entities provided by model providers and cloud service providers.

[0029] Large-scale model services may not initially run in a highly secure execution environment; that is, they may initially run in a normal execution environment. To ensure service security, customers need to deploy the large-scale model service in a more secure execution environment, where security level refers to data security. In related technologies, if information from a large-scale model service deployed in a normal execution environment is directly transferred to a more secure execution environment, the difference in execution environments may cause the large-scale model service to fail to start or malfunction.

[0030] This paper presents a deployment method for a large model service. The security level of the first execution environment is higher than that of the second execution environment, which can be considered a normal execution environment. The large model service is deployed in the second execution environment. To improve the security of the large model service, it needs to be deployed in the more secure first execution environment. To deploy the large model service in the first execution environment, it is necessary to obtain deployment information adapted to the first execution environment so that the large model service can be successfully deployed without affecting its normal startup and operation.

[0031] The first execution environment can be an execution environment based on a Trusted Execution Environment (TEE), a Trusted Platform Module (TPM), or a Hardware Root of Trust (HROT). The second execution environment can be an execution environment that is not configured with a TEE, TPM, or HROT.

[0032] Figure 1 This is a schematic diagram illustrating one application scenario, such as... Figure 1 As shown, the application scenario includes a first physical machine 11 and a second physical machine 12. The first physical machine 11 is configured with a first execution environment, and the second physical machine 12 is configured with a second execution environment. The large model service has already been deployed in the second execution environment of the second physical machine 12. To improve data security, it is desirable to deploy the large model service in the first execution environment of the first physical machine 11. This can be understood as migrating the large model service from the second execution environment of the second physical machine 12 to the first execution environment of the first physical machine 11. The first execution environment can be a virtual machine (VM) or a bare metal server (BMS) on the first physical machine 11.

[0033] Taking a TEE as the first execution environment as an example, after deploying a large model service in the first execution environment, the TEE where the large model service resides can refer to a hardware-based TEE, which provides data security operations such as encryption and decryption. For example, a hardware-based TEE can include a CPU (Central Processing Unit) TEE and a GPU (Graphics Processing Unit) TEE. A CPU TEE is an isolated, secure operating area integrated within the CPU and provided by the CPU hardware. A GPU TEE is an isolated, secure operating area integrated within the GPU and provided by the GPU hardware. In other words, a hardware-based trusted execution environment can be used to harden the virtual machine used to deploy the large model service, enabling the large model service to run within a TEE. The GPU can be considered an accelerator, but other chips can also be used, such as an NPU (Neural Processing Unit). The following explanation uses a GPU as an example of an accelerator.

[0034] Currently, many large model services are distributed. For example, inference services may use a PD (Prefill Decode) architecture, and training services often employ a distributed approach as well. Distributed large model services are those that provide services by deploying large model instances across multiple physical machines.

[0035] Figure 2 This is a schematic diagram illustrating an example of a reasoning service employing a PD architecture, such as... Figure 2 As shown, the distributed large model service can include multiple large model instances for prefilling and decoding stages to handle inference requests. The controller receives inference requests and, based on current system load and resource usage, appropriately distributes them to the large model instances responsible for the prefilling stage. During the prefilling stage, parallel computation is performed using a parallel runtime in conjunction with multiple GPUs (Graphics Processing Units) to improve the inference speed of the large model instances.

[0036] The initialization results generated in the pre-filling stage (such as attention cache) are passed to the large model instance responsible for the decoding stage in the form of key-value pair data (KV Cache). The large model instance responsible for the decoding stage then generates the final output text sequence step by step based on the initialization results from the pre-filling stage. During the decoding stage, parallel computation using multiple GPUs is employed to improve the inference speed of the large model instance.

[0037] It's important to note that multiple large model instances responsible for the pre-filling stage and multiple large model instances responsible for the decoding stage can be deployed on different physical machines. Furthermore, Figure 2 The distributed large model service shown is for illustrative purposes only. In real-world applications, other types of distributed architectures can also be used for distributed large model services, and no specific limitations are made here.

[0038] For distributed large model services, this can be understood as migrating large model services deployed on multiple physical machines separately. For example, if the large model instance in the pre-population phase is deployed in the normal execution environment of physical machine A, and the large model instance in the decoding phase is deployed in the normal execution environment of physical machine B, the large model instance in the pre-population phase can be deployed in the trusted execution environment of physical machine AA, and the large model instance in the decoding phase can be deployed in the trusted execution environment of physical machine BB. The configurations of physical machine AA and physical machine BB can be the same or different.

[0039] The following describes an example of a deployment method for large model services.

[0040] Figure 3 This is a flowchart illustrating an exemplary method for deploying a large model service, such as... Figure 3 As shown, the deployment method for large model services can specifically be implemented through a large model service deployment device, which can be implemented in software and / or hardware. For example... Figure 3 As shown, the deployment method for the large model service includes steps 31 to 34.

[0041] Step 31: Obtain the first configuration information.

[0042] The first configuration information includes the configuration information of the first execution environment. The large model service is deployed in the second execution environment. The security level of the first execution environment is higher than that of the second execution environment.

[0043] The first configuration information includes, for example, the CPU model, GPU model, operating system distribution and kernel version, GPU driver version, related dependent library versions, and first environment configuration information of the first execution environment. The first environment configuration information can be used to characterize the hardware and version information providing the first execution environment. Based on the first environment configuration information, the environment characteristic information of the first execution environment can be obtained, and this environment characteristic information can be used to indicate operations incompatible with the first execution environment.

[0044] Taking a TEE as the first execution environment as an example, the environment configuration information may include the hardware and version information of the CPU TEE and the GPU TEE. Based on the CPU TEE's hardware and version information, environment characteristic information of the CPU TEE can be obtained. This environment characteristic information, for example, is stored in the form of a knowledge base and can be used to indicate operations incompatible with the CPU TEE. Similarly, based on the GPU TEE's hardware and version information, environment characteristic information of the GPU TEE can be obtained. This environment characteristic information, for example, is stored in the form of a knowledge base and can be used to indicate operations incompatible with the GPU TEE.

[0045] Step 32: Generate a first strategy based on the first configuration information.

[0046] The first strategy includes the strategy required to obtain the first deployment information, which includes deployment information for deploying the large model service in the first execution environment.

[0047] For example, deployment information for large model services includes model weight information, model structure information, container images, source code, pipelines, startup scripts, environment variables, dependency libraries, and service configuration information.

[0048] The information includes: Model weight information (such as neuron weights and attention parameters); Model structure information (indicating the network topology of the large model used by the service); Container image (files related to the service, instantiated as a container within a Pod to package the user-provided service image into a standardized unit for deployment); Source code (the code implementing the service); Pipeline (indicating how to compile and build the container image, which can be seen as the product of the pipeline); Startup script (script used to start or launch the service, containing entry commands); Environment variables (dynamic configuration parameters for the service's runtime); Dependency libraries (software dependencies of the service's execution environment); and Service configuration information (e.g., computing resource configuration and memory access configuration).

[0049] The second deployment information includes the deployment information of the large model service in the second execution environment. Both the first and second deployment information can include the deployment information of the large model service in the example above, and the first and second are used to distinguish the deployment information of different execution environments.

[0050] The first strategy includes the strategy required to obtain the first deployment information. For example, if certain operations are not supported in the first execution environment, the first strategy can be used to indicate how to adjust the second deployment information to obtain the first deployment information adapted to the first execution environment. If the first deployment information is adapted to the first execution environment, then based on the first deployment information, the large model service is deployed in the first execution environment, and the large model service can be successfully deployed and start and run normally.

[0051] Step 33: Execute the first strategy to obtain the first deployment information.

[0052] Step 34: Based on the first deployment information, deploy the large model service in the first execution environment.

[0053] Using the above method, the large model service is deployed in the second execution environment. To improve the security of the large model service, it needs to be deployed in the first execution environment, which has a higher level of security. Based on the configuration information of the first execution environment, a first policy is generated. The first policy can be understood as an adaptation policy, that is, the policy to be executed in order to obtain the first deployment information adapted to the first execution environment. Executing the first policy yields the first deployment information. Based on the first deployment information, the large model service is deployed in the first execution environment. In this way, the large model service can be deployed and run reliably in the first execution environment with a higher level of security, mitigating the problem of the large model service failing to start or running abnormally after deployment in the first execution environment due to differences in execution environments.

[0054] Figure 4 This is a flowchart illustrating an exemplary method for generating a first policy based on first configuration information, such as... Figure 4 As shown, step 32 may include steps 41 and 42.

[0055] Step 41: Based on the first configuration information and the second configuration information, obtain the difference information. The second configuration information includes the configuration information of the second execution environment.

[0056] Step 42: Generate the first strategy based on the difference information.

[0057] The difference information can be used to show the differences between the first execution environment and the second execution environment. Based on these differences, a first policy is generated. The first policy can be used to indicate how to adjust the second deployment information to obtain the first deployment information adapted to the first execution environment.

[0058] For example, the difference information includes first difference information and / or second difference information. The first difference information is used to indicate a first operation, which includes an operation that is compatible with the second execution environment but incompatible with the first execution environment. The second difference information is used to indicate configuration differences between the first execution environment and the second execution environment. The configuration differences may include software configuration differences and hardware configuration differences.

[0059] The second configuration information includes, for example, the CPU model, GPU model, operating system distribution and kernel version, GPU driver version, related dependency library versions, and second environment configuration information of the second execution environment. This second environment configuration information can be used to characterize the hardware and version information corresponding to the second execution environment. Based on this information, the environment characteristic information of the second execution environment can be obtained, and this characteristic information can be used to indicate compatible operations for the second execution environment.

[0060] Thus, based on operations compatible with the second execution environment and operations incompatible with the first execution environment, the first difference information can be obtained.

[0061] The operating system distribution and kernel version, GPU driver version, and related dependency library versions are software configuration information for the execution environment, while the CPU model and GPU model are hardware configuration information for the execution environment. Taking dependency library versions as an example, the dependency libraries corresponding to the first execution environment are different from those corresponding to the second execution environment, representing a software configuration difference between the two execution environments.

[0062] For example, the difference information includes first difference information, and a first policy is generated based on a first operation indicated by the first difference information. Alternatively, the difference information includes second difference information, and a first policy is generated based on configuration differences indicated by the second difference information. Or, the difference information includes both first and second difference information, and a first policy is generated based on a first operation indicated by the first difference information and configuration differences indicated by the second difference information.

[0063] Optionally, the difference information includes first difference information; step 42 may include: From the second deployment information, identify the first content, which includes the content corresponding to the first operation, and the second deployment information includes the deployment information of the large model service deployed in the second execution environment; A first adjustment strategy is generated, which is used to adjust the first content. The first strategy includes the first adjustment strategy.

[0064] Specifically, the second deployment information can be compared with the first operation. For example, by analyzing the source code and service configuration information in the second deployment information, the first content can be identified. The first content includes the content corresponding to the first operation. The first content in the second deployment information can be understood as an operation that the large model service can use in the second execution environment but is incompatible or unsupported in the first execution environment.

[0065] The first adjustment strategy is used to adjust the first content, thereby disabling or replacing operations that are incompatible or unsupported by the first execution environment with corresponding operations that are compatible with the first execution environment.

[0066] Through the above scheme, because the first execution environment has a higher level of security, to ensure data security, it typically does not support some operations that affect data security. The first operation includes operations that are compatible with the second execution environment but incompatible with the first execution environment. The first operation can be understood as an operation that the large model service can use in the second execution environment but not in the first execution environment. The first adjustment strategy is used to adjust the first content, thereby adjusting operations that are incompatible or unsupported in the first execution environment, so that the first deployment information is adapted to the first execution environment. This avoids the problem of the large model service being unable to be deployed in the first execution environment due to operations that are not supported by the first execution environment.

[0067] In one embodiment, the first operation includes a first interface call operation, the second deployment information includes the source code of the large model service, and the first content includes the content corresponding to the first interface call operation in the source code.

[0068] The first interface call operation can refer to an API (Application Programming Interface) call operation that is incompatible with the first execution environment. To ensure data security, the first execution environment may not support calls to some APIs, such as performance counter API calls. If the first interface call operation appears in the source code of the large model service in the second deployment information, the corresponding content in the source code can be adjusted. This adjustment can include, for example, commenting out the content corresponding to the first interface call operation in the source code (i.e., invalidating this part of the code), or replacing the content corresponding to the first interface call operation with the corresponding interface call operation compatible with the first execution environment.

[0069] Thus, by implementing the first adjustment strategy, the source code in the second deployment information can be adjusted, and the adjusted source code can be used as the source code in the first deployment information, thereby obtaining the source code of the large model service adapted to the first execution environment.

[0070] In one embodiment, the first operation includes a first memory access operation, the second deployment information includes service configuration information of the large model service, and the first content includes the content corresponding to the first memory access operation in the service configuration information.

[0071] For example, to ensure data security, the first execution environment may not support certain forms of memory access operations. Taking Remote Direct Memory Access (RDMA) as an example, when deploying a large model service in the second execution environment, RDMA memory access might be used to accelerate computation. The first adjustment strategy can be used to adjust the service configuration information of the large model service in the second deployment information. This adjustment, for example, is replacing it with a memory access method supported by the first execution environment.

[0072] Thus, by executing the first adjustment strategy, the service configuration information in the second deployment information can be adjusted, and the adjusted service configuration information can be used as the service configuration information in the first deployment information, thereby obtaining the service configuration information of the large model service adapted to the first execution environment.

[0073] Optionally, the difference information includes second difference information; step 42 may include: A second adjustment strategy is generated based on the second difference information. The first strategy includes the second adjustment strategy.

[0074] The second adjustment strategy is used to adjust the second content in the second deployment information. The second content includes content related to the execution environment of the large model service. The second deployment information includes the deployment information of the large model service deployed in the second execution environment.

[0075] For example, content related to the execution environment of a large model service may include environment variables, dependency libraries, startup scripts, container images, and pipelines.

[0076] Environment variables may include dynamic configuration parameters for the large model service runtime. Since the hardware and software configurations of the first execution environment and the second execution environment are different, the second adjustment strategy can be used to adjust the environment variables to adapt them to the hardware and software configurations of the first execution environment.

[0077] The dependency library refers to the software dependency library of the execution environment of the large model service. The second adjustment strategy can be used to adjust the dependency library to adopt a software library that is compatible with the software configuration of the first execution environment.

[0078] The startup script is used to start the large model service. The way to start the large model service is different in different execution environments, such as different file paths. The second adjustment strategy can be used to adjust the startup script to obtain a startup command that is compatible with the first execution environment.

[0079] The pipeline is used to indicate how to compile and build container images. Different hardware and software configurations result in different container image building methods. The second tuning strategy can be used to adjust the pipeline to obtain a pipeline that is adapted to the first execution environment.

[0080] The second adjustment strategy can also be used to instruct the container image in the second deployment information to be updated or recompiled and built based on the adjusted pipeline to obtain the container image in the first deployment information.

[0081] Thus, based on the second difference information, a second adjustment strategy is generated. The second adjustment strategy is used to adjust the content related to the execution environment of the large model service in the second deployment information, so as to obtain environment deployment information such as container images, startup scripts, and dependency libraries that are adapted to the first execution environment.

[0082] In an embodiment where the difference information includes first difference information and second difference information, first content can be identified from the second deployment information to generate a first adjustment strategy, and a second adjustment strategy can be generated based on the second difference information.

[0083] By adopting the above technical solution, the source code of the large model service in the second deployment information can be adjusted so that no interface call operations incompatible with the first execution environment appear in the first deployment information. The service configuration information of the large model service in the second deployment information can be adjusted so that no content access operations incompatible with the first execution environment appear in the first deployment information. Furthermore, the content related to the execution environment of the large model service in the second deployment information can be adjusted so that the execution environment corresponding to the first deployment information is compatible with the first execution environment.

[0084] In one embodiment, generating a first strategy based on the difference information may include: A first strategy is generated using a first machine learning model, the input of which includes the new difference information and a first reference adjustment strategy, the first reference adjustment strategy being used to indicate a reference strategy for adjusting the second deployment information.

[0085] The first machine learning model can be any pre-trained first language model. For example, the step of generating the first policy can be performed by an agent. The agent can query a knowledge base that records historical adjustment policies obtained based on differences in different execution environments. The agent can identify adjustment policies with similar difference information from the knowledge base as the first reference adjustment policy. Based on the difference information and the first reference adjustment policy, the agent can construct prompt words for the first machine learning model, call the first machine learning model, and obtain the first policy. These prompt words are used to guide the first machine learning model to generate the first policy, and the first reference adjustment policy can serve as reference content for the first machine learning model. The first machine learning model can identify first content from second deployment information and generate the first adjustment policy based on the first content, as well as generate a second adjustment policy based on the second difference information.

[0086] In this embodiment, an agent can generate a first strategy based on difference information. The first strategy can be in the form of a structured instruction set to automate the deployment process of large model services.

[0087] In one embodiment, step 32 may include: Based on the first configuration information, a first acquisition strategy is generated. The first acquisition strategy is used to indicate the acquisition of the third content corresponding to the first execution environment, and the first deployment information is obtained based on the third content. The third content includes template content related to the execution environment of the large model service. The template content includes at least one of the following: template content of the startup script, template content of the environment variables, and template content of the pipeline. The pipeline is used to build the container image of the large model service.

[0088] In this embodiment, considering that at least one of the startup script, environment variables, and pipeline adapted to the first execution environment can be templated, that is, at least one of the template content of the startup script, environment variables, and pipeline adapted to the first execution environment can be pre-built. Thus, in an optional embodiment, one or more of the startup script, environment variables, and pipeline can be obtained based on the template content without adjusting the second deployment information, thereby improving the generation efficiency of the first deployment information.

[0089] For example, template content corresponding to different execution environments can be pre-built. Based on the first configuration information of the first execution environment, a first acquisition strategy can be generated. The first acquisition strategy is used to indicate the acquisition of the third content corresponding to the first execution environment. Based on the third content, the content related to the large model service in the third content is populated to obtain the first deployment information. For example, based on the template content of the startup script, the startup script of the large model service in the first deployment information can be obtained.

[0090] Based on the template content of the pipeline, after obtaining the pipeline in the first deployment information, the container image in the first deployment information can be compiled and built according to the pipeline in the first deployment information.

[0091] Through the above technical solution, deployment information related to the execution environment can be templated. The first deployment information can be obtained based on the template content, eliminating the need to update or adjust the second deployment information, thus improving the generation efficiency of the first deployment information.

[0092] In this embodiment, in addition to startup scripts, environment variables, pipelines, and container images, source code and service configuration information can be obtained through the adjustment strategies described above.

[0093] In addition, the model weight information in the second deployment information can be used as the model weight information in the first deployment information, and the model structure information in the second deployment information can be used as the model structure information in the first deployment information. That is, when the execution environment is changed, the model weights and structure can remain unchanged.

[0094] The above describes the implementation method for obtaining the first deployment information. Based on the first deployment information, a large model service can be deployed in the first execution environment. Figure 5 This is a flowchart illustrating an exemplary method for deploying a large model service, such as... Figure 5 As shown, the deployment method for the large model service may include steps 31 to 37. Steps 31 to 34 have been described above.

[0095] Step 35: Run the large model service in the first execution environment.

[0096] For example, the large model service can be started or launched in the first execution environment through the startup script in the first deployment information. After the large model is successfully started, the large model service can be run.

[0097] Step 36: Obtain runtime information.

[0098] For example, the runtime information includes at least one of the following: runtime log information, runtime security information, functional test results of the large model service, and performance evaluation results of the large model service.

[0099] During the startup and operation of the large model service, operation log information and operation security information can be obtained. The operation log information can be used to capture abnormal information related to the interaction between the large model service and hardware, driver calls, memory allocation and cross-node communication. The operation security information, for example, indicates whether abnormalities have occurred in data security operations such as encryption and decryption during the operation of the large model service.

[0100] For example, a standardized test suite can be used to test the functionality of the large model service. The output of the large model service deployed in the first execution environment is compared with the baseline results of the large model service in the second execution environment to determine whether the large model service can provide accurate results in the first execution environment. In addition, functional testing can also include API consistency testing. In the first execution environment, the API endpoints provided by the large model service are called to verify whether its request / response format, status codes, and other behaviors are consistent with the specifications.

[0101] For example, a standardized performance test load is performed on the large model service in the first execution environment, key performance indicators are measured, and compared with baseline values ​​measured in the first execution environment. Performance evaluation indicators include, for example, first-to-first-token latency (TTFT), time per output token (TPOT), throughput, and goodput.

[0102] First-word latency measures the efficiency of the Prefill phase, while time per output token measures the efficiency of the Decode phase. Throughput measures the number of requests processed or tokens generated per unit of time; effective throughput is a comprehensive metric combining throughput and service quality (such as latency).

[0103] Step 37: Based on the runtime information, obtain the deployment result information. The deployment result information indicates whether the large model service has been successfully deployed in the first execution environment.

[0104] As an example, if no abnormalities occur during operation, the functional test results indicate that the large model service functions normally. If the key performance indicators are compared with the baseline values ​​measured in the first execution environment, and the quantified performance degradation rate does not exceed the preset percentage, the large model service can be considered to have been successfully deployed in the first execution environment.

[0105] In this way, deployment result information can be obtained based on the running information of the large model service in the first execution environment, so as to determine whether the large model service has been successfully deployed in the first execution environment from the dimensions of running logs, running security, functional testing, and performance evaluation.

[0106] Alternatively, the deployment methods for large model services may also include: In response to the deployment result information indicating that the large model service was not successfully deployed in the first execution environment, a second policy is generated based on the runtime information. The second policy is used to indicate adjustments to the third deployment information, which includes the deployment information of the large model service in the first execution environment. Execute the second strategy to obtain the fourth deployment information; Based on the fourth deployment information, deploy the large model service in the first execution environment.

[0107] Afterwards, you can return to the step of running the large model service in the first execution environment until the number of iterations reaches the preset number, or the deployment result information indicates that the large model service has been successfully deployed in the first execution environment.

[0108] The third deployment information could be the deployment information for the large model service deployed in the first execution environment during this iteration. For example, based on the runtime information, the reasons why the large model service failed to be deployed successfully in the first execution environment can be analyzed. For instance, the agent can attribute the failure to deployment and generate a second strategy. The agent can use a rule engine, pattern matching, or machine learning model for attribution.

[0109] For example, if the agent determines that the time per output lexical term of the large model service is higher than the preset time, and a large number of memory swapping-related warnings appear in the operation log information, it can be inferred that the anomaly is in the management of KV Cache or the memory bandwidth bottleneck. The second strategy can be used to improve the memory allocation of the large model service.

[0110] For example, if the agent analyzes that the cause is the obstruction of cross-GPU communication under the PD architecture, the second strategy can be used to schedule the pre-filling stage and the decoding stage to the same physical machine to take advantage of high-speed internal communication, as well as to optimize the communication channel between physical machine AA and physical machine BB in software.

[0111] The second strategy can be seen as a new adaptation strategy to better enable the large model service to run in the first execution environment. Executing the second strategy yields fourth deployment information, which is then used to deploy the large model service in the first execution environment. In each iteration, the agent can record detailed analysis reports and the strategy generation process. If the automated process iterates to a preset number of times and the large model service still fails to deploy successfully in the first execution environment, it can output a prompt message to the developers and provide a complete analysis report and strategy generation process for manual intervention in debugging and decision-making.

[0112] In one embodiment, generating a second strategy based on runtime information may include: A second strategy is generated using a second machine learning model, the input of which includes the operational information and a second reference adjustment strategy, which is used to indicate a reference strategy for adjusting the third deployment information.

[0113] The second machine learning model can be any pre-trained second-largest language model; the first and second-largest language models can be the same or different. For example, the step of generating a second policy based on runtime information can be executed by an agent. The agent can query a knowledge base that records historical adjustment policies obtained based on different runtime information. The agent can find adjustment policies with similar runtime information from this knowledge base as a second reference adjustment policy. Based on the runtime information and the second reference adjustment policy, the agent can construct prompt words for the second machine learning model, call the second machine learning model, and obtain the second policy. These prompt words are used to guide the second machine learning model in generating the second policy, and the second reference adjustment policy can serve as reference content for the second machine learning model.

[0114] In this embodiment, the agent generates a second strategy based on operational information. The second strategy can be in the form of a structured instruction set to automate the deployment process of large model services and improve the deployment efficiency of large model services.

[0115] By employing the above technical solution, through difference analysis and strategy generation, automated deployment and debugging, quantitative evaluation and iterative optimization, efficient adaptation of large model services to the primary execution environment can be achieved. This can be understood as an efficient and reliable migration of large model services to the primary execution environment. The automated deployment process can improve the deployment efficiency and success rate of large model services.

[0116] Based on the same inventive concept, a deployment device for large-scale model services is also provided. Figure 6 This is a block diagram illustrating an exemplary deployment apparatus for a large model service, such as... Figure 6 As shown, the deployment device 60 for the large model service includes: The first acquisition module 61 is configured to acquire first configuration information, which includes configuration information of a first execution environment. The large model service is deployed in a second execution environment, and the security level of the first execution environment is higher than that of the second execution environment. The first generation module 62 is configured to generate a first strategy based on the first configuration information. The first strategy includes a strategy to be executed to obtain the first deployment information. The first deployment information includes deployment information for deploying the large model service in the first execution environment. The first execution module 63 is configured to execute the first strategy and obtain the first deployment information; The first deployment module 64 is configured to deploy the large model service in the first execution environment based on the first deployment information.

[0117] The deployment architecture for the large model service employs a first execution environment with a higher security level than the second execution environment. The large model service is initially deployed in the second execution environment. To improve its security, it needs to be deployed in the more secure first execution environment. Based on the configuration information of the first execution environment, a first policy is generated. This first policy can be understood as an adaptation policy—the policy to be executed to obtain deployment information adapted to the first execution environment. Executing the first policy yields the first deployment information. Based on this information, the large model service is deployed in the first execution environment. This ensures the large model service can operate reliably in the more secure first execution environment, mitigating the problem of the service failing to start or malfunctioning due to differences in execution environments.

[0118] Optionally, the first generation module 62 includes: The difference acquisition module is configured to obtain difference information based on the first configuration information and the second configuration information, wherein the second configuration information includes the configuration information of the second execution environment; The first generation submodule is configured to generate a first strategy based on the difference information.

[0119] Optionally, the difference information includes first difference information, which is used to indicate a first operation, the first operation including an operation that is compatible with the second execution environment but incompatible with the first execution environment; The first generation submodule is configured as follows: From the second deployment information, identify the first content, which includes the content corresponding to the first operation, and the second deployment information includes the deployment information of the large model service deployed in the second execution environment; A first adjustment strategy is generated, which is used to adjust the first content.

[0120] Optionally, the first operation includes a first interface call operation, the second deployment information includes the source code of the large model service, and the first content includes the content corresponding to the first interface call operation in the source code.

[0121] Optionally, the first operation includes a first memory access operation, the second deployment information includes the service configuration information of the large model service, and the first content includes the content corresponding to the first memory access operation in the service configuration information.

[0122] Optionally, the difference information includes second difference information, which is used to indicate the configuration differences between the first execution environment and the second execution environment; The first generation module 62 includes: The second generation submodule is configured to generate a second adjustment strategy based on the second difference information. The second adjustment strategy is used to adjust the second content in the second deployment information. The second content includes content related to the execution environment of the large model service. The second deployment information includes the deployment information of the large model service deployed in the second execution environment.

[0123] Optionally, the first generation module 62 is configured as follows: Based on the first configuration information, a first acquisition strategy is generated. The first acquisition strategy is used to indicate the acquisition of the third content corresponding to the first execution environment, and the first deployment information is obtained based on the third content. The third content includes template content related to the execution environment of the large model service. The template content includes at least one of the following: template content of startup script, template content of environment variables, and template content of pipeline. The pipeline is used to build the container image of the large model service.

[0124] Optionally, the deployment device 60 further includes: The runtime module is configured to run the large model service in the first execution environment; The second acquisition module is configured to acquire runtime information; The result acquisition module is configured to obtain deployment result information based on the running information. The deployment result information is used to indicate whether the large model service has been successfully deployed in the first execution environment.

[0125] Optionally, the deployment device 60 further includes: The second generation module is configured to, in response to the deployment result information indicating that the large model service has not been successfully deployed in the first execution environment, generate a second strategy based on the running information. The second strategy is used to indicate adjustments to the third deployment information, which includes the deployment information of the large model service in the first execution environment. The second execution module is configured to execute the second strategy to obtain the fourth deployment information; The second deployment module is configured to deploy the large model service in the first execution environment based on the fourth deployment information.

[0126] Optionally, the operational information includes at least one of the following: operational log information, operational security information, functional test results of the large model service, and performance evaluation results of the large model service.

[0127] For the effects that each module in the deployment device 60 of the large model service can achieve, please refer to the effect description of the corresponding step in the deployment method of the large model service.

[0128] The following is for reference. Figure 7The diagram illustrates a structural schematic of an electronic device 600 suitable for implementing the above-described method. The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Personal Computers), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs (Televisions), desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not be construed as limiting its functionality or scope of use.

[0129] like Figure 7 As shown, electronic device 600 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. The random access memory 603 also stores various programs and data required for the operation of electronic device 600. The processing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0130] Typically, the following devices can be connected to the input / output interface 605: input devices 606 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 608 including, for example, magnetic tape, hard disk, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0131] In particular, depending on certain circumstances, the processes described in the flowchart above can be implemented as computer software programs. For example, a computer program product is provided, comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. This computer program can be downloaded and installed from a network via communication device 609, or installed from storage device 608, or installed from read-only memory 602. When the computer program is executed by processing device 601, it performs the functions defined in the above-described methods.

[0132] It should be noted that the aforementioned computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM, or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In one case, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In another case, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.

[0133] In some scenarios, clients and servers can communicate using any currently known or future-developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include Local Area Networks (LANs), Wide Area Networks (WANs), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad-hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0134] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0135] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: obtain first configuration information, the first configuration information including configuration information of a first execution environment, the large model service being deployed in a second execution environment, and the security level of the first execution environment being higher than that of the second execution environment; Based on the first configuration information, a first strategy is generated. The first strategy includes the strategy required to obtain the first deployment information, which includes deployment information for deploying the large model service in the first execution environment. Execute the first strategy to obtain the first deployment information; Based on the first deployment information, the large model service is deployed in the first execution environment.

[0136] Computer program code for performing the above operations can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages, as well as conventional procedural programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0137] The flowcharts and block diagrams in the accompanying figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products under various scenarios. Each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that, in some alternative cases, the functions indicated in the blocks may occur in a different order than those indicated in the figures. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0138] The modules mentioned above can be implemented in software or hardware. In some cases, the name of a module does not necessarily limit the module itself; for example, the first acquisition module can also be described as "the module that acquires the first configuration information".

[0139] The functions described above can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field-Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application-Specific Standard Parts (ASSPs), Systems on Chips (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0140] In this context, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0141] The above description is merely illustrative and illustrates the technical principles employed. Those skilled in the art should understand that the scope of this document is not limited to the specific combinations of the above-described technical features, but should also cover any combination of the above-described technical features or their equivalents without departing from the above concept.

[0142] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Multitasking and parallel processing may be advantageous in certain contexts. Similarly, while some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of this paper. Certain features described in the context of a single example can also be implemented in combination in a single example. Conversely, various features described in the context of a single example can also be implemented individually or in any suitable sub-combination in multiple examples.

[0143] Although this document has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims. Regarding the aforementioned apparatus, the specific manner in which the various modules perform their operations has already been described in detail in the section concerning the method, and will not be elaborated upon here.

Claims

1. A method for deploying a large model service, comprising: Obtain first configuration information, which includes configuration information of a first execution environment. The large model service is deployed in a second execution environment, and the security level of the first execution environment is higher than that of the second execution environment. Based on the first configuration information, a first strategy is generated. The first strategy includes the strategy required to obtain the first deployment information, which includes deployment information for deploying the large model service in the first execution environment. Execute the first strategy to obtain the first deployment information; Based on the first deployment information, the large model service is deployed in the first execution environment.

2. The deployment method for large model services according to claim 1, wherein generating a first strategy based on the first configuration information includes: Based on the first configuration information and the second configuration information, difference information is obtained, wherein the second configuration information includes the configuration information of the second execution environment; Based on the difference information, a first strategy is generated.

3. The deployment method for large model services according to claim 2, wherein the difference information includes first difference information, the first difference information being used to indicate a first operation, the first operation including an operation compatible with the second execution environment and incompatible with the first execution environment; The step of generating a first strategy based on the difference information includes: From the second deployment information, identify the first content, which includes the content corresponding to the first operation, and the second deployment information includes the deployment information of the large model service deployed in the second execution environment; A first adjustment strategy is generated, which is used to adjust the first content.

4. The deployment method of the large model service according to claim 3, wherein the first operation includes a first interface call operation, the second deployment information includes the source code of the large model service, and the first content includes the content corresponding to the first interface call operation in the source code.

5. The deployment method for a large model service according to claim 3, wherein the first operation includes a first memory access operation, the second deployment information includes service configuration information of the large model service, and the first content includes the content corresponding to the first memory access operation in the service configuration information.

6. The deployment method for a large model service according to claim 2, wherein the difference information includes second difference information, the second difference information being used to indicate configuration differences between the first execution environment and the second execution environment; The step of generating a first strategy based on the difference information includes: Based on the second difference information, a second adjustment strategy is generated. The second adjustment strategy is used to adjust the second content in the second deployment information. The second content includes content related to the execution environment of the large model service. The second deployment information includes the deployment information of the large model service deployed in the second execution environment.

7. The deployment method for large model services according to claim 1, wherein generating a first strategy based on the first configuration information includes: Based on the first configuration information, a first acquisition strategy is generated. The first acquisition strategy is used to indicate the acquisition of the third content corresponding to the first execution environment, and the first deployment information is obtained based on the third content. The third content includes template content related to the execution environment of the large model service. The template content includes at least one of the following: template content of startup script, template content of environment variables, and template content of pipeline. The pipeline is used to build the container image of the large model service.

8. The deployment method for large model services according to claim 1, further comprising: The large model service is run in the first execution environment; Obtain runtime information; Based on the operational information, deployment result information is obtained, which is used to indicate whether the large model service has been successfully deployed in the first execution environment.

9. The deployment method for large model services according to claim 8, further comprising: In response to the deployment result information indicating that the large model service was not successfully deployed in the first execution environment, a second strategy is generated based on the running information. The second strategy is used to indicate adjustments to the third deployment information, which includes the deployment information of the large model service in the first execution environment. Execute the second strategy to obtain the fourth deployment information; Based on the fourth deployment information, the large model service is deployed in the first execution environment.

10. The deployment method of the large model service according to claim 8 or 9, wherein the operation information includes at least one of the following: operation log information, operation security information, functional test results of the large model service, and performance evaluation results of the large model service.

11. A deployment apparatus for a large model service, comprising: The first acquisition module is configured to acquire first configuration information, which includes configuration information of a first execution environment. The large model service is deployed in a second execution environment, and the security level of the first execution environment is higher than that of the second execution environment. The first generation module is configured to generate a first strategy based on the first configuration information. The first strategy includes the strategy to be executed in order to obtain the first deployment information. The first deployment information includes deployment information for deploying the large model service in the first execution environment. The first execution module is configured to execute the first strategy to obtain the first deployment information; The first deployment module is configured to deploy the large model service in the first execution environment based on the first deployment information.

12. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by the processing device, it implements the steps of the method according to any one of claims 1 to 10.

13. An electronic device, comprising: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images