Serverless large model reasoning service system, method, equipment and medium

Abstract

The embodiment of the invention relates to the field of large model facilities, and discloses a Serverless large model reasoning service system, method and equipment and a medium. Receiving an inference service creation request sent by a user, and forwarding the inference service creation request to a corresponding cluster; processing the north-south traffic of the cluster; converting the reasoning service creation request into an object, and configuring a running environment; according to the object, a Pod is created, the Pod is used for controlling a model service workload, the Pod comprises a user container, and the user container is constructed based on a basic mirror image comprising an operating system, a Supervisor and a Docker and is used for running a large model service; and the queue agent is responsible for processing routing, load balancing and request queuing of an incoming request based on a lightweight agent Sidecar. The technical problem that large model reasoning service deployment is difficult can be solved at least.

Description

Technical Field

[0001] The present application relates to the field of large model facilities, and in particular to a Serverless large model reasoning service system, method, device and medium. Background Art

[0002] With the rapid development of artificial intelligence technology, large-scale pre-trained models (such as GPT-3 and GLM-130B) have shown extensive application potential in multiple fields. These models have the characteristics of large parameter scale and high computational complexity, which puts higher requirements on reasoning performance, deployment flexibility, and operating environment adaptability. At present, the deployment methods of large models mainly include physical machine deployment and containerized deployment. Physical machine deployment relies on high-performance hardware and strictly matched operating systems and driver environments. Although containerized deployment improves portability through standardized images, it still has complexity issues when adapting to heterogeneous hardware and operating systems in multi-cloud environments.

[0003] As a popular container orchestration technology, Kubernetes has become an important foundation for deploying large model inference services with its high scalability and cross-cloud support capabilities. However, the traditional deployment method of large models faces significant technical challenges when combined with the Serverless architecture based on Kubernetes. Large model providers usually develop deployment scripts for specific physical machines or container environments. These scripts rely on specific operating systems and software environments, which are quite different from the serverless architecture of Kubernetes and Knative. As a result, deployment scripts cannot be directly reused and require a complex adaptation process.

[0004] In addition, the heterogeneity of nodes in a multi-cloud environment makes it more difficult to deploy and adapt large models. Different nodes may use different operating systems, instruction sets (such as ARM and x86), driver versions, and dependent environments, which increases the workload and time cost of deployment and debugging. In this context, traditional deployment methods are difficult to shield the underlying environment differences, hindering the rapid deployment of large models. At the same time, traditional deployment methods lack elastic scaling capabilities when dealing with dynamic inference loads, and cannot meet the real-time and high concurrency requirements of large-scale inference services. Summary of the invention

[0005] One purpose of the present application is to provide a Serverless large-model reasoning service system, method, device and medium, at least to solve the technical problem of difficulty in deploying large-model reasoning services.

[0006] To achieve the above objectives, some embodiments of the present application provide the following aspects: In the first aspect, some embodiments of the present application also provide a Serverless large model inference service system including a public / private DNS service, which is used to receive inference service creation requests issued by users and forward them to the corresponding Kubernetes cluster; a load balancer, which is used to process the north-south traffic of the Kubernetes cluster; an inference cloud control plane component, which is used to receive the inference service creation request, convert the inference service creation request into a Knative Service object, and configure the operating environment; create a Knative Service Pod according to the Knative Service object, and the Knative Service Pod is used to control the model service workload including: a user container, which is built based on a basic image including an operating system, Supervisord and Dockerd, and is used to run a large model service; a queue agent, which is responsible for handling the routing, load balancing and request queuing of incoming requests based on the lightweight agent Sidecar.

[0007] On the second aspect, some embodiments of the present application also provide a Serverless large model inference service method, including receiving a large model inference service creation request submitted by a user through a public or private DNS service, and forwarding the large model inference service creation request to a target Kubernetes cluster; generating a Knative Service object according to the large model inference service creation request through an inference cloud control plane component, and configuring the operating environment, including mounting model files, setting acceleration card devices, and log directories; creating a Knative Service Pod through the Knative Service object, and the Pod includes a user container and a queue agent, wherein: the user container runs a basic image based on an operating system, Supervisord, and Dockerd, and is used to start a large model inference service; the queue agent handles the routing, load balancing, and request queuing of incoming requests; runs a container daemon in the user container through the Nested Containers technology, and executes the model provider's deployment script to start the large model service; dynamically adjusts the number of Pod copies according to the service load through Knative to achieve elastic expansion and reduction of the inference service; exposes the inference service API address through a load balancer and DNS service for users to call the large model inference function.

[0008] In a third aspect, some embodiments of the present application further provide an electronic device, comprising: one or more processors; and a memory storing computer program instructions, wherein the computer program instructions, when executed, cause the processor to perform the steps of the method described above.

[0009] In a fourth aspect, some embodiments of the present application further provide a computer-readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method as described above.

[0010] Compared with the related technology, the solution provided in the embodiment of the present application is a technical solution for quickly deploying large models as elastic reasoning services based on a Serverless architecture based on Kubernetes. It solves the core problems in the existing technology through standardized basic images and Nested Containers technology, and can directly reuse the deployment scripts of large model providers to reduce adaptation costs; it can shield the heterogeneity of host machines in multi-cloud environments to achieve rapid adaptation; it can provide elastic scaling functions through the Serverless architecture to meet dynamic load requirements; and it can improve the degree of automation of deployment, realize one-click deployment of large model reasoning services, significantly reduce deployment costs, and improve operating efficiency and adaptability in multi-cloud environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0012] Figure 1 A schematic diagram of the structure of a Serverless large model inference service system provided according to an embodiment of the present application; Figure 2 A schematic diagram of the structure of a large model inference host provided according to an embodiment of the present application; Figure 3 A schematic diagram of a process for deploying a large model reasoning service according to an embodiment of the present application; Figure 4 The present invention is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solution and advantages of this embodiment clearer, the technical solution in the embodiment of this application will be clearly and completely described below in conjunction with the drawings in the embodiment of this application. Obviously, the described embodiment is a part of the embodiment of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0014] The following terms are used in this article.

[0015] Kubernetes: An open source container orchestration platform.

[0016] Docker: It is built on the operating system-level virtualization technology (i.e., container technology), and uses the Linux kernel's cgroups and namespaces functions to achieve resource isolation and process isolation, providing a lightweight, portable application environment.

[0017] Knative: Knative is built on Kubernetes, using its container orchestration capabilities and combining it with the concept of serverless computing to provide automatic expansion, event-driven, and simplified application deployment capabilities, aiming to make it easier for developers to build and run cloud-native applications.

[0018] Supervisord: A general-purpose process management tool developed based on Python. It can manage and monitor multiple processes in Linux / Unix environments, provide automatic restart, log capture and centralized control, and is often used for the management of background services and daemons.

[0019] Nested Containers: Starting a Docker container within a Docker container, and starting a Docker container within a Containerd container is called container nesting, which is generally used to isolate the development environment and CI / CD links.

[0020] Kubernetes (a container orchestration technology) is becoming increasingly popular as the operating environment for microservice applications. A variety of Serverless frameworks built on it, such as Knative, have emerged. More and more large model applications choose to be deployed in the Kubernetes environment for reasoning. However, many large model providers, such as the domestic large model (GLM-130B), focus on large model optimization and parameter increase. Large model deployment is mainly based on physical machine deployment and containerized deployment. How to quickly deploy the physical machine deployment scripts or containerized (Docker-based) deployment scripts provided by the model vendor as elastically scalable Serverless inference services on public clouds, private clouds, hybrid clouds, and edge clouds is a difficult problem in the industry.

[0021] Currently, in the field of large-model Serverless private deployment, the existing technology is to manually rewrite the deployment script into a Serverless model service script based on Kubernetes Knative based on the physical machine deployment script or containerized deployment script provided by the large model vendor in a multi-cloud Kubernetes environment where Knative components are deployed. The shortcomings of this solution are as follows: It is impossible to reuse the deployment scripts of large model providers because most of their deployment scripts are based on physical machines / Docker containers. The environment checks and environment operation dependencies in the scripts are all based on specific operating systems and software. Since these dependencies are missing in the Knative environment of Kubernetes and the way of orchestrating containers is inconsistent, these two deployment methods cannot be used directly in the Knative environment based on Kubernetes. Complex semantic conversion costs are required to support deployment as Knative's Serverless service.

[0022] The deployment cost of large-model Serverless services is high because there are many large-model providers. Different large models provided by each large-model provider have different deployment methods. Some are deployed on physical machines, and some are containerized deployments based on Docker. As an inference service platform, it is necessary to convert the unified semantics of these model deployments to Serverless platform deployments, and subsequent model iterations will also require redeployment semantic conversion, which results in high deployment costs.

[0023] The adaptability of multi-cloud heterogeneous environments is poor. The deployment of large models is extremely dependent on the specified operating system environment, instruction set type (ARM, X86), driver version, package dependency and other runtime environments. For large models that provide physical machine deployment and Docker containerized script deployment, the model provider will specify the operating system version and driver version used to run the deployment script in the deployment document. These environments, especially in private clouds and hybrid cloud environments, have large differences between different nodes. Knative can only control the operating system sandbox environment of the inference service but cannot control the operating system environment of the deployment inference service script.

[0024] First embodiment The first embodiment of the present application relates to a Serverless large model reasoning service system. Figure 1 As shown, the system may include: Public / private DNS service, used to receive inference service creation requests from users and forward them to the corresponding Kubernetes cluster; A load balancer for handling north-south traffic of the Kubernetes cluster; The inference cloud control plane component is used to receive the inference service creation request, convert the inference service creation request into a Knative Service object, and configure the operating environment; A Knative Service Pod is created according to the Knative Service object. The Knative Service Pod is used to control the model service workload including: A user container, which is built based on a basic image including an operating system, Supervisord, and Dockerd, and is used to run a large model service; The queue proxy, based on the lightweight proxy Sidecar, is responsible for handling the routing, load balancing, and request queuing of incoming requests.

[0025] The following is a detailed description of each of the above steps.

[0026] The public / private DNS service is used to receive the inference service creation request from the user and route the request to the corresponding Kubernetes cluster. The user initiates the deployment request of the large model inference service through the provided domain name resolution service, and the DNS service selects the target Kubernetes cluster based on the request information.

[0027] The load balancer is located at the entrance of the Kubernetes cluster and is used to handle the north-south traffic of the cluster, that is, the traffic interaction between users and the cluster. It can dynamically distribute traffic to different nodes in the cluster to ensure the stability and efficiency of the inference service.

[0028] The inference cloud control plane component is responsible for receiving inference service creation requests forwarded from the DNS service and generating Knative Service objects according to user needs. This component also configures the operating environment, including loading large model files, mounting accelerator card devices, setting log directories and other key operations to ensure that the service can run correctly in the specified environment.

[0029] Knative Service Pod, based on the Knative Service object generated by the inference cloud control plane component, the system creates a Knative Service Pod in the target Kubernetes cluster. Each Knative Service Pod is used to control the specific workload of the large model service and includes the following core modules: User container: User container is built based on the basic image including operating system, Supervisord and Dockerd, and is used to run large model services. In the container, the Nested Containers technology (i.e., nested containers in containers) is used to simulate the operating environment consistent with the deployment script provided by the large model provider, and the model deployment script is directly executed.

[0030] Queue Proxy: Queue Proxy runs as a lightweight proxy sidecar and is responsible for routing incoming requests, load balancing, and queue management of requests. It ensures request order and service stability when serving high loads.

[0031] Taking the user's request to create a large model inference service as an example, the specific process is as follows: the user sends an inference service creation request through the DNS service; the DNS service forwards it to the target Kubernetes cluster based on the request information; the inference cloud control plane component receives the request, generates a Knative Service object, and configures the model running environment; the system creates a Knative Service Pod in the target Kubernetes cluster, loads the user container and queue agent; after the user container is started, the deployment script provided by the model provider is run through the NestedContainers technology to start the large model inference service; after the service is started, the load balancer and DNS service expose the inference API address to the user, and the user calls the inference service through this address and obtains the inference result; Knative dynamically adjusts the number of Pod copies according to the service load to ensure efficient operation of the service.

[0032] It is not difficult to find that compared with the related technologies, in the solution provided in the embodiment of the present application, the system can efficiently receive and parse user requests; dynamically allocate and schedule request traffic; quickly create large model inference services based on the Serverless architecture, and provide elastic scaling support in a multi-cloud environment; shield the differences in the underlying heterogeneous environments and directly reuse the deployment scripts of large model providers.

[0033] Second embodiment The second embodiment of the present application relates to a serverless large model inference service system. The second implementation is an improvement on the first embodiment, and the specific improvements are: like Figure 2 As shown in the figure, Knative Service Pod is automatically created by the inference cloud service cluster based on user requests. Each Knative Service Pod corresponds to a large model inference service workload, and Knative is responsible for traffic control. Knative can automatically adjust the number of replicas of the workload based on the number of requests per second (QPS) and the configured load policy to achieve elastic expansion and contraction. This enables the large model inference service to efficiently handle unstable traffic loads and ensure stable operation under high load.

[0034] The User Container is the core container of the large model inference service. The container is built on a base image (such as Ubuntu 22.04) that includes an operating system, a process management tool Supervisord, and a container daemon Dockerd. Inside this container, multiple processes (such as model inference processes and API service processes) are run. These processes share the same network and PID namespace to ensure efficient operation of the service. The User Container is responsible for starting and managing the inference process of the large model, and running the model's API interface for external calls.

[0035] Queue Proxy is a lightweight sidecar proxy container designed to handle the routing, load balancing, and request queuing of incoming requests. Queue Proxy ensures the stability and reliability of the inference service under high load. By queuing when the request traffic reaches a certain threshold, Queue Proxy can effectively prevent the system from crashing or performance degradation due to too many requests. The proxy is also responsible for routing requests to the appropriate user container to ensure that each request is processed in a timely manner.

[0036] Supervisord-Dockerd is a combined component that uses Supervisord to start and manage the Dockerd process. As a Docker daemon, Dockerd is responsible for managing the container lifecycle, network configuration, and storage management within the user container. Through Supervisord's management of Dockerd, the reasoning process and API process inside the container can run stably, and the container is automatically restarted when needed to ensure uninterrupted service.

[0037] Model Container is a Docker container for running large models in a containerized manner. This container is used to load and run the binary files of large models. The startup of the model container is managed by Docker's Dockerd process in the user container. Each model container can process multiple inference tasks in parallel and dynamically allocate computing resources according to the configured resource limits (such as CPU, memory).

[0038] Model Process is the inference process of the large model binary. It is responsible for executing the model's inference tasks, passing input data into the model for processing, and returning the inference results. This process is usually a highly computationally intensive task that uses hardware resources such as accelerator cards to improve inference speed and efficiency.

[0039] OpenAI APIServer Container is a container that runs the large model API service. This container is responsible for providing an externally exposed API interface through which users can call the large model for reasoning. The API server container usually provides services in the form of RESTful or gRPC interfaces, supporting high-concurrency request processing.

[0040] The OpenAI APIServer Process is the process that runs the big model API binary. It is responsible for receiving and processing the user's reasoning requests and passing the requests to the corresponding Model Process for processing. The API process calls the corresponding big model for reasoning and returns the results based on the type and parameters of the request.

[0041] Configuring the operating environment includes: setting the model deployment script and the image of the user container, mounting the inference accelerator card device and setting the log directory. The inference cloud control plane component is responsible for configuring the startup command of the user container, including the deployment script obtained from the large model provider. The script is responsible for configuring the environment, dependencies and starting the inference process of the inference service. To ensure the high performance of the inference service, the user container image will mount the necessary hardware accelerator cards (such as GPU, TPU, etc.). This step realizes the virtualization of hardware resources through container technology, so that the model inference task can run efficiently. The inference cloud control plane component is also responsible for setting the log directory for storing log information during the large model inference process. By setting the log directory, users can collect performance data and debugging information during the operation of the inference service.

[0042] User containers use the Nested Containers technology to nest container daemons in containers, simulating an operating environment consistent with the deployment scripts provided by the model provider. User containers are built based on a base image that includes an operating system, Supervisord, and Dockerd, ensuring that containers can run consistently in multiple cloud environments. Processes within the container share the same network and PID namespace, ensuring that the inference service can run smoothly and providing an operating environment consistent with a physical machine or other containers, avoiding the impact of underlying host differences on deployment scripts.

[0043] The inference cloud control plane components include: the elastic scaling strategy for setting the Knative Service, including the maximum number of replicas, the minimum number of replicas, the request timeout, and the health check parameters. Define the range within which the Knative Service Pod can be elastically scaled, and adjust the number of Pod replicas based on actual traffic conditions. According to the characteristics of the inference task, set a reasonable request timeout to avoid the system being in a high-load state for a long time. The inference cloud control plane components also set a health check strategy to ensure that each Pod is in a healthy state when providing services, and to promptly detect and automatically repair faulty Pods.

[0044] The base image includes a verified operating system, device drivers, and environment dependencies, which is used to shield the differences in host operating systems, instruction sets, and driver versions in a multi-cloud environment. Use a verified operating system image as the base image (such as Ubuntu 22.04), and install the necessary device drivers and environment dependencies on this basis.

[0045] The operating system and dependent libraries included in the basic image can effectively shield the differences in different host operating systems, instruction sets (such as ARM, x86), and driver versions in multi-cloud environments. Through standardized operating system images and device drivers, the system can avoid compatibility issues caused by differences in host environments and ensure that large model inference services can run seamlessly on various cloud platforms.

[0046] The queue proxy is used to queue and distribute requests when the inference service is highly loaded. As a lightweight sidecar container, the queue proxy is responsible for routing incoming inference requests to the appropriate user container for processing. When the number of inference requests is large, the queue proxy distributes the requests according to the load to avoid overloading a single user container. When the service load is too high, the queue proxy can queue the requests to ensure that the service does not crash due to sudden requests and to ensure the stability of the system.

[0047] The system also includes: an open API container, which is managed by Dockerd in the user container and exposes the API interface of the large model reasoning service; the load balancer supports cross-cloud traffic scheduling for multiple Kubernetes clusters, and provides the reasoning API address for users to call through the DNS service.

[0048] The open API container interacts with external users through standardized APIs (such as RESTful or gRPC interfaces) and supports multiple data formats and communication protocols. Users submit reasoning requests to the system through the API interface, and the API container passes the request to the reasoning process in the user container for processing and returns the reasoning result.

[0049] The load balancer distributes requests to the appropriate Kubernetes cluster and Knative Service Pod based on the request volume, resource allocation, and service health. Through the DNS service, the system can assign an API address to each large model inference service for users to call. The DNS service will dispatch the inference request to the best cluster node based on the traffic situation, improving the system's response speed and stability.

[0050] It is not difficult to find that in the embodiments of the present application, the Serverless inference cloud platform based on Kubernetes uses standardized operating system images and NestedContainers (Docker in Docker / Containerd) technology to realize the direct reuse of large model deployment scripts of large model providers in multi-cloud environments and decouples them from multi-cloud environments and specific operating system driver versions to achieve one-time build and run everywhere, which greatly reduces the cost and complexity of large model deployment in multi-tenant, multi-cloud environments and realizes true one-click cloud rapid deployment.

[0051] Third embodiment The third embodiment of the present application relates to a Serverless large model inference service system. The third implementation is an improvement on the first embodiment, and the specific improvements are: like Figure 3 As shown in the figure, the entire workflow is introduced by taking the integration of a model image and startup script delivered by a large model service provider in a private inference cloud platform as an example of elastically scaling Serverless inference service.

[0052] The management side of the inference cloud platform stores the physical machine or container-based deployment script files provided by the large model provider in the platform's NAS storage.

[0053] The inference cloud platform management side creates a basic operating system image that has been verified by a large model provider, installs Docker and SuperVisord, and specifies the startup container script to start a Dockerd process managed by SuperVisord.

[0054] The inference cloud platform user selects the large model to be deployed. The inference cloud service platform management side will create a corresponding Knative Service for the inference service and configure the request timeout, maximum / minimum number of replicas, health check, etc. of the service. The large model file / OCI model image will be mounted to the Pod corresponding to the Knative Service in the form of PVC mounting and the Pod startup command of the Knative Service will be set to start the Supervisord process and the Dockerd process it manages first, and then run the deployment script provided by the large model service provider intact.

[0055] Grayscale release is a gradual upgrade and deployment strategy that allows the functions of the new version to be gradually opened to users so that their stability and compatibility can be tested on a small scale. This approach can reduce the risk during the upgrade process. Grayscale release is located in the Knative Serverless service part and is an important functional module of cloud services. It supports users to gradually open new functions to a part of users for verification when updating models or services without affecting the normal use of all users. When a large model provider uploads a new model file, the old model can be gradually replaced through grayscale release to ensure that it does not affect the stability of the overall service. Provide new version services to a small number of users, collect feedback, optimize performance or fix problems before fully launching.

[0056] Version management is the process of controlling, recording, and tracking the versions of software, models, or other releasable resources, so as to facilitate backtracking or switching to older versions and ensure the maintainability and stability of services. The version management module is located in the KnativeServerless service part and is closely connected with NAS storage, etc. It performs version control on uploaded model files or user images, allowing users to roll back to stable versions; track and manage updates of different versions; and can cooperate with grayscale release to select the appropriate version for release.

[0057] The inference cloud platform successfully generates the corresponding Knative Service for the large model inference service. After the corresponding Serverless inference service Pod is successfully started (the model inference service is considered to be successfully started after passing the health check of the model inference API), Knative will be responsible for elastic scaling and traffic control of the inference large model service according to requests and expose the large model inference service API address to platform users. The inference cloud platform users call the corresponding large model through the API address to obtain the large model inference result.

[0058] It should be noted that the third embodiment of the present application may also be an improvement based on the second embodiment.

[0059] It is not difficult to find that in the embodiment of the present application, the deployment script of the large model provider is reused to achieve seamless deployment. The NestedContainers technology is used to start Docker in the Docker / Containerd container to simulate the virtual machine environment, which is consistent with the deployment script running environment required by the large model provider. The large model is deployed at a very low cost. The inference cloud platform provider produces a large model base image (including operating system and driver). Subsequent model iterations do not need to make any modifications to the image to repeatedly deploy different versions of the large model in this image, completely eliminating the need to customize the writing of the corresponding Serverless service startup script for different large models. Rapid adaptation of multi-cloud heterogeneous environments. Since the deployment of large models in multi-cloud and private cloud environments is extremely complex, it is necessary to consider the operating system used by the node where the deployment is located, the driver version-related environment variables, including dependency conflicts and other matters, and debugging is extremely time-consuming. By providing a unified basic OCI image (including operating system and driver) as the context-dependent environment for model deployment, it completely avoids the additional deployment and debugging costs caused by different operating systems, driver versions, package versions, and environment variables of different nodes in hybrid cloud and multi-cloud environments, and truly achieves one-time construction and deployment and operation everywhere.

[0060] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application, but this does not mean that there are no other units in this embodiment.

[0061] Fourth embodiment The fourth embodiment of the present application relates to a Serverless large model reasoning service method, the method comprising: Receive a large model inference service creation request submitted by a user through a public or private DNS service, and forward the large model inference service creation request to a target Kubernetes cluster; Generate a Knative Service object according to the large model inference service creation request through the inference cloud control plane component, and configure the operating environment, including mounting the model file, setting the accelerator card device and the log directory; Create a Knative Service Pod through the Knative Service object. The Pod contains a user container and a queue agent, where: User containers run basic images based on the operating system, Supervisord, and Dockerd, which are used to start large model inference services; The queue proxy handles the routing, load balancing, and request queuing of incoming requests; Run the container daemon in the user container through the Nested Containers technology and execute the model provider's deployment script to start the large model service; Knative dynamically adjusts the number of Pod replicas based on service load to achieve elastic expansion and reduction of inference services; The inference service API address is exposed through the load balancer and DNS service for users to call the large model inference function.

[0062] This example describes how to quickly deploy a Serverless large-model inference service on an inference cloud platform built on Kubernetes. The method includes steps such as user image production, inference cloud platform deployment, basic operating system image production, and Knative Service creation, ultimately achieving rapid startup and elastic scaling of large-model inference services. Take a specific implementation plan as an example: User images are the basis for deploying large model inference services. The specific steps are as follows: Download the base operating system image and download Ubuntu 24.04 from a reliable Docker image source as the base image. This image contains the required operating system environment and provides a unified basis for subsequent installation and configuration.

[0063] Install Supervisord and Docker components. Install the required components in the Ubuntu 24.04 image: Supervisord (version >= 4.2.5): As a process management tool, it is used to manage the process life cycle in the container. Docker (version >= 27.3): As a containerized deployment tool, it is used to start and manage containers.

[0064] Configure the Docker startup script and write the start-docker.sh script to put the Docker service into Supervisord management. This script will be executed when the container starts, start the Dockerd process through Supervisord, ensure the correct management of the container environment, and automatically start the model inference container.

[0065] Through the above steps, the user image is successfully built and is ready to be used to run large model inference services in the Inference Cloud Platform.

[0066] The deployment steps of the inference cloud platform are as follows: Install Knative and Kubernetes. Install Knative version >= 1.0.0. Knative is a Serverless framework that provides elastic scaling, traffic control, and other functions. Install Kubernetes version >= 1.22. Kubernetes is a container orchestration platform that manages the lifecycle and resource allocation of container clusters.

[0067] Prepare the basic operating system image of the OCI standard. The inference cloud platform pre-creates the basic operating system image that complies with the OCI standard, including the installed Docker and Supervisord, and uploads it to the platform's NAS storage. This image will serve as the basic environment for subsequent large model inference services.

[0068] Upload large model files and startup scripts, upload different large model files and corresponding startup scripts to the platform's NAS storage. These files include the binary files of the large model and the configuration files required for deployment, ensuring that they can be accessed and used in the subsequent deployment process.

[0069] In the inference cloud platform, the steps to deploy a large model inference service are as follows: The user selects the prepared basic operating system image in the inference cloud platform and specifies the name of the large model to be deployed. This step ensures that the appropriate operating system and hardware configuration are selected to support the operation of the large model inference task.

[0070] Create a Knative Service. The inference cloud platform automatically creates a Knative Service as a large model inference service based on the operating system image and large model name selected by the user. This service is responsible for managing the life cycle, elastic scaling, and traffic control of large models. The system will configure the following content according to demand: request timeout, set a reasonable timeout to avoid inference requests timing out due to long periods of non-response; maximum / minimum number of replicas, configure the maximum and minimum number of replicas according to the inference load requirements to ensure that the service can automatically scale up and down according to the request traffic; health check, configure the health check mechanism to ensure that the large model inference service is always in a healthy state.

[0071] Mount the model file and startup script. The inference cloud platform mounts the OCI image or model file of the large model into the Pod of the Knative Service in the form of PVC (persistent volume claim). The system starts the Supervisord process and the Dockerd process it manages through the startup command, and then runs the deployment script provided by the large model service provider.

[0072] Start the inference service. In the Knative Service Pod, Supervisord starts and manages the Docker container. By executing the deployment script provided by the large model provider, the model is loaded, the inference service is started, and the initial configuration is completed. Once the model is successfully loaded and running, the inference service begins to accept user requests.

[0073] Elastic scaling and API exposure: Once the large model inference service is successfully started, Knative will automatically adjust the number of Knative Service Pod replicas based on the real-time request volume. By monitoring the load and traffic of the service, Knative will automatically expand or reduce Pod replicas to ensure that the inference service can run stably and can cope with sudden traffic peaks. After the inference service is successfully started and runs stably, Knative will expose an API interface for users to call. The inference cloud platform routes the request to the corresponding Knative Service Pod through the DNS service. Users can submit requests to the large model inference service through the API address and obtain inference results.

[0074] The user calls the inference service: the user submits a request to the inference cloud platform through the provided API address. The inference cloud platform routes the request to the correct Knative Service Pod through the service controlled by Knative and starts the corresponding large model inference process. After the inference is completed, the inference result will be returned to the user through the API interface.

[0075] It is not difficult to find that in the embodiment of the present application, the user selects the required large model and operating system image on the inference cloud platform and submits a deployment request; the system automatically creates a Knative Service and configures service parameters (request timeout, maximum number of copies, health check, etc.); the OCI image or file of the large model is mounted to the Knative Service Pod, and the inference container is started to execute the large model deployment script; after the inference service is successfully started, Knative is responsible for automatic elastic scaling, adjusting the number of Pod copies according to traffic, and exposing the API address for user calls; the user submits an inference request through the API interface to obtain the inference result. Through automated configuration and management, the efficient deployment, flexible scaling and reliable operation of the large model inference service in a multi-cloud environment are ensured. Through Knative's Serverless architecture, the large model inference service can be automatically expanded according to actual needs to meet the requirements of inference tasks of different scales.

[0076] The step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this application.

[0077] In addition, some embodiments of the present application also provide an electronic device. The electronic device may be a digital computer in various forms, such as a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, etc. The electronic device may also be a mobile device in various forms, such as a personal digital processing, a cellular phone, a smart phone, a wearable device, and other similar computing devices.

[0078] The electronic device includes: one or more processors; and a memory storing computer program instructions, wherein when the computer program instructions are executed, the processor executes the steps of the method provided in any one or more of the above embodiments. Figure 4 An exemplary structural diagram of the electronic device is disclosed. Figure 4 As shown, the electronic device includes: one or more processors 1101, a memory 1102, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, if necessary, multiple processors and / or multiple buses can be used with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Among them, the components shown in this article, their connections and relationships, and their functions are only examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0079] The electronic device may further include: an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103 and the output device 1104 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0080] The input device 1103 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1104 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0081] In order to provide interaction with the user, the electronic device may be a computer. The computer has: a display device (e.g., a cathode ray tube (CRT) or an LCD monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and the input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0082] In the embodiments of the present application, a computer program / instruction is stored on a computer-readable medium, and when the computer program / instruction is executed by a processor, the steps of the method provided by any one or more of the above embodiments are implemented. The computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more computer-readable instructions.

[0083] The memory 1102 can be used as a non-transient computer-readable storage medium, which can be used to store non-transient software programs, non-transient computer executable programs and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1102, so as to implement the program instructions / modules corresponding to the method provided by any one or more embodiments in the embodiments of the present application.

[0084] The memory 1102 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1102 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1102 may optionally include a memory remotely arranged relative to the processor 1101, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0085] It should be noted that the computer-readable medium described in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM, Erasable Programmable Read - Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM, Compact Disc Read - Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0086] Computer-readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change random-access memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0087] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0088] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. For example, an application-specific integrated circuit (ASIC), a general-purpose computer or any other similar hardware device may be used for implementation. In some embodiments, the software program of the present application may be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including related data structures) may be stored in a computer-readable recording medium, such as a RAM memory, a magnetic or optical drive or a floppy disk and the like. In addition, some steps or functions of the present application may be implemented by hardware, for example, as a circuit that cooperates with a processor to perform various steps or functions.

[0089] The computer program product provided in the embodiment of the present application includes one or more computer programs / instructions, which, when executed by the processor, generate in whole or in part the process or function described in the embodiment of the present application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL, Digital Subscriber Line)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive, SSD, solid statedisk), etc.

[0090] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0091] The scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims are included in the present application. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim may also be implemented by one unit or device through software or hardware. The words "first", "second", etc. are only used to distinguish the description, and do not indicate any particular order, nor can they be understood as indicating or implying relative importance.

[0092] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily mention changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims, and the above embodiments should be regarded as exemplary and non-restrictive.

Claims

1. A Serverless large model inference service system, characterized in that: The system comprises: Public / private DNS service, used to receive inference service creation requests from users and forward them to the corresponding Kubernetes cluster; A load balancer for handling north-south traffic of the Kubernetes cluster; The inference cloud control plane component is used to receive the inference service creation request, convert the inference service creation request into a Knative Service object, and configure the operating environment; Create a Knative Service Pod according to the Knative Service object. The Knative Service Pod is used to control the model service workload including: A user container, which is built based on a basic image including an operating system, Supervisord, and Dockerd, and is used to run a large model service; The queue proxy, based on the lightweight proxy Sidecar, is responsible for handling the routing, load balancing, and request queuing of incoming requests.

2. The system according to claim 1, characterized in that The configuration operating environment includes: Set up the model deployment script and the image of the user container to mount the accelerator card device used for inference and set the log directory.

3. The system according to claim 2, characterized in that The user container runs the container daemon process in a nested manner within the container through the Nested Containers technology, simulating an operating environment consistent with the deployment script provided by the model provider.

4. The system according to claim 3, characterized in that The inference cloud control plane components include: Used to set the elastic scaling policy of Knative Service, including the maximum number of replicas, the minimum number of replicas, the request timeout, and the health check parameters.

5. The system according to claim 4, characterized in that The basic image includes a verified operating system, device drivers, and environment dependencies, and is used to shield differences in host operating systems, instruction sets, and driver versions in a multi-cloud environment.

6. The system according to claim 5, characterized in that The queue agent is used to queue and distribute requests when the inference service is under high load.

7. The system according to any one of claims 1 to 6, characterized in that: The system further comprises: An open API container, managed by Dockerd in the user container, exposes the API interface of the large model inference service; The load balancer supports cross-cloud traffic scheduling for multiple Kubernetes clusters, and provides an inference API address for users to call through a DNS service.

8. A Serverless large model reasoning service method, characterized in that: The method comprises: Receive a large model inference service creation request submitted by a user through a public or private DNS service, and forward the large model inference service creation request to a target Kubernetes cluster; Generate a Knative Service object according to the large model inference service creation request through the inference cloud control plane component, and configure the operating environment, including mounting the model file, setting the accelerator card device and the log directory; Create a Knative Service Pod through the Knative Service object. The Pod contains a user container and a queue agent, where: User containers run basic images based on the operating system, Supervisord, and Dockerd, which are used to start large model inference services; The queue proxy handles the routing, load balancing, and request queuing of incoming requests; Run the container daemon in the user container through the Nested Containers technology and execute the model provider's deployment script to start the large model service; Knative dynamically adjusts the number of Pod replicas based on service load to achieve elastic expansion and reduction of inference services; The inference service API address is exposed through the load balancer and DNS service for users to call the large model inference function.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; and A memory storing computer program instructions which, when executed, cause the processor to perform the steps of the method of claim 8.

10. A computer readable medium having a computer program / instructions stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method of claim 8 are implemented.

Images