Data processing method, platform, medium and product

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a data processing method and platform, a medium and a product. The data processing method comprises the steps of receiving a reasoning request; judging whether a first Knative framework instance is operated or not at present; if the first Knative frame instance is operated at present, identifying whether a first reasoning cluster corresponding to the first Knative frame instance is matched with the reasoning request; and if the first Knative framework instance is not operated currently or the first reasoning cluster is not matched with the reasoning request, creating a second Knative framework instance based on the reasoning request, creating a second reasoning cluster based on the second Knative framework instance, and distributing the reasoning request to a corresponding service node in the second reasoning cluster for processing. The method and the device are at least used for solving the problems that service continuity is poor, performance bottleneck is caused during traffic peak, and large model reasoning of production-level high-concurrency long-output context cannot be supported.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI) technology, and in particular to data processing methods, platforms, media, and products. Background Art

[0002] Kubernetes (K8s) is an open-source container orchestration platform. Its core goal is to simplify the deployment, scaling, and management of containerized applications (such as Docker containers), helping users build highly available and resilient distributed systems. As an open-source container orchestration technology, Kubernetes has become the platform of choice for cloud-native application runtimes thanks to its powerful automated deployment, resource management, and cluster scheduling capabilities. Building on this foundation, various serverless frameworks (i.e., serverless architectures) such as Knative have emerged. These frameworks, leveraging serverless computing concepts, offer features such as automatic scaling, event-driven development, and simplified deployment, further promoting the development of cloud-native technologies. In recent years, with the increasing popularity of open-source large-scale models with extremely large parameter counts (such as DeepSeek V3 and R1, with a parameter scale of up to 671B), large-model inference tasks are increasingly being deployed in Kubernetes environments to leverage its distributed computing and resource integration capabilities. However, the current mainstream distributed inference frameworks (such as vLLM, DeepSpeed, TGI, SGLang, etc.) are diverse and each has its own characteristics. A single machine can no longer meet the needs of ultra-large-scale parameter weight storage and activation function calculation, making multiple machines and multiple cards (a distributed architecture across multiple nodes and multiple GPUs) the norm for large-scale model inference deployment.

[0003] While the multi-machine, multi-GPU architecture significantly improves inference performance, its deployment and maintenance complexity presents a significant bottleneck. Different inference frameworks have vastly different configuration requirements, requiring manual adaptation of the underlying hardware, network environment, and communication protocols, making deployment cumbersome and error-prone. Furthermore, multi-machine, multi-GPU inference services lack the core features of serverless frameworks, such as elastic resource scalability that dynamically adjusts to load, fine-grained concurrency control policies for high-concurrency traffic, and the ability to quickly redeploy after model fine-tuning. Furthermore, traditional serverless service design typically relies on a single HTTP / HTTPS port for external inference service provision (Serving) to simplify traffic management and user access. However, multi-machine, multi-GPU inference requires numerous additional ports for efficient cross-node communication during internal computations (such as tensor parallelism and pipeline parallelism). Some frameworks, such as Ray, even expose metric ports for performance monitoring and debugging. These complex port requirements conflict significantly with the single-port traffic management philosophy of serverless frameworks like Knative, further complicating integration.

[0004] Currently, in cloud-native environments based on Kubernetes and Knative, there are KubeRay-based deployment solutions and Knative-based single-server deployment solutions for multi-machine, multi-GPU inference deployment of extremely large parameter-rich models (such as DeepSeek V3 and R1). However, these solutions have limitations, including the following drawbacks:

[0005] 1. Poor service continuity: The Ray cluster is implemented using KubeRay, but the already started vLLM service (based on tensor parallelism and pipeline parallelism) cannot dynamically detect newly added worker nodes. Using the newly added nodes requires restarting the vLLM service, which interrupts the inference requests being processed and seriously affects service continuity.

[0006] 2. Performance bottlenecks during peak traffic and resource waste during low traffic: KubeRay and Ray lack the ability to perceive inference requests and are unable to dynamically adjust resources based on real-time load or implement fine-grained traffic control. This results in performance bottlenecks during peak traffic and resource waste during low traffic for large model inference services.

[0007] 3. It cannot support production-level, high-concurrency inference of large models with long output contexts. Single-machine deployment based on Knative is limited by the single-node's video memory capacity and cannot support models with large parameter counts with output lengths exceeding 8KB or high-concurrency batch size inference requests (e.g., batch size > 4). This can lead to service startup failures or performance degradation, highlighting the necessity of multi-machine and multi-GPU inference.

[0008] In summary, existing solutions have problems such as poor service continuity, performance bottlenecks during traffic peaks, and inability to support production-level, high-concurrency reasoning of large models with long output contexts on multiple machines and multiple GPUs when supporting reasoning of large models with extremely large parameter counts. Summary of the Invention

[0009] One purpose of this application is to provide a data processing method, platform, medium and product to at least solve the problems of poor service continuity, performance bottlenecks during traffic peaks, and inability to support production-level large-model reasoning with high concurrency and long output context.

[0010] To achieve the above objectives, some embodiments of the present application provide the following aspects:

[0011] In a first aspect, some embodiments of the present application further provide a data processing method, including:

[0012] receiving an inference request;

[0013] Determine whether the first Knative framework instance is currently running;

[0014] If the first Knative framework instance is currently running, identifying whether a first inference cluster corresponding to the first Knative framework instance matches the inference request;

[0015] If the first Knative framework instance is not currently running, or the first inference cluster does not match the inference request, a second Knative framework instance is created based on the inference request, a second inference cluster is created based on the second Knative framework instance, and the inference request is assigned to the corresponding service node in the second inference cluster for processing.

[0016] Optionally, also include:

[0017] If the first inference cluster matches the inference request, the inference request is assigned to a corresponding service node in the first inference cluster for processing.

[0018] Optionally, if the first Knative framework instance is currently running, identifying whether a first inference cluster corresponding to the first Knative framework instance matches the inference request includes:

[0019] If the first Knative framework instance is currently running, identifying whether the current request traffic meets the preset conditions based on the inference request;

[0020] If not, it indicates that the first inference cluster corresponding to the first Knative framework instance does not match the inference request;

[0021] If so, it indicates that the first inference cluster corresponding to the first Knative framework instance matches the inference request.

[0022] Optionally, the second Knative framework instance communicates with the second inference cluster via NCCL communication mode, MPI communication mode, or Gloo communication mode.

[0023] Optionally, the first Knative framework instance communicates with the first inference cluster via an NCCL communication mode, an MPI communication mode, or a Gloo communication mode.

[0024] Optionally, if the first Knative framework instance is not currently running, or the first inference cluster does not match the inference request, creating a second Knative framework instance based on the inference request, creating a second inference cluster based on the second Knative framework instance, and assigning the inference request to a corresponding service node in the second inference cluster for processing, the method further includes:

[0025] Free up idle resources in the second inference cluster according to preset rules.

[0026] Optionally, the first inference cluster includes at least two types of service nodes, and / or the second inference cluster includes at least two types of service nodes.

[0027] In a second aspect, some embodiments of the present application further provide a data processing platform, including:

[0028] Application layer, used to receive inference requests;

[0029] A service layer, configured to provide a first reasoning cluster and / or a second reasoning framework, and process the reasoning request through the first reasoning cluster or the second reasoning framework;

[0030] The logic management layer is configured to determine whether a first Knative framework instance is currently running. If so, identify whether a first inference cluster corresponding to the first Knative framework instance matches the inference request. If the first Knative framework instance is not currently running, or the first inference cluster does not match the inference request, create a second Knative framework instance based on the inference request, create a second inference cluster in the service layer based on the second Knative framework instance, and assign the inference request to a corresponding service node in the second inference cluster for processing.

[0031] A storage layer is used to store model files and data of the first inference cluster and / or the second inference cluster.

[0032] In a third aspect, some embodiments of the present application further provide a computer-readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method described above.

[0033] In a fourth aspect, some embodiments of the present application further provide a computer program product, comprising a computer program / instruction, which implements the steps of the above-described method when executed by a processor.

[0034] Compared with related technologies, in the solution provided by the embodiment of the present application, the data processing method can first determine whether the first Knative framework instance is currently running based on the received inference request. If the first Knative framework instance is currently running, the first inference cluster is matched with the inference request to determine whether the current first inference cluster can process the inference request; when the first inference cluster cannot process the inference request or the Knative framework instance is not currently running, it can be expanded to create a second Knative framework instance based on the inference request, and then a second inference cluster is created through the second Knative framework instance, and the inference request is processed by the second inference cluster. In the case where the current Knative framework instance cannot meet the inference request, the present application can ensure service continuity and achieve load balancing by expanding the inference cluster to avoid performance bottlenecks during traffic peaks; and use the inference cluster to achieve production-level high-concurrency large-model inference with long output context. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements. Unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0036] Figure 1 This is a schematic diagram of an exemplary flow chart of a data processing method provided in this application;

[0037] Figure 2 This is a schematic diagram of an exemplary flow chart of another data processing method provided according to the fourth embodiment of the present application;

[0038] Figure 3 This is an architectural diagram of a data processing platform provided in accordance with the present application;

[0039] Figure 4 The present invention provides an exemplary flowchart of an embodiment of processing an inference request according to the present application. DETAILED DESCRIPTION

[0040] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0041] The following terms are used in this document.

[0042] GPU stands for Graphics Processing Unit, graphics processing unit;

[0043] HTTP (Hypertext Transfer Protocol) is a transport protocol used to transfer hypertext from the Internet to a local browser. It defines the format of requests and responses between clients and servers, works on top of the TCP / IP model, and typically uses port 80.

[0044] HTTPS (Hypertext Transfer Protocol Secure) is a secure version of HTTP. By adding the SSL / TLS protocol to HTTP, it provides data encryption, identity authentication, and integrity verification functions to ensure the security of the transmission process.

[0045] Kubernetes is an open-source container orchestration platform that automates the deployment, expansion, and maintenance of containerized applications and supports large-scale cluster scheduling.

[0046] Knative, short for Kubernetes Native, is based on the Kubernetes Serverless framework and incorporates serverless computing concepts to provide automatic scaling, event-driven development, and simplified deployment capabilities.

[0047] Queue-Proxy is a core component of Knative. It runs as a sidecar container alongside the application container and is responsible for collecting metrics and buffering requests.

[0048] Autoscaler is a core component of Knative. It automatically adjusts the number of replicas based on collected metrics.

[0049] Kubeflow is a Kubernetes-based machine learning toolkit designed to simplify the deployment, management, and scaling of machine learning (ML) workflows. By integrating multiple open source machine learning tools, Kubeflow provides an end-to-end solution that supports the entire process from data preprocessing, model training, hyperparameter tuning, to model deployment.

[0050] vLLM, short for Virtual Large Language Model, is an efficient large language model inference framework that optimizes memory management and inference speed, and supports multi-GPU parallel computing based on Ray clusters.

[0051] TensorRT-LLM is an open source library launched by NVIDIA for defining, optimizing, and executing reasoning on large language models (LLMs) in production environments.

[0052] DeepSpeed is a deep learning optimization library developed by Microsoft that supports large model training and inference, and improves performance through distributed technologies such as ZeRO and pipeline parallelism;

[0053] TensorRT-LLM is an optimized large-model inference framework developed by NVIDIA. It supports distributed inference on multiple machines and multiple GPUs. It provides high performance through tensor parallelism and pipeline parallelism combined with NCCL, and is particularly suitable for NVIDIA GPU environments.

[0054] SGLang, short for Structured Generation Language, is an emerging high-performance reasoning framework that focuses on throughput and low latency. It optimizes batch processing and KV cache management and supports distributed reasoning.

[0055] Ray: an open source distributed computing framework that simplifies the scheduling and execution of large-scale parallel tasks, such as inference on large models.

[0056] KubeRay: An integrated extension of Ray on Kubernetes, providing containerized deployment, management, and elastic scaling support.

[0057] DeepSeek-V3 / R1: An open-source MoE model with a very large number of parameters (671 bytes) developed by DeepSeek, representing a typical example of a domestic large language model.

[0058] ICG, short for InferenceClusterGateway, is an inference gateway designed specifically for Kubernetes environments. It supports inference services for large language models (LLMs).

[0059] PyTorch is an open source deep learning framework based on Python;

[0060] PyTorchJob is a custom resource (CRD) designed in the Kubernetes ecosystem specifically for running distributed PyTorch training tasks.

[0061] An MPIJob is a parallel task that runs in a distributed computing environment based on the MPI (Message Passing Interface) standard. It is typically used in high-performance computing (HPC) or large-scale cluster training scenarios. In the Kubernetes ecosystem, an MPIJob is a custom resource (CRD) used to manage and schedule MPI-based distributed training or computing jobs.

[0062] RayCluster is a cluster based on Ray (a distributed computing framework), designed for high-performance machine learning, reinforcement learning, and large-scale parallel computing;

[0063] SglangCluster is a cluster based on SGLang (a structured generative language runtime for large language models (LLMs)), focusing on efficient LLM reasoning and service-oriented;

[0064] ImDeployCluster is IBM's middleware automation deployment and management tool for enterprise-level application clusters;

[0065] NAS stands for Network Attached Storage. Network attached storage is a network device specifically used for data storage and sharing.

[0066] First embodiment

[0067] The first embodiment of the present application relates to a data processing method. Figure 1 As shown, the method may include the following steps:

[0068] S1. Receive inference request;

[0069] S2. Determine whether the first Knative framework instance is currently running; if so, execute step S3; if not, execute step S4;

[0070] S3. Identify whether the first inference cluster corresponding to the first Knative framework instance matches the inference request; if not, execute step S4;

[0071] S4. Create a second Knative framework instance based on the inference request, create a second inference cluster based on the second Knative framework instance, and distribute the inference request to a corresponding service node in the second inference cluster for processing.

[0072] The first embodiment of the present application relates to a data processing method. The core of this embodiment lies in steps S1 to S4. The data processing method can first determine whether the first Knative framework instance is currently running based on the received inference request. If the first Knative framework instance is currently running, the first inference cluster is matched with the inference request to determine whether the current first inference cluster can process the inference request; when the first inference cluster cannot process the inference request or the Knative framework instance is not currently running, it can be expanded to create a second Knative framework instance based on the inference request, and then a second inference cluster is created through the second Knative framework instance. The second Knative framework instance is used to distribute the inference request to the corresponding service node in the second inference cluster, and the second inference cluster is used to process the inference request. In the case where the current Knative framework instance cannot meet the inference request, the present application can ensure service continuity and achieve load balancing by expanding the inference cluster to avoid performance bottlenecks during traffic peaks; and use the inference cluster to achieve production-level high-concurrency large-model inference with long output context.

[0073] The following is a detailed description of the implementation details of the data processing method of this embodiment. The following content is only provided for ease of understanding and is not necessary for implementing this solution.

[0074] In this embodiment, the first inference cluster may include at least two types of service nodes.

[0075] By way of example and not limitation, the first inference cluster may include at least two service nodes selected from the group consisting of a vLLM inference framework service node, an SGLang inference framework service node, and a TensorRT-LLM inference framework service node.

[0076] In this embodiment, the first inference cluster includes multiple types of service sets, each type of service set corresponds to a type of service node, and the service set is composed of multiple service nodes of the type.

[0077] vLLM Serve Cluster (i.e., vLLM type service collection): Use vLLM to start tensor parallel and pipeline parallel large model inference services on a multi-machine and multi-GPU Ray cluster. Service nodes communicate through NCCL and configure environment variables (such as MASTER_ADDR, MASTER_PORT, NCCL_SOCKET_IFNAME, NCCL_IB_DISAB LE, and NCCL_IB_HCA) to coordinate distributed tasks.

[0078] SGlang Serve Cluster (i.e., SGlang type service collection): Start SGLang through Python processes on multiple machines, use NCCL communication based on the PyTorch distributed backend, and specify MASTER_ADDR and WORLD_SIZE to synchronize service nodes.

[0079] TensorRT-LLM type service collection: Start TensorRT-LLM through Python processes on multiple machines, synchronize tensors between GPUs based on NCCL, and configure MASTER_ADDR to coordinate multi-node tasks.

[0080] In this embodiment, the second inference cluster may include at least two types of service nodes.

[0081] By way of example and not limitation, the second inference cluster may include at least two service nodes selected from the group consisting of a vLLM inference framework service node, an SGLang inference framework service node, and a TensorRT-LLM inference framework service node.

[0082] In this embodiment, the second inference cluster includes multiple types of service sets, each type of service set corresponds to a type of service node, and the service set is composed of multiple service nodes of this type.

[0083] In this embodiment, the second Knative framework instance communicates with the second inference cluster through the NCCL communication mode, the MPI communication mode, or the Gloo communication mode.

[0084] In this embodiment, the second Knative framework instance and the second inference cluster use a variety of different communication methods to adapt to different scenarios. The NCCL communication mode is suitable for collective communication, such as efficient communication between GPUs (deep learning scenarios); the MPI communication mode is suitable for point-to-point and collective communication, such as cross-node process communication; and the Gloo communication mode is suitable for collective communication, such as hybrid device communication (CPU / GPU cross-platform communication).

[0085] Second embodiment

[0086] The second embodiment of the present application relates to a data processing method. Figure 1 As shown, the second implementation is an improvement based on the first embodiment, and the specific improvement is: it also includes: if the first inference cluster matches the inference request, executing step S5.

[0087] S5. Allocate the inference request to a corresponding service node in the first inference cluster for processing.

[0088] It is not difficult to find that in the embodiment of the present application, the inference request is distributed to the corresponding service node in the first inference cluster for rapid processing through the first Knative framework instance. The first inference cluster supports the output length of more than 8K or high-concurrency processing of batch size inference requests for models with ultra-large parameters (such as: batch size>4).

[0089] Third embodiment

[0090] The third embodiment of the present application relates to a data processing method. The third embodiment is an improvement based on the first embodiment, and the specific improvements are:

[0091] Step S3 may include:

[0092] If the first Knative framework instance is currently running, identifying whether the current request traffic meets the preset conditions based on the inference request;

[0093] If not, it indicates that the first inference cluster corresponding to the first Knative framework instance does not match the inference request, then executing step S4;

[0094] If so, it indicates that the first inference cluster corresponding to the first Knative framework instance matches the inference request, and step S5 is executed.

[0095] As an example and not a limitation, the preset condition may be a query per second threshold QPS (full name: Queries Per Second, ie, the number of queries per second).

[0096] It should be noted that the query per second threshold QPS is the maximum query per second rate supported by a single inference cluster.

[0097] If the current request traffic exceeds QPS, that is, the current request traffic does not meet the preset conditions, it means that the current first inference cluster is in the peak traffic stage, and the inference cluster is expanded through step S4 to achieve load balancing; if the current request traffic does not exceed QPS, it means that the current first inference cluster can process the inference request normally, and the inference request is processed through step S5.

[0098] In practical applications, step S4 can process the inference request based on inference frameworks such as vLLM, SGLang, and TensorRT-LLM. KubeRay is used to create and manage the lifecycle of Ray clusters, supporting vLLM for launching tensor-parallel and pipeline-parallel inference services on multi-machine, multi-GPU Ray clusters. KubeFlow is used to create clusters such as SGLang and TensorRT-LLM that use the PyTorch learning framework for multi-machine, multi-GPU inference. Distributed inference services based on NCCL, MPI, or Gloo are launched through PyTorchJob (training resources) or MPIJob. Knative is used to provide fine-grained traffic control (via Queue-Proxy and Autoscaler components), elastic scaling based on request volume (QPS), and version management and fast switching of model services. The ICG Gateway, as a user container for Knative Ksvc, is responsible for launching and coordinating multi-machine, multi-GPU inference clusters such as vLLM, SGLang, and TensorRT-LLM. After the models in the inference cluster are successfully launched, inference requests are forwarded to the OpenAI-compatible port service (e.g., v1 / completions) of the inference cluster through ICG.

[0099] Furthermore, step S4 may include: when the request traffic (number of requests) exceeds the QPS threshold set for the service, Knative creates a Knative framework instance, and Knative triggers ICG's Pod horizontal expansion to create a new multi-machine and multi-card inference cluster through KubeRay or Kubeflow. ICG communicates with the inference cluster through the Kubernetes network identifier (FQDN, full name Fully Qualified Domain Name) to ensure that the new cluster name does not conflict with the existing cluster and maintain service continuity.

[0100] Specifically, see Figure 4As shown, the inference request sent by the client is received through the edge traffic ingress (Istio ingress gateway), and the inference request is load balanced and routed to the trigger component (Activator) service provided by Knative. Based on the request traffic received by Activator, Knative's elastic scaling component AutoScaler sends the request traffic to the Inference Cluster Gateway (ICG) through the Queue-Proxy, triggering the ICG service to scale from 0 Knative framework instances (i.e., Pods) to 1 or more Knative framework instances (Scale-from-Zero, achieving from zero to multiple framework instances). Each Knative framework instance includes a Queue-Proxy and an Inference Cluster Gateway (ICG). ICG starts and, based on inference service parameters (such as the inference framework type and model file path), creates a multi-machine, multi-GPU inference cluster, such as a Ray Cluster, Sglang Cluster, and ImDeploy Cluster, through KubeRay or Kubeflow (using PyTorchJob or MPIJob). It then mounts the shared model file through Kubernetes' Persistent Volume Claim (PVC) (using NAS storage and NFS for multi-node read / write support and simple deployment). After waiting for the inference cluster to be ready, it runs the model startup command (such as vllm serve). The inference cluster is exposed as a Knative Serverless service through ICG. After the ICG status changes to Ready, the edge traffic entry (Istio ingress gateway) sets the service to active (Serve Mode). Traffic directly enters Knative's Queue-Proxy and is then forwarded to the ICG, which routes it to the specific multi-machine, multi-GPU inference cluster (using NCCL or MPI communication).

[0101] It should be noted that: in this embodiment, the first Knative framework instance and the first inference cluster communicate through the NCCL communication mode, or the MPI communication mode, or the Gloo communication mode.

[0102] In this embodiment, multiple inference clusters can run simultaneously, sharing inference model data between them. Inference model data is stored in the storage layer. The storage layer is responsible for storing model files and providing persistent storage for model files and data for multi-machine, multi-GPU inference services. This storage layer is associated with a specific workload in the form of a PVC. The storage layer can include multiple storage types, including at least one of local SSD PV storage, NFS PV storage, HDFS PV storage, EBS PV storage, and NVMe PV storage.

[0103] Local SSD PV provides low-latency single-node read and write storage through the local SSD. It is bound only to a specific node and cannot be shared across nodes. It is suitable for the model preheating cache layer of the inference cluster (such as the local acceleration of vLLM).

[0104] NFS PV uses the network file system to support multi-node read-write shared storage. It is suitable for multi-machine and multi-card inference clusters (such as SGLang model loading). However, it is limited by network bandwidth and has low throughput.

[0105] HDFS PV provides high-throughput and fault-tolerant multi-node storage based on the Hadoop distributed file system. It is suitable for large-scale model and data processing, but its configuration is complex.

[0106] EBS PV uses AWS block storage to provide high-performance single-node read and write storage. It is suitable for single-machine inference tasks, but does not support multi-node sharing. It is suitable for cloud development environments.

[0107] NVMe PV provides extremely low-latency single-node high-performance storage through local NVMe storage. It is only bound to a specific node and does not support cross-node storage. It is suitable for high-performance single-machine inference tasks.

[0108] In this embodiment, the data processing method integrates the capabilities of Kubernetes and Knative, optimizes the deployment process of multi-machine and multi-card inference services, achieves seamless elastic scaling, fine-grained traffic management, and meets the compatibility of Serverless with multiple machines and multiple cards, providing stable and efficient production-level cloud-native inference support for large models with large parameters (such as DeepSeekV3 / R1).

[0109] Fourth embodiment

[0110] The fourth embodiment of the present application relates to a data processing method. The fourth embodiment is an improvement based on the first embodiment. Figure 2 As shown, the specific improvements are:

[0111] After executing step S4, the method further includes:

[0112] S6. Release idle resources in the second inference cluster according to preset rules.

[0113] The preset rule is: if the current request traffic A is less than the total number N of queries per second threshold QPS corresponding to all currently running Knative framework instances, and NA> the query per second threshold QPS, then the Knative framework instance is triggered to scale down the corresponding inference cluster to release idle resources.

[0114] It should be noted that the fourth embodiment of the present application may also be an improvement based on the third embodiment.

[0115] It is not difficult to find that in the embodiment of the present application, the framework instance can perform lifecycle management on the reasoning cluster associated with it according to preset rules, and release the idle resources of the corresponding reasoning cluster when the current traffic is idle to avoid waste of resources.

[0116] In this embodiment, the advantages of the data processing method are as follows:

[0117] 1. Seamless elastic scaling. Compared to KubeRay's vLLM deployment solution and the static deployment solution for inference services based on Kubeflow, this implementation of ICG combined with the serverless Knative service can achieve seamless elastic scaling for multi-machine inference, avoiding inference request interruptions caused by service restarts. It also supports dynamic scaling based on QPS, significantly improving the continuity of large-model inference services and increasing resource utilization.

[0118] 2. Efficient traffic control and model management. Compared to existing large-model distributed multi-machine inference service technologies that lack fine-grained traffic control and rapid model switching capabilities, the ICG in this embodiment combines Knative's Istio gateway and Activator components to achieve concurrent traffic management and rapid multi-machine deployment of large models of different versions. This greatly simplifies the development and testing process for fine-tuning models, reduces deployment time, and avoids resource waste.

[0119] 3. The perfect integration of Serveless technology and multi-machine inference resolves the conflict between the network design concept of existing Serveless frameworks (such as Knative) that only exposes a single HTTP port and the multi-machine, multi-GPU inference communication. By exposing a unified Serve port through ICG and forwarding inference requests to a specific InferenceCluster (an API object), multi-machine inference services can leverage the many advantages of Serveless technology, such as elastic scaling and infrastructure-awareness. This allows users to focus on model development and fine-tuning, thereby improving development efficiency.

[0120] 4. Production-level inference support and multi-cloud compatibility overcome the memory bottleneck of a single machine. By binding to the cloud-native infrastructure Kubernetes, it can run on any cloud and quickly support the deployment of large-parameter models such as DeepSeek V3 / R1 when resources are sufficient. It is also compatible with multiple inference frameworks and multi-cloud deployments, improving reliability and scalability compared to existing solutions.

[0121] The steps of the various methods above are divided only for the purpose of clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application; adding insignificant modifications or introducing insignificant designs to the algorithm or process without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0122] The data processing platform of this application is further explained with the following examples:

[0123] The software versions involved are Kubernetes ≥ 1.22, KubeRay ≥ 1.2.2, KubeFlow ≥ 1.10, and Knative ≥ 1.0.0. These versions support the core functions of multi-machine and multi-GPU distributed inference and Serverless elastic scaling. They are suitable for cloud-native environments based on NVIDIA GPU clusters (such as H100) and low-latency networks (such as InfiniBand), and are integrated with InferenceClusterGateway (ICG) to achieve efficient large-model inference.

[0124] The deployment process is as follows:

[0125] The service architecture of the data processing platform is as follows: Figure 3 As shown, install Knative ≥ 1.0.0, KubeRay ≥ 1.2.2, KubeFlow ≥ 1.10, and Kubernetes ≥ 1.22;

[0126] For details on how the data processing platform processes inference requests, see Figure 4 As shown:

[0127] Upload different large model files (such as DeepSeek-R1) to the high-performance network attached storage (NAS, such as NFS) of the data processing platform to ensure multi-node access and low latency.

[0128] In the inference cloud platform, select the name of the large model to be deployed (such as DeepSeek-R1), computing power specifications (such as two 8-card H100 hosts), and inference framework image (such as vLLM-0.7.0). Specify the range of elastically scalable Pods (such as Pod1, Pod2, Pod3), set the maximum query rate per second (QPS) that a single large model inference service can support, and trigger horizontal expansion if the QPS threshold is exceeded, and vice versa, trigger scaling down, supporting scaling down to 0 Pods. Finally, support user-defined model startup commands (such as the default vllm serve, or custom use of SGLang to start multi-machine and multi-card inference services) through InferenceClusterGateway (ICG).

[0129] The platform creates a corresponding Knative Service as a model service through ICG, and sets the startup command through KubeRay or Kubeflow (using PyTorchJob or MPIJob) to run a multi-machine and multi-GPU inference service cluster with different inference frameworks such as vLLM, SGLang, or TensorRT-LLM.

[0130] After the corresponding large-model Serverless inference service cluster is successfully started, the platform exposes the OpenAI-compatible inference API address through ICG for users to call. Knative's Autoscaler dynamically adjusts the number of pods for multi-machine and multi-GPU inference services based on QPS. Queue-Proxy is responsible for traffic control to ensure service efficiency and elastic scalability.

[0131] The data processing platform is a production-level multi-machine, multi-card, large-model inference technology based on Kubernetes Serverless's seamless auto-scaling, fine-grained traffic control, and rapid deployment, which supports multiple inference frameworks. Knative's Serverless framework is used to achieve dynamic horizontal expansion based on QPS, and multi-machine, multi-card inference clusters are quickly built through KubeRay and Kubeflow. InferenceClusterGateway (ICG) is responsible for the lifecycle management of the inference cluster, traffic proxy (such as dynamic batching and concurrency management), and communication port adaptation, ensuring support for multiple inference service frameworks (such as vLLM, SGLang, TensorRT-LLM, etc.) to achieve efficient, high-throughput production-level performance under high concurrent loads. Compared with the shortcomings of existing technologies in multi-framework compatibility, elastic scaling capabilities, and Serverless integration, this application demonstrates significant innovation and flexibility.

[0132] Fifth embodiment

[0133] See Figure 3 As shown, the fifth embodiment of the present application relates to a data processing platform, including: an application layer, a service layer, a logic management layer and a storage layer.

[0134] Application layer, used to receive inference requests;

[0135] In this embodiment, the application layer is used to provide an interactive interface for users and forward the inference request to the corresponding inference cluster through a preset gateway.

[0136] Users can directly access the out-of-the-box ChatBot, Gradio, Cloud Studio, and Prompt Generator components through the platform. The ChatBot and Prompt Generator components support dynamic input (such as prompts in an OpenAI-compatible format). The Gradio and Cloud Studio components provide a visual interface or code development support. As user interaction interfaces, they forward inference requests to the multi-machine, multi-GPU inference service within the inference cluster through the InferenceClusterGateway (ICG) and Istio ingress gateway.

[0137] The ChatBot component provides a conversational interface similar to a large-scale language model (ChatGPT) and supports streaming output;

[0138] The Prompt Generator component is a visual prompt word engineering tool that generates structured prompt words.

[0139] Gradio component quickly builds model demonstration user interface UI, supports file upload / parameter adjustment;

[0140] Cloud Studio is an online development environment for components, where you can write code and call the model application programming interface (API).

[0141] A service layer, configured to provide a first reasoning cluster and / or a second reasoning framework, and process the reasoning request through the first reasoning cluster or the second reasoning framework;

[0142] In this embodiment, the service layer is responsible for processing the Kubernetes computing unit (Pod) of the inference request initiated by the user, using different inference frameworks (such as vLLM, SGLang, TensorRT-LLM) to start the corresponding multi-machine and multi-card inference cluster, and exposing the OpenAI-compatible inference API (such as v1 / completions) through InferenceClusterGateway (ICG) and Istio intelligent traffic entry (IstioIngressGateway).

[0143] vLLM Serve Cluster (i.e., vLLM type service collection): Use vLLM to start tensor parallel and pipeline parallel large model inference services on a multi-machine and multi-GPU Ray cluster. Service nodes communicate through NCCL and configure environment variables (such as MASTER_ADDR, MASTER_PORT, NCCL_SOCKET_IFNAME, NCCL_IB_DISAB LE, and NCCL_IB_HCA) to coordinate distributed tasks.

[0144] SGlang Serve Cluster (i.e., SGlang type service collection): Start SGLang through Python processes on multiple machines, use NCCL communication based on the PyTorch distributed backend, and specify MASTER_ADDR and WORLD_SIZE to synchronize service nodes.

[0145] TensorRT-LLM Serve Cluster (i.e., TensorRT-LLM type service collection): Start TensorRT-LLM through Python processes on multiple machines, synchronize tensors between GPUs based on NCCL, and configure MASTER_ADDR to coordinate multi-node tasks.

[0146] The logic management layer is configured to determine whether a first Knative framework instance is currently running. If the first Knative framework instance is currently running, identify whether a first inference cluster corresponding to the first Knative framework instance matches the inference request. If the first Knative framework instance is not currently running, or the first inference cluster does not match the inference request, create a second Knative framework instance based on the inference request, create a second inference cluster in the service layer based on the second Knative framework instance, and distribute the inference request to a corresponding service node in the second inference cluster for processing through the second Knative framework instance.

[0147] In actual applications, the logic management layer can process inference requests based on inference frameworks such as vLLM, SGLang, and TensorRT-LLM. The logical management layer uses KubeRay to create and manage the lifecycle of Ray clusters, supporting vLLM to launch tensor-parallel and pipeline-parallel inference services on multi-machine, multi-GPU Ray clusters. The logical management layer uses KubeFlow to create clusters such as SGLang and TensorRT-LLM that use the PyTorch learning framework for multi-machine, multi-GPU inference. It also launches distributed inference services based on NCCL, MPI, or Gloo using PyTorchJob (training resources) or MPIJob. The logical management layer uses Knative to provide fine-grained traffic control (via the Queue-Proxy and Autoscaler components), elastic scaling based on request volume (QPS), and version management and fast switching of model services. The ICG gateway, as a user container for Knative Ksvc, is responsible for launching and coordinating multi-machine, multi-GPU inference clusters such as vLLM, SGLang, and TensorRT-LLM. After the models in the inference cluster are successfully launched, the ICG gateway forwards inference requests to the OpenAI-compatible port service (such as v1 / completions) of the inference cluster through the ICG.

[0148] A storage layer is used to store model files and data of the first inference cluster and / or the second inference cluster.

[0149] The storage layer is responsible for storing model files and providing persistent storage for model files and data for multi-machine, multi-GPU inference services. It is associated with specific workloads in the form of PVCs. The storage layer can include multiple storage types, including at least one of local SSD PV storage, NFS PV storage, HDFS PV storage, EBS PV storage, and NVMe PV storage.

[0150] Local SSD PV provides low-latency single-node read and write storage through the local SSD. It is bound only to a specific node and cannot be shared across nodes. It is suitable for the model preheating cache layer of the inference cluster (such as the local acceleration of vLLM).

[0151] NFS PV uses the network file system to support multi-node read-write shared storage. It is suitable for multi-machine and multi-card inference clusters (such as SGLang model loading). However, it is limited by network bandwidth and has low throughput.

[0152] HDFS PV provides high-throughput and fault-tolerant multi-node storage based on the Hadoop distributed file system. It is suitable for large-scale model and data processing, but its configuration is complex.

[0153] EBS PV uses AWS block storage to provide high-performance single-node read and write storage. It is suitable for single-machine inference tasks, but does not support multi-node sharing. It is suitable for cloud development environments.

[0154] NVMe PV provides extremely low-latency single-node high-performance storage through local NVMe storage. It is only bound to a specific node and does not support cross-node storage. It is suitable for high-performance single-machine inference tasks.

[0155] In this embodiment, the data processing platform can receive an inference request through the application layer, and determine whether the first Knative framework instance is currently running through the logic management layer. If the first Knative framework instance is currently running, the first inference cluster is matched with the inference request to determine whether the current first inference cluster can process the inference request; when the first inference cluster cannot process the inference request or the Knative framework instance is not currently running, it can be expanded to create a second Knative framework instance based on the inference request, and then a second inference cluster is created through the second Knative framework instance, and the second inference cluster is used to process the inference request. In the case where the current Knative framework instance cannot meet the inference request, this application can ensure service continuity and achieve load balancing by expanding the inference cluster to avoid performance bottlenecks during traffic peaks; and use the inference cluster to achieve production-level high-concurrency large-model inference with long output context.

[0156] It is not difficult to find that this embodiment is a platform embodiment corresponding to the first embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the first embodiment.

[0157] It is worth mentioning that all modules involved in this embodiment are logical modules. In actual applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, to highlight the innovation of this application, this embodiment does not include units that are not closely related to solving the technical problems proposed by this application. However, this does not mean that other units do not exist in this embodiment.

[0158] In the embodiments of the present application, a computer program / instruction is stored on a computer-readable medium. When executed by a processor, the computer program / instruction implements the steps of the method provided in any one or more of the above embodiments. The computer-readable medium may be included in the electronic device described in the above embodiments, or it may exist independently and not be incorporated into the device. The computer-readable medium carries one or more computer-readable instructions.

[0159] It should be noted that the computer-readable medium described in this application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device.

[0160] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology for information storage. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc-read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0161] Computer program code for carrying out the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as the "Like" language or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. Where a remote computer is involved, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0162] In the above-described embodiment, can realize wholly or in part by software, hardware, firmware or its arbitrary combination.For example, can adopt application-specific integrated circuit (ASIC), general computer or any other similar hardware device to realize.In certain embodiments, the software program of the present application can be carried out to realize above steps or function by processor.Similarly, the software program of the present application (comprising relevant data structure) can be stored in computer-readable recording medium, for example, RAM memory, magnetic or optical drive or floppy disk and similar device.In addition, some steps or functions of the present application can adopt hardware to realize, for example, as the circuit that cooperates with processor to perform each step or function.

[0163] The computer program product provided by the embodiment of the present application includes one or more computer programs / instructions, and when the computer program / instructions are executed by the processor, all or part of the process or function described in the embodiment of the present application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instruction can be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium. For example, the computer instruction can be transmitted from a website site, a computer, a server or a data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium can be a magnetic medium, (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state hard disk (SSD)).

[0164] The flowcharts or block diagrams in the accompanying drawings illustrate the possible architectures, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the module, program segment or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-specific system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0165] The scope of this application is defined by the appended claims rather than the foregoing description and is therefore intended to encompass within this application all changes that come within the meaning and range of equivalents of the claims. Any reference signs in the claims should not be construed as limiting the claims to which they relate. In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim may also be implemented by one unit or device through software or hardware. Words such as "first" and "second" are only used to distinguish the description and do not indicate any particular order, nor should they be understood as indicating or implying relative importance.

[0166] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art may easily propose variations or substitutions within the technical scope disclosed in the present application, and such variations or substitutions shall be encompassed within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the scope of protection of the claims, and the above embodiments shall be regarded as exemplary and non-limiting.

Claims

1. A data processing method, characterized in that: include: receiving an inference request; Determine whether the first Knative framework instance is currently running; If the first Knative framework instance is currently running, identifying whether a first inference cluster corresponding to the first Knative framework instance matches the inference request; If the first Knative framework instance is not currently running, or the first inference cluster does not match the inference request, a second Knative framework instance is created based on the inference request, a second inference cluster is created based on the second Knative framework instance, and the inference request is assigned to the corresponding service node in the second inference cluster for processing.

2. The data processing method according to claim 1, wherein: Also includes: If the first inference cluster matches the inference request, the inference request is assigned to a corresponding service node in the first inference cluster for processing.

3. The data processing method according to claim 1 or 2, characterized in that: If the first Knative framework instance is currently running, identifying whether a first inference cluster corresponding to the first Knative framework instance matches the inference request includes: If the first Knative framework instance is currently running, identifying whether the current request traffic meets the preset conditions based on the inference request; If not, it indicates that the first inference cluster corresponding to the first Knative framework instance does not match the inference request; If so, it indicates that the first inference cluster corresponding to the first Knative framework instance matches the inference request.

4. The data processing method according to claim 1, wherein: The second Knative framework instance communicates with the second inference cluster through the NCCL communication mode, the MPI communication mode, or the Gloo communication mode.

5. The data processing method according to claim 1, wherein: The first Knative framework instance communicates with the first inference cluster via an NCCL communication mode, an MPI communication mode, or a Gloo communication mode.

6. The data processing method according to claim 1, wherein: After the step of: if the first Knative framework instance is not currently running or the first inference cluster does not match the inference request, creating a second Knative framework instance based on the inference request, creating a second inference cluster based on the second Knative framework instance, and assigning the inference request to a corresponding service node in the second inference cluster for processing, the method further includes: Free up idle resources in the second inference cluster according to preset rules.

7. The data processing method according to claim 1, wherein: The first inference cluster includes at least two types of service nodes, and / or the second inference cluster includes at least two types of service nodes.

8. A data processing platform, characterized in that: include: Application layer, used to receive inference requests; A service layer, configured to provide a first reasoning cluster and / or a second reasoning framework, and process the reasoning request through the first reasoning cluster or the second reasoning framework; The logic management layer is configured to determine whether a first Knative framework instance is currently running. If so, identify whether a first inference cluster corresponding to the first Knative framework instance matches the inference request. If the first Knative framework instance is not currently running, or the first inference cluster does not match the inference request, create a second Knative framework instance based on the inference request, create a second inference cluster in the service layer based on the second Knative framework instance, and assign the inference request to a corresponding service node in the second inference cluster for processing. A storage layer is used to store model files and data of the first inference cluster and / or the second inference cluster.

9. A computer-readable medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images