A heterogeneous adaptive access system and method for multi-class artificial intelligence inference engines

Abstract

The application discloses a heterogeneous adaptive middleware for multi-model collaborative scheduling, which is used for connecting a multi-model collaborative scheduling platform (Mparty) and a GPU / NPU heterogeneous computing resource pool; the middleware is composed of an adaptive management layer, a protocol conversion layer, a data regularization layer and an operation and maintenance aggregation layer. The Mparty platform issues an inference task, which is sequentially authenticated, adapted, protocol-converted and data-preprocessed by the middleware, and then is issued to an inference engine, the inference engine loads a large language model and relies on Kubernetes to schedule heterogeneous computing power to complete calculation; the operation and maintenance aggregation layer aggregates whole-link operation data and feeds back to the Mparty platform. The application relies on a hierarchical architecture to shield the heterogeneous differences between models and hardware, realizes standardized connection of large model inference tasks, and improves the utilization rate of heterogeneous computing power and the deployment universality.

Description

Technical Field

[0001] This invention relates to the fields of AI computing power scheduling, cloud-native middleware, and heterogeneous service access technology, specifically to a heterogeneous adaptation access system and method for multiple types of artificial intelligence inference engines. Background Technology

[0002] With the rapid development of large-scale models and AI inference technologies, computing clusters now contain various AI inference engines with different architectures and protocols, including image inference frameworks (TensorRT, ONNX Runtime, Triton) and large language inference services (LLaMA series). These inference engines exhibit significant heterogeneity in communication protocols, request parameter formats, multidimensional tensor data structures, operational health monitoring mechanisms, and performance metrics.

[0003] Existing computing power scheduling platforms mostly use fixed adapter code to hard-code connections to specific inference engines, or rely on general API gateways and ordinary adapters to complete simple HTTP protocol forwarding. These existing technologies struggle to solve the following technical problems: General gateways only forward packets and do not handle AI-specific data such as tensor structures, memory usage, and inference time, making them unsuitable for addressing the core data heterogeneity issues in AI inference scenarios. Furthermore, their high degree of hard-coded coupling means that adding a new engine requires modifying the scheduling kernel source code and restarting the service, impacting cluster stability. The inconsistent operational status and performance metrics of each engine prevent the scheduling platform from forming a globally standardized operational view, resulting in extremely high operational costs for large-scale clusters.

[0004] Therefore, the industry urgently needs a heterogeneous access solution that is specifically designed for AI inference scenarios, features kernel decoupling, dynamic hot-swappability, and adaptability to tensor heterogeneity and inference-specific metrics. Summary of the Invention

[0005] The purpose of this invention is to provide a heterogeneous adaptation access system and method for multiple types of artificial intelligence inference engines. By constructing an independent adaptation middleware that is completely decoupled from the computing power scheduling kernel, it addresses the core issues of heterogeneous communication protocols, tensor data structures, and operation and maintenance indicator systems of AI inference engines. This enables non-intrusive and rapid access to heterogeneous AI inference services, reduces platform coupling, and improves the scalability, stability, and operation and maintenance uniformity of heterogeneous AI clusters.

[0006] Technical solution To achieve the above objectives, the present invention adopts the following technical solution: This invention deploys an intermediate adaptation layer independently, completely isolated from the computing power scheduling kernel. It does not participate in resource allocation and scheduling decisions, but is only responsible for protocol translation, data normalization, lifecycle management, and indicator normalization of heterogeneous engines.

[0007] The adaptation management layer is based on a unified interface contract and loads independent adaptation logic for different inference engines. It supports dynamic registration and uninstallation at runtime without restarting the kernel and intermediate adaptation layer services.

[0008] The protocol conversion layer enables bidirectional parsing, encapsulation, and forwarding of the standard protocol of the scheduling gateway and the private protocols of each engine, breaking down barriers to heterogeneous communication.

[0009] The data normalization layer is specifically optimized for AI inference scenarios. It performs unified mapping and normalization for tensor shapes, data types, and memory layouts of different frameworks, shielding the differences in underlying model data.

[0010] A lifecycle monitoring layer and an indicator aggregation layer are set up to unify health detection, status reporting, and collection of memory and inference performance indicators, forming standardized operation and maintenance data output. Beneficial effects

[0011] Compared with the prior art, the beneficial effects of the present invention are as follows: It possesses unique creativity specific to AI scenarios. This invention differs from general API gateways and ordinary adapters. It is not limited to general message forwarding, but specifically addresses the unique problems of AI inference, such as multi-dimensional tensor structure normalization, memory resource monitoring, single inference time statistics, and model inference parameter mapping. It belongs to a dedicated adaptation architecture for AI computing power clusters, rather than a simple combination and reuse of general software adaptation modes.

[0012] Completely decouples the scheduling kernel, significantly improving cluster stability. The adaptation layer runs independently and is logically transparent to the scheduling kernel. Iterating engine interfaces or adding new engines does not require modifying the scheduling kernel source code, effectively avoiding platform cascading restarts or version incompatibility issues caused by traditional hard-coded adaptation methods.

[0013] It supports dynamic hot-swappable expansion to adapt to the iteration pace of cloud-native AI. Adaptive plugins can be dynamically loaded or unloaded during cluster operation without restarting the service, meeting the business needs of high-frequency iteration and rapid deployment of AI model services.

[0014] A unified heterogeneous operation and maintenance system reduces the difficulty of operating and maintaining large-scale clusters. Fragmented engine status, memory metrics, and inference performance metrics are output in a unified and standardized manner, enabling the scheduling platform to have a global and consistent view of the operation and maintenance of heterogeneous AI clusters.

[0015] Enhance the robustness of heterogeneous AI inference systems. By normalizing the tensor structure through a data normalization layer, the inference crash issue caused by the incompatibility of NCHW / NHWC memory layouts under heterogeneous frameworks is resolved, significantly improving the operational stability and fault tolerance of heterogeneous AI inference clusters. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall layered architecture of the present invention.

[0017] Figure 2 This is a schematic diagram illustrating the dynamic adaptation and access process for multiple types of artificial intelligence inference engines according to the present invention. Detailed Implementation

[0018] This invention is a heterogeneous adaptation middleware solution for an AI computing power scheduling platform, deployed in a Kubernetes cloud-native environment and running independently of the scheduling kernel. For example... Figure 1 As shown, the intermediate adaptation layer of this invention is deployed between the multi-model collaborative scheduling platform (Mparty scheduling platform) and various heterogeneous AI inference engines. It remains logically transparent to the scheduling kernel, does not participate in the allocation of computing resources and task scheduling decisions, and is only responsible for the protocol conversion, tensor data normalization, lifecycle management and index normalization of heterogeneous engines.

[0019] The technical solution of the present invention will be fully disclosed below with reference to two typical embodiments.

[0020] Example 1: Triton Inference Server Image Inference Engine Integration The Triton inference engine natively supports both HTTP / REST and gRPC protocols. The inference request body stores FP32 multidimensional tensor data in nested fields and comes with a Prometheus metric endpoint.

[0021] like Figure 2 As shown, the adaptation process in this embodiment specifically includes the following steps: Step S101: Adapt the management layer for hot loading. The management layer responds to the Triton engine type to be integrated by dynamically hot loading the Triton-specific adaptation plugin, which is bound to the platform's predefined unified interface contract.

[0022] Step S102: Protocol Conversion. The protocol conversion layer converts the standard gRPC inference requests issued by the Mparty scheduling platform into a Triton-compatible request format.

[0023] Step S103: Data Warming. The data warming layer performs tensor mapping: mapping the platform-unified InferenceRequest.Tensor.Data structure to Triton's native inputs[].data field, while unifying the memory layout and data type verification of NCHW (batch-channel-height-width) and NHWC (batch-height-width-channel), solving the problem of incompatibility in video memory data arrangement under heterogeneous frameworks.

[0024] Step S104: Result Return. After inference is complete, the Triton outputs[].data tensor result is deserialized into a platform standard response object and returned to the scheduling platform.

[0025] Step S105: Lifecycle Monitoring. The lifecycle monitoring layer periodically accesses Triton's health probe interface and encapsulates it into standardized online / offline status messages for reporting.

[0026] Step S106: Metric Aggregation. The metric aggregation layer periodically collects the number of successful nv_inference_request_success queries, GPU memory usage, and single inference time, and encapsulates them into platform standard metric messages, which are then reported to the Mparty scheduling platform.

[0027] This embodiment does not modify any source code of the Mparty scheduling platform kernel. It completes the non-intrusive integration of the high-performance image inference engine only through hot-loading adaptation logic.

[0028] Example 2: LLaMA Large Language Inference Service Access This embodiment selects a large model scenario with communication and data structures that are completely heterogeneous with the aforementioned Triton image engine to verify the cross-category adaptation capability of this middleware.

[0029] The LLaMA inference service uses a lightweight REST JSON interface, with prompt text parameters and token generation parameters as core inputs. It supports streaming output, has no standard tensor structure, and is completely heterogeneous with the image inference engine in terms of communication protocol and data structure.

[0030] The adaptation process in this embodiment specifically includes the following steps: Step S201: Adapt to hot reloading of the management layer. Adapt the management layer to load the LLaMA large model-specific adaptation plugin.

[0031] Step S202: Protocol Conversion. The protocol conversion layer completes the bidirectional conversion between the Mparty scheduling platform standard protocol and the LLaMAJSON REST protocol.

[0032] Step S203: Data Warming. The data warming layer performs standardized field mapping on the prompt and max_tokens parameters, and performs unified concatenation and format normalization on the streaming token-by-token output.

[0033] Step S204: Lifecycle Monitoring. Liveness detection is performed via the HTTP health probe endpoint, and standardized alarms are triggered and reported to the scheduling platform when an anomaly occurs.

[0034] Step S205: Metric Aggregation. Statistical analysis of text inference concurrency and single-round generation time is compiled and reported to the Mparty scheduling platform.

[0035] This embodiment demonstrates that the present invention is compatible with two completely heterogeneous AI engines: image reasoning and large language reasoning. It has a wide range of applications and strong versatility, and maintains logical transparency and non-intrusive decoupling from the scheduling kernel throughout the process.

Claims

1. A heterogeneous adaptation and access system for multiple types of artificial intelligence inference engines, characterized in that, Deployed between various heterogeneous artificial intelligence inference engines and platform scheduling gateways, it is constructed as an intermediate adaptation layer decoupled from the computing power scheduling kernel logic; The system includes: The management layer is configured to respond to the type of AI inference engine to be connected and load the corresponding adaptation logic for that engine. The adaptation logic follows a predefined unified interface contract. The protocol conversion layer is configured to intercept and bidirectionally convert the first communication protocol used by the platform scheduling gateway and the second communication protocol used by the artificial intelligence inference engine; The data normalization layer configures the field mapping and tensor structure normalization of inference request parameters and inference result data to output data packets that conform to the platform's agreed format. The processing logic of the intermediate adaptation layer is logically transparent to the computing power scheduling kernel and does not participate in the allocation of computing power resources and task scheduling decisions.

2. The system according to claim 1, characterized in that, The adaptation management layer supports dynamically registering or uninstalling the adaptation logic during the operation of the intermediate adaptation layer without restarting the computing power scheduling kernel and the intermediate adaptation layer.

3. The system according to claim 1, characterized in that, It also includes a lifecycle monitoring layer, which encapsulates unified engine start / stop commands and health detection rules, is backward compatible with the native operation and maintenance interfaces of different artificial intelligence inference engines, and outputs standardized online / offline status messages.

4. The system according to claim 1, characterized in that, It also includes an indicator aggregation layer, which is used to periodically collect the memory usage, number of concurrent inference tasks and single inference time of different artificial intelligence inference engines, and encapsulate the heterogeneous raw running data into a platform-standardized indicator format before reporting it to the scheduling gateway.

5. A heterogeneous adaptation and access method for multiple types of artificial intelligence inference engines, characterized in that, Using the system according to any one of claims 1 to 4, the steps include: Loading steps: Dynamically load the corresponding adaptation logic according to the type of the artificial intelligence inference engine; Protocol adaptation steps: Receive a call request from the platform scheduling gateway to the artificial intelligence inference engine, convert the communication protocol of the call request into a protocol format supported by the artificial intelligence inference engine, and then forward it; Data transformation steps: Perform field mapping and data structure normalization on the tensor descriptors in the request input parameters and the inference output in the returned results; Reporting steps: Monitor the engine's operating status and collect operating metrics, and report them back to the platform's scheduling gateway in a standardized format.

6. The method according to claim 5, characterized in that, The adaptation logic only implements the protocol parsing method, parameter mapping method, and indicator extraction method defined in the unified interface contract, so that the access of the new artificial intelligence inference engine does not require modification of the source code of the computing power scheduling kernel.

Images