MaaS platform large model management method and system based on intelligent orchestration engine

Abstract

The application provides a MaaS platform large model management method and system based on an intelligent arrangement engine, comprising: performing format verification and metadata extraction on an uploaded large model file; encapsulating operations and configurations in a model deployment process as standardized components; generating a candidate deployment scheme set based on constraint conditions of the metadata and the components; performing multi-dimensional scoring and sorting on the candidate deployment schemes, and outputting an optimal deployment scheme; and performing pre-release pre-checking, container image building and Kubernetes resource arrangement according to the optimal deployment scheme, and completing automatic deployment of a model service. Through a component encapsulation mechanism, the application uniformly manages various configuration templates, supports component reuse across models and across scenes, can directly inherit verified component configurations when a new model is deployed, avoids repeated development work, shortens the cross-scene adaptation cycle, and reduces the probability of configuration errors.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a method and system for managing large models on a MaaS platform based on an intelligent orchestration engine. Background Technology

[0002] With the widespread application of large-scale pre-trained models such as GPT, LLaMA, and BERT in fields such as computer vision, natural language processing, and multimodal understanding, MaaS platforms, as the infrastructure for deploying and serving large models, are becoming increasingly important. However, existing MaaS platforms face numerous technical challenges in managing large models, which severely restricts the large-scale application and promotion of these models.

[0003] Existing platforms generally employ fixed hardware allocation strategies, lacking intelligent matching mechanisms between model characteristics and hardware resources. Operations personnel must manually select GPU models, configure memory specifications, and set parallel strategies based on experience. This extensive resource management approach leads to low GPU resource utilization and significant waste of computing power. Models of different sizes have significantly different requirements for memory, computing power, and bandwidth. For example, a 7B parameter model and a 70B parameter model differ by orders of magnitude in hardware configuration, but existing platforms lack automated requirements analysis and resource matching capabilities, often resulting in small models consuming excessive resources or large models lacking sufficient resources.

[0004] Automation is also insufficient. Model deployment involves multiple steps, including API documentation generation, user experience page configuration, authentication method setting, billing rule configuration, and product information maintenance, each requiring manual intervention. This cumbersome deployment process is not only time-consuming but also prone to configuration errors due to human error, impacting service quality and user experience. When multiple models need to be deployed in batches, the repetitive manual operations further highlight the efficiency bottleneck.

[0005] Current technologies rely heavily on human experience. First, hardware adaptation requires maintenance personnel to possess deep expertise, enabling them to determine suitable hardware configurations based on factors such as model parameter count, computational complexity, and accuracy requirements. Insufficient experience can easily lead to resource waste or performance bottlenecks. Second, the selection of inference frameworks lacks systematic guidance. Different frameworks exhibit significant performance differences in different scenarios. For example, vLLM is suitable for long-context continuous batch processing, TensorRT-LLM for low-latency scenarios, and DeepSpeed ​​for deploying ultra-large models, but existing platforms fail to provide framework recommendation mechanisms based on model characteristics. Third, the development of interface protocols involves a significant amount of repetitive work. Different models require separate development of input / output adaptation interfaces, resulting in high configuration complexity and a high risk of errors.

[0006] In summary, existing technologies are insufficient to meet the needs of MaaS platforms, and a systematic solution is urgently needed to improve the management and deployment efficiency of large models. Summary of the Invention

[0007] To achieve the above objectives, this invention proposes a method for managing large models on a MaaS platform based on an intelligent orchestration engine, comprising the following steps:

[0008] S1. Perform format verification and security scanning on large model files uploaded by users, and automatically extract the structured metadata of the model through the metadata parsing engine. The metadata includes model architecture features, computation and parameter features, input and output configuration, and hardware and framework dependency information. S2. All operations and configurations in the three stages of model management, deployment and release are encapsulated into standardized components. The standardized components include hardware adaptation components, inference framework components, interface protocol components, interface specification components and model release components. Each component has an independent range of applicable parameters, constraints and compatibility rules, and supports parameter configuration within the component and cross-scenario reuse. S3. Based on the metadata and constraints of each component, construct a component constraint graph and perform constraint checks to exclude incompatible component combinations and generate a set of candidate deployment schemes; obtain the weight configuration of the evaluation dimensions from the preset scenario-weight mapping table according to the application scenario tags, use a multi-objective optimization function to perform multi-dimensional scoring and sorting of the candidate deployment schemes, and output the optimal deployment scheme; S4. Based on the optimal deployment scheme, perform configuration file generation, resource pre-check, and container image building; S5. Based on Kubernetes, containerized resource orchestration is used to complete the automated deployment of model services and synchronize the deployed model service information to the inference gateway. S6. Persist the model listing information to the database, automatically generate experience pages and interface documents, and complete the e-commerce release of the model service.

[0009] Furthermore, metadata extraction includes: The model architecture features include: model type, total number of layers, number of attention heads, hidden layer dimension, and positional encoding type, where the positional encoding type includes RoPE and ALiBi; The computational and parameter features include: total parameter size, training and inference accuracy, weight sparsity ratio, and activation function type, where the accuracy type includes FP32, FP16, BF16, and FP8; The input / output configuration includes: maximum sequence length, word segmenter type, multimodal support type, and default values ​​for generation parameters; The hardware and framework dependency information includes: minimum memory requirements, CUDA version, acceleration library compatibility, training framework type, and required instruction set.

[0010] Furthermore, the hardware adaptation components include three adaptation dimensions: graphics memory optimization, computing core selection, and storage bandwidth configuration. Memory optimization dimension: Based on the number of model parameters, the memory requirements for forward computation, and the KV Cache requirements, the total memory requirements are dynamically calculated using the memory requirement formula. A single-card or multi-card parallel solution is selected, and FP32, FP16, and INT8 memory precision switching is supported. The memory requirement formula is: Total memory requirement = Parameter memory + Activated memory + KV Cache memory, where parameter memory (FP16) = Number of parameters × 2 bytes. Computing core selection dimensions: For high computing power scenarios, prioritize hardware that supports Tensor Cores or TPU matrix units. For edge scenarios, support lightweight deployment of CPUs or NPUs. Computing power requirements are calculated using the formula: Computing power requirements = Model FLOPs / Hardware peak FLOPs. Storage bandwidth configuration dimension: Automatically allocate disk space based on model file size and storage precision, and verify whether the network bandwidth threshold meets the preset requirements when loading multiple cards.

[0011] Furthermore, the inference framework component provides multi-framework compatibility adaptation, including: vLLM framework: suitable for long-context continuous batch processing optimization scenarios; TensorRT-LLM framework: suitable for low-latency scenarios of tensor parallelism on NVIDIA GPU clusters; DeepSpeed-Inference framework: suitable for ZeRO-Infinity memory offloading scenarios for ultra-large models; ONNX Runtime Framework: Suitable for cross-platform deployment scenarios of CPU, GPU, TPU, and ARM; SparseML framework: suitable for edge deployment scenarios with pruning and quantization-based extreme compression.

[0012] Furthermore, the matching rules for the inference framework components include: When the model uses RoPE positional encoding and requires sequence length extrapolation, exclude frames that do not support dynamic length expansion; When the number of model parameters exceeds a preset threshold, only frameworks that support distributed parallelism will be retained. When the scene is labeled as a low-latency API, prioritize frameworks that support Tensor Core acceleration. When the scenario is labeled as edge deployment, lightweight frameworks that support model quantization should be selected first.

[0013] Furthermore, the matching of the interface protocol components includes: The protocol type is determined based on the expected request frequency, latency requirements, whether streaming is required, and the type of deployment environment. RESTful HTTP protocol: uses JSON text transmission, is compatible with all HTTP clients, and is suitable for general API calls and rapid integration scenarios; gRPC protocol: It adopts the Protobuf binary protocol, supports streaming communication, and is suitable for low-latency interaction scenarios between microservices with high frequency and low load. WebSocket protocol: It adopts a full-duplex long connection, supports real-time push of word-by-word generation, and is suitable for scenarios of word-by-word response in dialogue systems and long text streaming output. MQTT protocol: A lightweight IoT protocol with QoS guarantees, suitable for edge device inference scenarios.

[0014] Furthermore, the interface specification component defines a standardized input / output structure, including: Wenshengwen Interface Specification: The input format is a JSON object containing an array of messages, where each message in the message array contains a role and a content field; the output format is a JSON object containing an array of selections, where each selection in the selection array contains the generated message content. The Wenshengtu interface specification states that the input format is a JSON object containing input text and parameter configuration, including image ratio and editing mode; the output format is a JSON object containing a data array, where each element contains the Base64 encoded image or URL address. Multimodal Mixed Interface Specification: The input format supports nesting multimodal objects containing type and resource address within the message content; the output format includes both text and image content.

[0015] Furthermore, the model publishing component includes: Authentication mode configuration: Supports API Key open access mode and OAuth 2.0 enterprise single sign-on mode; Metering configuration: Supports token-based billing mode, which charges based on input and output length; supports fixed-fee mode, which charges per call. Experience the template library: Provides standard page templates for text-to-text, text-to-image, and speech recognition, and supports preset sample data.

[0016] Furthermore, the constraint check in step S3 includes: Construct a component constraint graph, where nodes are optional components and edges represent the constraint relationships between components; The constraints are divided into hard constraints and soft constraints; the hard constraints include the compatibility constraints between the location encoding type and the inference framework, and the driver compatibility constraints between the GPU model and the framework; the soft constraints include performance optimization suggestion constraints. Traverse the component constraint graph, mark components that violate hard constraints as unselectable, and record penalty items for components that violate soft constraints for subsequent score adjustments.

[0017] Furthermore, the multi-objective optimization function is: ; in, This refers to resource costs, including the weighted overhead of GPU memory, CPU core count, and system memory. Inference latency is estimated based on the framework performance table; Historical success rate of component combinations; , , These are the weighting coefficients; The weighting coefficients are obtained from the scenario-weighting mapping table based on the application scenario. High-concurrency scenarios correspond to latency-priority weighting, cost-sensitive scenarios correspond to the highest weighting of the resource cost dimension, and high-availability scenarios correspond to the highest weighting of the deployment reliability dimension.

[0018] Furthermore, the multidimensional scoring includes: Evaluation dimensions include resource cost, inference latency, deployment reliability, and elastic scaling capability; The scenario-weight mapping table pre-sets the correspondence between various scenario types and weight configurations. Cost-sensitive scenarios correspond to the highest weight in the resource cost dimension, latency-sensitive scenarios correspond to the highest weight in the inference latency dimension, and high-availability scenarios correspond to the highest weight in the deployment reliability dimension. For each candidate solution, a score is calculated for each dimension. The optimal deployment solution is then output after weighting and sorting according to the weight configuration.

[0019] Furthermore, step S3 also includes dynamic memory calculation: Based on the model metadata and expected concurrency, the total video memory requirement is calculated using the following formula: Total video memory = Parameter video memory + KV cache + Activation value Wherein, parameter memory = number of parameters × number of precision bytes; KV Cache = number of concurrency × sequence length × 2 × number of layers × hidden layer dimension × number of precision bytes; Activation value = Parameter memory × Preset activation coefficient; Based on the calculated total video memory requirements and single-card capacity, determine the number of GPUs: Number of GPUs = Rounded up (Total video memory / Single-card capacity).

[0020] Further, the configuration file generation in step S4 includes: Select the corresponding Helm Chart configuration template based on the orchestration scheme; Through dynamic parameter rendering, parameters such as the number of GPUs, frame type, and model path in the orchestration scheme are injected into template variables; Generate a Kubernetes configuration manifest, including Deployment configuration, Service configuration, ConfigMap configuration, and PersistentVolumeClaim configuration; The parameter mapping relationships include: the number of hardware GPUs in the orchestration scheme is mapped to the resource limit configuration item of the Deployment, the framework type is mapped to the environment variable configuration item of the container, the model storage path is mapped to the capacity configuration item of the persistent storage volume, and the service interface type is mapped to the port configuration item of the Service.

[0021] Furthermore, the resource pre-check in step S4 includes: Resource availability check: Query the real-time resource status of the target cluster nodes to verify that the available GPUs, CPUs, and memory meet the deployment requirements; Quota verification: Check whether the resource quota of the target namespace is sufficient, and verify that the resources requested by the current deployment do not exceed the namespace quota limit; Conflict detection: Detects whether the service name and port are already in use, and verifies that the cluster has permission to pull images from the image repository; If any check fails, the release process is terminated and a pre-check report containing detailed error information is output.

[0022] Furthermore, the container image building in step S4 includes: Base Image Selection: Select the base image based on the inference framework type and target hardware platform. For the TensorRT-LLM framework, select the NVIDIA base image that includes the CUDA environment. For the vLLM framework, select the base image that integrates PyTorch and CUDA. For the DeepSpeed ​​framework, select the development version base image that includes the compilation environment. Automatic Dockerfile generation: Select and populate the corresponding Dockerfile template based on the orchestration results. The template content includes base image selection, system dependency installation, Python framework installation, model file copying, and startup script configuration. Image layering optimization: The image layers are designed from low to high change frequency, namely system dependency layer, framework dependency layer, configuration file layer and model file layer. The model file is treated as an independent layer so that the model layer cache can be reused when the framework or configuration changes.

[0023] Furthermore, the Kubernetes resource orchestration in step S5 includes: Hardware resource configuration: Configure the CPU or TPU model and quantity, GPU memory size, storage size, RAM size and network bandwidth requirements according to the orchestration scheme; Deployment configuration: Set the number of replicas, container image address, GPU resource limit, memory limit, and model storage volume mount path; Storage resource configuration: Create a PersistentVolumeClaim to create a persistent storage volume for model storage, and configure the access mode and storage capacity; Service exposure configuration: Create a Service, configure load balancing, set the service selector, port mapping, and service type; configure HPA elastic scaling policy, and set the threshold for triggering expansion based on CPU utilization.

[0024] Furthermore, the inference gateway synchronization in step S5 includes: Synchronize model call information, including service identifier, service name, service code, scope of use, service status, version number, call type, minimum number of instances, QPS limit, and gateway instance path; Synchronize authentication information, including application identifier, service identifier, total number of calls, remaining number of calls, validity period start and end time, service type, QPS limit and billing type; The inference gateway receives external call requests and forwards them to the specific model container based on synchronized routing information and authentication rules.

[0025] Furthermore, the online store listing in step S6 includes: Model information persistence: Persistently store model identifier, model name, model description, and experience page address in the database; Experience page generation: Based on model metadata and scene tags, the template engine automatically renders dialogue templates, pre-sets industry sample prompts, and integrates the API call SDK; When a user visits, the backend queries the database for model introduction, model experience, and experience page information, and returns it to the frontend for display. The experience page is then redirected via the address returned by the backend.

[0026] Furthermore, after outputting the optimal deployment plan, user intervention steps are also included: The model metadata is rendered in a visual interface, displaying the optional components and system-recommended options for each stage; Users can modify model feature parameters or manually replace component selections. After modification, dynamic validation is performed in real time to verify component compatibility. After the verification passes, the constraint check and scoring sorting are re-executed to generate an updated deployment configuration file.

[0027] Furthermore, it also includes feedback optimization steps: Collect deployment results and runtime performance metrics, including deployment success or failure status, actual inference latency, and actual resource usage; Update the historical success rate of the component combination based on the deployment results, and use it for compatibility dimension calculation in subsequent scoring and ranking. If a component combination fails to deploy multiple times consecutively, it will be added to the hard constraint rule library and automatically excluded from subsequent orchestration.

[0028] The MaaS platform large model management system based on an intelligent orchestration engine is applicable to the aforementioned MaaS platform large model management methods based on intelligent orchestration engines, including: The metadata parsing engine is used to perform format verification and security scanning on uploaded large model files, extract the model's architectural features, computation and parameter features, input and output configurations, and hardware and framework dependency information, and generate structured metadata. The component library stores configuration templates for hardware adaptation components, inference framework components, interface protocol components, interface specification components, and model publishing components. Each component has its own applicable parameter range, constraints, and compatibility rules. The intelligent orchestration engine is used to build component constraint graphs based on model metadata and component constraints, perform hard and soft constraint checks, and exclude incompatible component combinations; it obtains weight configurations from the scenario-weight mapping table according to application scenario tags, uses a multi-objective optimization function to score and rank candidate solutions, and outputs the optimal deployment solution. The configuration generator maps deployment scheme parameters to Kubernetes configuration items, generating configuration manifests for Deployment, Service, ConfigMap, and PersistentVolumeClaim. The pre-check engine is used to verify cluster resource availability, quota limits, and conflict detection. The image builder is used to select a base image based on the inference framework type, build container images according to a layered strategy, and push them to the image repository. The automated deployment module is used to perform containerized resource orchestration based on Kubernetes, complete the deployment of model services, and synchronize service information to the inference gateway; The e-commerce platform publishing module is used to persist model listing information to the database and automatically generate experience pages and API documentation.

[0029] Furthermore, the intelligent orchestration engine includes: The constraint checking unit is used to construct the component constraint graph, where nodes are optional components and edges represent the constraint relationships between components; it performs hard constraint checks, marking components that violate constraints as unselectable; and it performs soft constraint checks, recording penalty items for score adjustment. The video memory computing unit is used to calculate the total video memory requirement based on the model parameter quantity, precision, concurrency, sequence length, number of layers, and hidden layer dimension, and to determine the number of GPUs and parallel strategy. The scene recognition unit is used to obtain the weight configuration from the scene-weight mapping table based on the application scene label, and determine the weight coefficients of each dimension of resource cost, inference latency, deployment reliability and elastic scalability. The scoring and sorting unit is used to score candidate solutions in multiple dimensions using a multi-objective optimization function, and outputs the optimal deployment solution by sorting the comprehensive scores. Interactive units are used to render model metadata and optional components, receive user adjustment commands, and trigger recalculation.

[0030] Furthermore, the automated publishing module includes: The container orchestration unit is used to create Deployment, Service, and PersistentVolumeClaim resources in a Kubernetes cluster based on the configuration manifest, and to configure GPU resource limits, memory limits, storage volume mounting, and load balancing. The elastic scaling unit is used to configure HorizontalPodAutoscaler, setting the threshold for triggering scaling based on CPU utilization and the maximum and minimum number of replicas; The gateway synchronization unit is used to synchronize the deployed model service information to the inference gateway, including service routing address, load balancing strategy, authentication method and call quota configuration.

[0031] Furthermore, it also includes a feedback optimization module: The metrics collection unit is used to collect deployment results and runtime performance metrics, including deployment success or failure status, actual inference latency, and actual resource usage. The success rate update unit is used to update the historical success rate of the component combination based on the deployment results, and store the success rate data in the component library for subsequent orchestration. The rule base update unit is used to monitor the number of consecutive failures of a component combination. When the number of consecutive failures exceeds a preset threshold, the component combination is added to the hard constraint rule base.

[0032] Furthermore, the configuration templates for each component in the component library include: Hardware adaptation component template: includes a list of chip models, resource specifications, video memory capacity, computing power indicators, storage configuration and network bandwidth configuration, as well as the applicable parameter range and constraints for each configuration item; Inference framework component template: includes framework name, supported model architecture types, supported precision types, whether distributed parallelism is supported, whether length extrapolation is supported, performance metrics, and compatibility rules; Interface protocol component template: includes protocol type, transmission format, whether streaming transmission is supported, applicable scenario tags, and performance characteristics; Interface specification component template: includes interface type, input format definition, output format definition, parameter validation rules, and applicable model types; Model publishing component template: includes authentication mode configuration, metering method configuration, experience page template, and product information field definition.

[0033] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention provides a method and system for managing large models on a MaaS platform based on an intelligent orchestration engine, significantly improving hardware resource utilization. Through formulaic calculation of GPU memory requirements based on model metadata and an intelligent hardware matching mechanism, the system can automatically calculate the optimal hardware configuration based on factors such as the number of model parameters, accuracy type, and concurrency requirements. This avoids the resource waste caused by traditional manual configuration methods, significantly reduces GPU idle time, and effectively improves the efficiency of computing resources.

[0034] 2. This invention provides a method and system for managing large models on a MaaS platform based on an intelligent orchestration engine. By encapsulating operations such as hardware adaptation, inference framework selection, and interface protocol matching into standardized components, and having the intelligent orchestration engine automatically complete component combination and parameter configuration based on model characteristics and application scenarios, this invention enables operation and maintenance personnel without in-depth professional backgrounds to quickly complete the deployment of large models, thereby reducing the technical threshold and labor costs.

[0035] 3. This invention provides a method and system for managing large models on a MaaS platform based on an intelligent orchestration engine. Through an automated release module, it enables one-click completion of interface document generation, experience page configuration, authentication method setting, and billing rule configuration, compressing the traditional manual release process that takes several hours to minutes, greatly improving the efficiency of model deployment and supporting rapid batch deployment of multiple models.

[0036] 4. This invention provides a method and system for managing large models on a MaaS platform based on an intelligent orchestration engine. By collecting deployment results and runtime performance indicators through a feedback optimization module, the historical success rate of component combinations is continuously updated, and continuously failed component combinations are automatically added to the constraint rule base. This enables the system to learn from historical experience, continuously improve the accuracy and reliability of orchestration decisions, and achieve intelligent continuous improvement. Attached Figure Description

[0037] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 This is a flowchart of the MaaS platform large model management method provided in the embodiments of the present invention; Figure 2 This is a diagram of the MaaS platform large model management system architecture provided in an embodiment of the present invention. Detailed Implementation

[0039] The technical solution of the present invention will be more clearly and completely explained below with reference to the accompanying drawings and through the description of preferred embodiments of the present invention.

[0040] like Figure 1 As shown, the method for managing large models on a MaaS platform based on component-based encapsulation and intelligent orchestration engine provided by this invention includes the following steps: Step S1 involves format verification and metadata extraction of the uploaded large model file. The system first uses the SHA256 algorithm to perform hash verification on the model file to ensure file transmission integrity. Then, it verifies whether the model file format conforms to the system's supported format list, including .bin, .safetensors, PyTorch, ONNX, etc. Finally, by parsing the model configuration file, it automatically extracts the model's architectural features, computational features, and hardware dependency information. Architectural features include the number of layers, number of parameters, number of attention heads, hidden layer dimension, and positional encoding type. Computational features include precision type, sparsity, and activation function. Hardware dependency information includes minimum GPU memory requirements, CUDA version requirements, and instruction set dependencies.

[0041] Step S2 encapsulates the operations and configurations in the model deployment process into standardized components, each with clearly defined applicable parameter ranges and constraints. The hardware adaptation component dynamically calculates hardware configuration based on a memory requirement formula: total memory equals parameter memory plus KV Cache plus activation value. It supports single-card or multi-card parallel solutions and heterogeneous hardware scheduling such as NVIDIA GPUs and domestic Ascend chips. The inference framework component pre-configures multiple inference framework templates, including vLLM, TensorRT-LLM, DeepSpeed-Inference, and ONNX Runtime, each labeled with its applicable scenarios, performance characteristics, and compatibility constraints. The interface protocol component provides protocol templates such as RESTful HTTP, gRPC, WebSocket, and MQTT, automatically matching based on request frequency, latency requirements, and streaming needs. The model publishing component encapsulates configuration items such as authentication modes, metering methods, and user experience page templates.

[0042] Step S3: Based on metadata and component constraints, incompatible component combinations are excluded through constraint checks, generating a set of candidate deployment schemes. The system first constructs a component constraint graph, with selectable components as nodes and constraint relationships between components as edges. Then, constraints are divided into hard constraints and soft constraints. Hard constraints include compatibility constraints between location encoding type and inference framework, and GPU model and driver version constraints. Soft constraints include performance optimization suggestions. Next, the component constraint graph is traversed, marking components that violate hard constraints as unselectable, and recording penalty items for components that violate soft constraints for subsequent scoring adjustments. Finally, components that pass the constraint checks are combined to generate a set of candidate deployment schemes.

[0043] Step S4: Based on the application scenario tags, retrieve the weight configurations of the evaluation dimensions from the preset scenario-weight mapping table, and perform multi-dimensional scoring and ranking of candidate deployment solutions. Evaluation dimensions include resource cost, inference latency, deployment reliability, and elastic scaling capability. The scenario-weight mapping table pre-defines the correspondence between various scenario types and weight configurations: cost-sensitive scenarios have the highest weight for the resource cost dimension, latency-sensitive scenarios have the highest weight for the inference latency dimension, and high-availability scenarios have the highest weight for the deployment reliability dimension. The system uses a multi-objective optimization function to calculate a comprehensive score for each candidate solution, outputs the optimal deployment solution in ascending order of comprehensive score, and allows users to adjust component selections or modify parameters through a visual interface. The system re-executes constraint checks and scoring / ranking in real time.

[0044] Step S5: Based on the optimal deployment plan, perform pre-deployment checks, container image building, and Kubernetes resource orchestration to complete the automated deployment of the model service. Pre-deployment checks include querying the target cluster's allocable resources to verify deployment requirements, checking resource quotas in the target namespace, and detecting service name and port conflicts. If any check fails, the deployment process terminates and an error message is output. Container image building selects a base image based on the inference framework type and target hardware platform, designing image layers from low to high change frequency: system dependency layer, framework dependency layer, configuration file layer, and model file layer. Model files are treated as an independent layer to facilitate caching and reuse. Kubernetes resource orchestration maps deployment plan parameters to K8s configuration items, generating Deployment and Service configuration manifests. The system also synchronizes the deployed model service information to the inference gateway, including service routing addresses, load balancing strategies, authentication methods, and call quota configurations.

[0045] like Figure 2 As shown, the MaaS platform large model management system based on component-based encapsulation and intelligent orchestration engine provided by this invention includes a metadata parsing engine, a component library, an intelligent orchestration engine, an automated release module, and a feedback optimization module.

[0046] The metadata parsing engine is used to perform format validation on uploaded large model files and extract the model's architectural features, computational features, and hardware dependency information. The engine uses structured parsing technology to automatically extract metadata information such as the number of layers, number of parameters, and number of attention heads from the model configuration file and output it in JSON format for use by subsequent modules.

[0047] The component library stores configuration templates for hardware adaptation components, inference framework components, interface protocol components, and model publishing components. Each component has a clearly defined range of applicable parameters and constraints, and supports adding, deleting, modifying, querying, and version management of components.

[0048] The intelligent orchestration engine performs intelligent orchestration based on model metadata and component constraints. It includes a constraint checking unit, a scene recognition unit, a scoring and ranking unit, and an interaction unit. The constraint checking unit is used to construct component constraint graphs and perform checks on hard and soft constraints. The scene recognition unit is used to obtain weight configurations from the scene-weight mapping table based on application scene labels. The scoring and ranking unit is used to perform multi-dimensional scoring on candidate solutions and sort them according to the comprehensive score. The interaction unit is used to receive user adjustment instructions and trigger recalculation.

[0049] The automated deployment module is used to perform automated deployments based on the optimal deployment scheme. It includes a pre-check engine, a configuration generator, and an image builder. The pre-check engine is used to verify the availability of cluster resources and quota limits. The configuration generator is used to map deployment scheme parameters to Kubernetes configuration items and generate Deployment and Service configuration manifests. The image builder is used to build container images according to a layered strategy and push them to the image repository.

[0050] The feedback optimization module is used to collect deployment results and runtime performance metrics, update the historical success rate of component combinations, and add continuously failing component combinations to the constraint rule base to achieve system self-optimization and continuous improvement.

[0051] As a specific embodiment, taking the deployment of an LLaMA-2 model with 70B parameters as an enterprise-level API service as an example, the specific application process of the method of the present invention is illustrated. The input model file is LLaMA-2-70B-chat.safetensors, with a file size of approximately 140GB. The application scenario is an enterprise-level API service, requiring low latency and high availability, with an expected concurrency of 100 concurrent requests.

[0052] In step S1, the system performs metadata extraction on the model file to obtain the following structured information: the number of parameters is 70B, the precision type is FP16 (i.e., 2 bytes per parameter), the maximum sequence length is 4096 tokens, the number of layers is 80, the hidden layer dimension is 8192, and the position encoding type is RoPE.

[0053] In step S2, the system automatically calculates the hardware requirements based on the metadata. The parameter video memory equals 70 bytes multiplied by 2 bytes, which equals 140GB. The KV Cache equals 100 concurrent requests multiplied by 4096 sequence length multiplied by 2 multiplied by 80 layers multiplied by 8192 dimensions multiplied by 2 bytes, approximately 26GB. The active value video memory is approximately 20% of the parameter video memory, i.e., 28GB. The total video memory requirement equals 140 + 26 + 28 equals 194GB.

[0054] In step S3, the system performs constraint checks. For hard constraints, since the model uses RoPE positional encoding, the system excludes vLLM frameworks that do not support length extrapolation; since the parameter count of 70B exceeds the 10B threshold, the system only retains frameworks that support distributed parallelism. For soft constraints, since the application scenario is an enterprise-level API, the system prioritizes low-latency frameworks. The available frameworks after constraint checks are TensorRT-LLM and DeepSpeed-Inference.

[0055] In step S4, the system retrieves the weight configuration from the scene-weight mapping table based on the scene tag "Enterprise-level API". The resource cost weight is 0.3, the inference latency weight is 0.5, and the reliability weight is 0.2. The system scores the candidate solutions. Solution 1 uses 4 A100-80GB cards with TensorRT-LLM and gRPC protocol, with a resource cost of $3.2 per hour, an inference latency of 35 milliseconds, a reliability of 0.95, and a comprehensive score of 18.06. Solution 2 uses 8 A100-40GB cards with DeepSpeed ​​and gRPC protocol, with a resource cost of $6.4 per hour, an inference latency of 45 milliseconds, a reliability of 0.92, and a comprehensive score of 25.48. The system outputs Solution 1, with the lowest score, as the optimal deployment solution.

[0056] In step S5, the system first performs a pre-check to confirm that the target cluster's four A100-80GB GPUs are available, have sufficient quotas, and have no port conflicts. Then, it builds a container image, selecting nvidia / cuda:12.1-runtime-ubuntu20.04 as the base image and installing the TensorRT-LLM v0.8.0 framework. Next, it generates a Kubernetes configuration, configuring the Deployment with four GPUs, a memory limit of 80GB per GPU, and a tensor parallelism of 4. Finally, it synchronizes the inference gateway, with the service route / v1 / chat / llama-70b, authentication via API Key, and a QPS limit of 1000.

[0057] The comparison of the effects after deployment shows that the deployment cycle was shortened from 4 hours in the traditional solution to 8 minutes, a reduction of 96%; GPU utilization increased from 35% to 92%, an improvement of 163%; inference latency decreased from 120 milliseconds to 35 milliseconds, a reduction of 71%; and resource costs decreased from $8.5 per hour to $3.2 per hour, a reduction of 62%.

[0058] Another example is deploying a lightweight 7B-parameter model to an edge device. The input model file is Qwen-7B-Chat, approximately 14GB at FP16 precision. The application scenario is an industrial quality inspection edge box, requiring low cost and support for offline operation. The target hardware is a single RTX 4090 card with 24GB of VRAM. In the constraint check, the VRAM constraint is that 14GB of parameter VRAM plus 4GB of KV cache is less than 24GB, allowing deployment on a single card. The scenario constraint is edge deployment, prioritizing lightweight frameworks that support quantization. The system selects the ONNX Runtime framework and enables INT8 quantization. The model is compressed to 7GB, and the interface protocol is the lightweight IoT protocol MQTT. After deployment, the deployment cycle is 3 minutes, the inference latency is 50 milliseconds, the VRAM usage is 7GB, and offline operation is supported.

[0059] The above-described specific embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Various modifications, substitutions, and improvements made by those skilled in the art to the technical solutions of the present invention based on the provided textual description and drawings, without departing from the design concept and spirit of the present invention, should all fall within the scope of protection of the present invention. The scope of protection of the present invention is determined by the claims.

Claims

1. A method for managing large models on a MaaS platform based on an intelligent orchestration engine, characterized in that, include: Step S1: Perform format verification and metadata extraction on the uploaded large model file. The metadata includes model architecture features, computational features, and hardware dependency information. Step S2: Encapsulate the operations and configurations in the model deployment process into standardized components. The standardized components include hardware adaptation components, inference framework components, interface protocol components, and model publishing components. Each component has parameter ranges and constraints. Step S3: Based on metadata and component constraints, incompatible component combinations are excluded through constraint checks, generating a set of candidate deployment schemes; the constraint checks include: Construct a component constraint graph, where nodes are optional components and edges represent the constraint relationships between components; The constraints are divided into hard constraints and soft constraints. The hard constraints include the compatibility constraints between the location encoding type and the inference framework, and the driver compatibility constraints between the GPU model and the framework. The soft constraints include performance optimization suggestion constraints. Traverse the component constraint graph, mark components that violate hard constraints as unselectable, and record penalty items for components that violate soft constraints for subsequent score adjustments; Step S3 also includes dynamic memory calculation: Based on the model metadata and expected concurrency, the total video memory requirement is calculated using the following formula: Total video memory = Parameter video memory + KV cache + Activation value Wherein, parameter memory = number of parameters × number of precision bytes; KV Cache = number of concurrency × sequence length × 2 × number of layers × hidden layer dimension × number of precision bytes; Activation value = Parameter memory × Preset activation coefficient; Based on the calculated total video memory requirements and single-card capacity, determine the number of GPUs: Number of GPUs = Rounded up (Total video memory / Single-card capacity); Step S4: Obtain the weight configuration of the evaluation dimensions from the preset scenario-weight mapping table according to the application scenario tags, use a multi-objective optimization function to perform multi-dimensional scoring and ranking of candidate deployment schemes, and output the optimal deployment scheme; The multi-objective optimization function is: ； in, This refers to resource costs, including the weighted overhead of GPU memory, CPU core count, and system memory. Inference latency is estimated based on the framework performance table; Historical success rate of component combinations; , , These are the weighting coefficients; The weight coefficients are obtained from the scenario-weight mapping table according to the application scenario. High concurrency scenarios correspond to latency priority weight configuration, cost-sensitive scenarios correspond to the highest weight of resource cost dimension, and high availability scenarios correspond to the highest weight of deployment reliability dimension. Step S5: Based on the optimal deployment scheme, perform pre-release checks, container image building, and Kubernetes resource orchestration to complete the automated deployment of the model service; It also includes feedback optimization steps: Collect deployment results and runtime performance metrics, including deployment success or failure status, actual inference latency, and actual resource usage; Update the historical success rate of the component combination based on the deployment results, and use it for compatibility dimension calculation in subsequent scoring and ranking. If a component combination fails to deploy multiple times consecutively, it will be added to the hard constraint rule library and automatically excluded from subsequent orchestration.

2. The method for managing large models on a MaaS platform based on an intelligent orchestration engine according to claim 1, characterized in that, The multidimensional scoring in step S4 includes: Evaluation dimensions include resource cost, inference latency, deployment reliability, and elastic scaling capability; The scenario-weight mapping table pre-sets the correspondence between various scenario types and weight configurations. Cost-sensitive scenarios correspond to the highest weight in the resource cost dimension, latency-sensitive scenarios correspond to the highest weight in the inference latency dimension, and high-availability scenarios correspond to the highest weight in the deployment reliability dimension. For each candidate solution, a score is calculated for each dimension, and the scores are then weighted and sorted according to the weight configuration before being output.

3. The method for managing large models on a MaaS platform based on an intelligent orchestration engine according to claim 1, characterized in that, In step S2, the constraints of the inference framework components include: When the model uses RoPE positional encoding and requires sequence length extrapolation, exclude frames that do not support dynamic length expansion; When the number of model parameters exceeds a preset threshold, only frameworks that support distributed parallelism will be retained. When the scene is labeled as a low-latency API, prioritize frameworks that support Tensor Core acceleration. When the scenario is labeled as edge deployment, lightweight frameworks that support model quantization should be selected first.

4. The method for managing large models on a MaaS platform based on an intelligent orchestration engine according to claim 1, characterized in that, In step S2, the matching of interface protocol components includes: The protocol type is determined based on the expected request frequency, latency requirements, whether streaming is required, and the type of deployment environment. High-frequency, low-load scenarios are suited to the gRPC protocol; streaming scenarios are suited to the WebSocket protocol; edge IoT scenarios are suited to the MQTT protocol; and general scenarios are suited to the RESTful HTTP protocol.

5. The method for managing large models on a MaaS platform based on an intelligent orchestration engine according to claim 1, characterized in that, In step S5, the pre-release checks include: Query the available resources of the target cluster and verify that they meet the deployment requirements; Check if the resource quota of the target namespace is sufficient; Check for conflicts between the service name and port; If any check fails, the release process will be terminated and an error message will be output.

6. The method for managing large models on a MaaS platform based on an intelligent orchestration engine according to claim 1, characterized in that, In step S5, the container image building includes: Select the base image based on the inference framework type and the target hardware platform; The mirror hierarchy is designed from low to high change frequency, in the following order: system dependency layer, framework dependency layer, configuration file layer, and model file layer. Treating the model file as an independent layer allows the model layer cache to be reused when the framework or configuration changes.

7. The method for managing large models on a MaaS platform based on an intelligent orchestration engine according to claim 1, characterized in that, It also includes synchronizing the deployed model service information to the inference gateway, including service routing address, load balancing strategy, authentication method and call quota configuration; It also includes feedback optimization: Collect deployment results and runtime performance metrics; Update the historical success rate of the component combination based on the deployment results; When a component combination fails to deploy multiple times consecutively, it is added to the hard constraint rule library; After outputting the optimal deployment plan, users can adjust component selection or modify parameters through a visual interface, and the system will re-execute constraint checks and scoring sorting in real time.

8. A MaaS platform large model management system based on an intelligent orchestration engine, applicable to the MaaS platform large model management method based on an intelligent orchestration engine as described in any one of claims 1-7, characterized in that, include: The metadata parsing engine is used to perform format validation on uploaded large model files and extract the model's architectural features, computational features, and hardware dependency information. The component library stores configuration templates for hardware adaptation components, inference framework components, interface protocol components, and model publishing components. Each component has parameter ranges and constraints. The intelligent orchestration engine performs constraint checks based on model metadata and component constraints, eliminates incompatible component combinations, obtains weight configurations based on application scenario tags, scores and sorts candidate solutions, and outputs the optimal deployment solution. The intelligent orchestration engine includes: The constraint checking unit is used to construct component constraint diagrams and perform checks on hard and soft constraints. The scene recognition unit is used to obtain the weight configuration from the scene-weight mapping table based on the application scene label; The scoring and ranking unit is used to score candidate solutions in multiple dimensions and rank them according to their comprehensive scores. The interaction unit is used to receive user adjustment instructions and trigger recalculation; The automated deployment module includes: A pre-check engine is used to verify cluster resource availability and quota limits; The configuration generator maps deployment scheme parameters to Kubernetes configuration items and generates Deployment and Service configuration manifests. Image builder, used to build container images according to a layered strategy and push them to an image repository; It also includes a feedback optimization module, used to collect deployment results, update the historical success rate of component combinations, and add continuously failing component combinations to the constraint rule base; the feedback optimization module includes: The metrics collection unit is used to collect deployment results and runtime performance metrics, including deployment success or failure status, actual inference latency, and actual resource usage. The success rate update unit is used to update the historical success rate of the component combination based on the deployment results, and store the success rate data in the component library for subsequent orchestration. The rule base update unit is used to monitor the number of consecutive failures of a component combination. When the number of consecutive failures exceeds a preset threshold, the component combination is added to the hard constraint rule base. The automated deployment module is used to perform resource pre-checks, build container images, and generate Kubernetes configuration manifests based on the optimal deployment plan, thereby completing the deployment of the model service. The automated deployment module includes: The container orchestration unit is used to create Deployment, Service, and PersistentVolumeClaim resources in a Kubernetes cluster based on the configuration manifest, and to configure GPU resource limits, memory limits, storage volume mounting, and load balancing. The elastic scaling unit is used to configure HorizontalPodAutoscaler, setting the threshold for triggering scaling based on CPU utilization and the maximum and minimum number of replicas; The gateway synchronization unit is used to synchronize the deployed model service information to the inference gateway, including service routing address, load balancing strategy, authentication method and call quota configuration.

Images