Multi-objective-based large language model arrangement method and device, equipment and medium

Abstract

The application relates to the technical field of artificial intelligence, and discloses a large language model arrangement method and device based on multiple targets, equipment and a medium, which comprises the following steps: performing complexity analysis on a large language model prompt request, generating a complexity level or a complexity probability distribution; calculating a multi-target arrangement score for candidate service instances in a preset service matrix according to the complexity level or the complexity probability distribution; taking the candidate service instance with the highest multi-target arrangement score as a target service instance of the large language model; dynamically scheduling computing resources according to the current running state of the target service instance, executing the large language model prompt request through the computing resources, returning an inference result, performing confidence self-evaluation on the inference result, generating a confidence score, and performing secondary inference through the confidence score. The application can be applied to business scenarios such as financial technology, intelligently identifies task complexity, dynamically routes among multiple models and arranges resources, and solves the efficiency and cost problems in a self-hosting environment.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for arranging large language models based on multiple objectives. Background Technology

[0002] Against the backdrop of rapid iteration in large language model technology, many enterprises and research institutions are choosing to deploy large models locally on internal servers to meet the core demands of data privacy protection, industry compliance management, and customized business transformation. Compared to calling API services of third-party commercial models, building a self-hosted service system based on open-source models can not only break free from the technical constraints of a single vendor, but also eliminate the risk of leakage from the external transmission of internal core data, offering stronger security and autonomy. However, significant application challenges are commonly encountered in the process of implementing self-hosted large language models. On the one hand, resource consumption is a prominent issue. In the traditional static deployment mode, large models continuously consume high amounts of GPU memory resources, and even during idle business periods, redundant hardware resources cannot be released, further increasing the overall cost of power consumption and hardware maintenance. On the other hand, model adaptability has shortcomings. A single model architecture cannot adapt to diverse business scenarios. General-purpose large models have stronger understanding and generation capabilities and higher output accuracy, but suffer from long inference latency and high operating costs. Lightweight small models have fast response speeds and lower deployment and usage costs, but their output quality and processing capabilities drop significantly when faced with complex logical operations and deep inference tasks.

[0003] Current mainstream optimization methods still have significant limitations. Conventional scheduling methods mostly rely on static routing rules or basic load data to complete task allocation, failing to recognize the semantic complexity of user requests for large language model prompts and unable to achieve precise adaptation of task attributes to the capabilities of different model specifications. Meanwhile, mainstream container orchestration tools such as Kubernetes HPA only support service scaling based on basic hardware metrics such as CPU and memory, lacking the specific characteristics of large language model inference and failing to conduct collaborative optimization around multiple core objectives such as inference accuracy, response latency, operational costs, and service cold start efficiency. Overall, existing technical solutions are ill-suited to the operational characteristics of self-hosted large models, unable to dynamically allocate hardware resources across multi-model clusters, and struggle to balance service quality and operational costs to form an efficient and balanced operating system.

[0004] In fintech business scenarios, the implementation of self-hosted large language models has exposed several prominent technical problems. Financial business data is highly sensitive; transaction information, user asset data, and credit information cannot be shared externally. Enterprises must adopt a local self-hosted deployment model, but traditional static deployment methods consume GPU memory resources for extended periods. The 24 / 7 operation of financial systems further exacerbates hardware energy consumption and maintenance costs. Financial business needs vary greatly, ranging from lightweight, high-frequency tasks like simple inquiries and bill checks to deep, complex tasks such as risk assessment, credit review, and intricate risk control logic deduction. A single model cannot balance response speed and inference accuracy. Existing scheduling strategies lack semantic recognition capabilities and cannot match models to the complexity of financial tasks, leading to inaccurate judgments by small models in risk control tasks and redundant resource waste by large models in basic business tasks. Furthermore, general container scaling mechanisms do not meet the core requirements of low latency, high stability, and high accuracy in financial scenarios, failing to balance inference timeliness, service costs, and business compliance, and making it difficult to ensure the stable and consistent allocation of resources across multiple financial model clusters and the quality of business services.

[0005] The implementation of self-hosted large language models in the healthcare field also faces significant technical challenges. Medical data includes confidential information such as patient medical records, examination reports, and private treatment information. Compliance requirements mandate localized data processing, pushing institutions to build their own LLM inference services. However, static deployment models constantly occupy hardware resources, and the multi-department, all-day service needs of healthcare institutions exacerbate the waste of resources during idle periods, increasing the cost of IT infrastructure development. Healthcare services are diverse, with simple tasks like registration and consultation, and health education coexisting with high-logic tasks such as comprehensive case analysis, multi-disease joint diagnosis, and in-depth medical literature review. The performance limitations of both large and small models cannot meet the diverse diagnostic and treatment needs. Existing task scheduling systems lack semantic complexity awareness, failing to accurately classify medical tasks and match them to corresponding models, easily leading to inaccurate diagnostic and treatment suggestions and slow consultation responses. Conventional container orchestration tools rely solely on basic hardware metrics to regulate resources, failing to adapt to the accuracy requirements of medical inference, emergency response latency limitations, and cold start requirements. They cannot achieve dynamic optimization and allocation of resources across multiple models, hindering the efficient and stable operation of intelligent medical assistance services. Summary of the Invention

[0006] The main objective of this invention is to provide a method, apparatus, device, and storage medium for orchestrating large language models based on multiple objectives, aiming to solve the technical problems of resource waste or insufficient model processing capacity caused by existing technologies when facing user requests of varying complexity.

[0007] To achieve the above objectives, this invention provides a multi-objective large language model orchestration method, comprising: Receive large language model prompts from user input; The complexity of the large language model prompt request is analyzed to generate a complexity level or complexity probability distribution; Calculate the multi-objective orchestration score of candidate service instances in the preset service matrix based on the complexity level or complexity probability distribution; The candidate service instance with the highest multi-objective orchestration score is used as the target service instance of the large language model. Based on the current running status of the target service instance, computing resources are dynamically scheduled to execute the large language model prompt request and return the inference result. The reasoning results are self-assessed to generate a confidence score. When the confidence score is lower than the preset confidence threshold, a secondary inference request is automatically triggered and the secondary inference request is routed to the backup service instance. The results of the secondary inference are compared with the original inference results, and the complexity probability distribution of the large language model prompt request is updated based on the comparison results.

[0008] Furthermore, to achieve the above objectives, the present invention provides a multi-objective large language model orchestration device, comprising: The API gateway module is used to receive large language model suggestion requests from users. The routing module is used to perform complexity analysis on the large language model prompt request and generate a complexity level or complexity probability distribution. The multi-objective scoring module is used to calculate the multi-objective orchestration score of candidate service instances in the preset service matrix based on the complexity level or complexity probability distribution. The dynamic orchestration module is used to select the candidate service instance with the highest multi-objective orchestration score as the target service instance of the large language model. The inference backend module is used to dynamically schedule computing resources based on the current running status of the target service instance, execute the large language model prompt request through computing resources, and return the inference result; The confidence module is used to perform a confidence self-evaluation on the inference results and generate a confidence score; The secondary inference module is used to automatically trigger a secondary inference request when the confidence score is lower than a preset confidence threshold, and to route the secondary inference request to a backup service instance. The complexity update module is used to compare the secondary inference result with the original inference result and update the complexity probability distribution of the large language model prompt request based on the comparison result.

[0009] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a multi-objective large language model orchestration program stored in the memory and executable on the processor, wherein when the multi-objective large language model orchestration program is executed by the processor, it implements the steps of the multi-objective large language model orchestration method as described above.

[0010] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a multi-objective large language model orchestration program, wherein when the multi-objective large language model orchestration program is executed by a processor, it implements the steps of the multi-objective large language model orchestration method described above.

[0011] Beneficial Effects: This invention relates to the field of artificial intelligence technology and discloses a method, apparatus, device, and medium for orchestrating a large language model based on multiple objectives. The method includes: receiving a large language model prompt request input by a user; performing complexity analysis on the large language model prompt request to generate a complexity level or complexity probability distribution; calculating a multi-objective orchestration score for candidate service instances in a preset service matrix based on the complexity level or complexity probability distribution; selecting the candidate service instance with the highest multi-objective orchestration score as the target service instance of the large language model; dynamically scheduling computing resources based on the current running state of the target service instance, executing the large language model prompt request using the computing resources, and returning an inference result; performing a confidence self-evaluation on the inference result to generate a confidence score; automatically triggering a secondary inference request when the confidence score is lower than a preset confidence threshold, and routing the secondary inference request to a backup service instance; comparing the secondary inference result with the original inference result, and updating the complexity probability distribution of the large language model prompt request based on the comparison result. This invention can be applied to business scenarios such as fintech, and solves efficiency and cost problems in self-hosted environments by intelligently identifying task complexity, dynamically routing and orchestrating resources among multiple models. Attached Figure Description

[0012] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for a multi-objective large language model arrangement method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the multi-objective large language model arrangement method of the present invention; Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the multi-objective large language model arrangement device of the present invention; Figure 4This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0013] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0014] The multi-objective large language model arrangement method provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can receive large language model prompt requests from users through the client; perform complexity analysis on the large language model prompt requests to generate complexity levels or complexity probability distributions; calculate the multi-objective orchestration scores of candidate service instances in a preset service matrix based on the complexity levels or complexity probability distributions; select the candidate service instance with the highest multi-objective orchestration score as the target service instance of the large language model; dynamically schedule computing resources based on the current running status of the target service instance, execute the large language model prompt requests using computing resources, and return the inference results; perform confidence self-evaluation on the inference results to generate confidence scores; when the confidence score is lower than a preset confidence threshold, automatically trigger a secondary inference request and route the secondary inference request to a backup service instance; compare the secondary inference results with the original inference results, and update the complexity probability distribution of the large language model prompt requests based on the comparison results. This invention can be applied to business scenarios such as fintech, solving efficiency and cost issues in self-hosted environments by intelligently identifying task complexity, dynamically routing and orchestrating resources among multiple models. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0015] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the multi-objective large language model orchestration method provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0016] like Figure 2 As shown, the multi-objective large language model arrangement method proposed in this invention includes the following steps: S100: Receive user input for a large language model prompt request; S200. Perform complexity analysis on the large language model prompt request and generate a complexity level or complexity probability distribution; S300. Calculate the multi-objective orchestration score of candidate service instances in the preset service matrix based on the complexity level or complexity probability distribution. S400: The candidate service instance with the highest multi-objective orchestration score is taken as the target service instance of the large language model; S500. Based on the current running status of the target service instance, dynamically schedule computing resources, execute the large language model prompt request through computing resources, and return the inference result; S600. Perform a confidence self-evaluation on the inference result and generate a confidence score; S700. When the confidence score is lower than the preset confidence threshold, a secondary inference request is automatically triggered and the secondary inference request is routed to the backup service instance. S800. Compare the secondary reasoning result with the original reasoning result, and update the complexity probability distribution of the large language model prompt request based on the comparison result.

[0017] In this embodiment, the user's input of a large language model prompt request is first received by the API gateway (Application Programming Interface). As the unified entry point of the system, the gateway is responsible for receiving requests from users in the form of HTTP (Hypertext Transfer Protocol) or gRPC (gRPC Remote Procedure Calls), forwarding the prompt request to the downstream processing components, and finally returning the inference result to the user.

[0018] Subsequently, the routing module performs complexity analysis on the prompt requests. This process employs a hybrid routing strategy, combining a rule engine and a lightweight semantic classifier. The rule engine makes an initial determination by detecting whether the prompt request contains preset indicative keywords. For example, if the request contains keywords indicating low complexity, it is matched to the low complexity level; if it contains keywords indicating high complexity, it is matched to the high complexity level. Simultaneously, the system utilizes a pre-trained lightweight semantic classifier based on the Transformer architecture, such as DistilBERT (Distilled BERT, a lightweight pre-trained language model that undergoes knowledge distillation), to perform deep encoding and semantic classification on the prompt requests, outputting probability distribution vectors indicating whether the request belongs to low, medium, or high complexity levels. Finally, the system can determine the complexity level of the prompt request based on the rule matching results or the category with the highest probability in the probability distribution, or output the complete complexity probability distribution for subsequent calculations.

[0019] After obtaining the complexity analysis results, the orchestrator calculates a multi-objective orchestration score for each candidate service instance in the pre-defined service matrix. The service matrix consists of multiple large language models and their corresponding inference backends. Each combination corresponds to at least one callable inference service entity running on a container replica. Backend types may include vLLM (a high-throughput large language model inference engine that uses PagedAttention technology to optimize memory management) or TGI (Text Generation Inference). The scoring process is based on a normalized multi-objective scoring function, quantifying the evaluation from three dimensions: model relevance, inference latency, and resource cost. The model relevance score is obtained by weighted summing of the complexity probability distribution vector and the capability vector of the corresponding model in the pre-defined model capability matching matrix, reflecting the model's suitability for the current task. The latency score and cost score are determined based on the average inference latency and the estimated resource consumption per unit request, respectively, collected in real-time from the service registry. All three indicators are normalized using Min-Max (minimum value - maximum value) to eliminate differences in dimensions and ensure comparability between different indicators. The weights of each dimension are calculated based on the operator's preset preference parameters. Among them, accuracy preference, delay preference and resource efficiency preference are combined by weight to form the final orchestration score.

[0020] After comparing the orchestration scores of all candidate service instances, the orchestrator determines the candidate service instance with the highest score as the target service instance of the large language model. This instance represents the model and backend combination that achieves the optimal balance between model capabilities, response speed and resource costs under the current task characteristics and system state.

[0021] Once the target service instance is identified, the orchestrator dynamically schedules computing resources based on its real-time running status. The orchestrator is deeply integrated with Kubernetes (an open-source container orchestration platform for automating the deployment, scaling, and management of containerized applications) clusters, controlling the creation, scaling up, scaling down, and destruction of Pods (the smallest deployable computing unit in Kubernetes, typically containing one or more containers) via the Kubernetes API, thus achieving full lifecycle management of model containers. Furthermore, the system maintains a preheating pool to mitigate cold start latency. Standby replicas in the preheating pool keep container processes running, preload model weights into GPU (Graphics Processing Unit) memory or RAM, and use readiness probes (a probe mechanism used to detect whether a container is ready to receive traffic) to prevent it from being added to the Service's Endpoints (a list of server endpoints that determines which backend replicas can receive external requests), thereby maintaining rapid activation capabilities even when not providing inference services externally. When a request needs to be executed, if the target instance is in the preheating pool, the orchestrator activates the standby replica to start receiving and processing requests; if the target instance is already active, the active replica is directly scheduled to execute the inference task, and the inference result is returned to the user via the API gateway after the calculation is completed.

[0022] After performing a large language model inference task and returning the result, the system performs a self-assessment of the confidence level of the generated inference result, thus generating a quantified confidence score. This assessment can be done in several ways: for example, by utilizing the word-level probability distribution output by the model during the decoding phase (such as average log-likelihood, minimum marginal probability), or by introducing a lightweight auxiliary confidence assessment model (such as a binary classifier based on attention entropy or hidden state stability) to score the reliability of the inference result. The score is usually normalized to between 0 and 1, with higher values ​​indicating greater confidence in the model's output. When the calculated confidence score is lower than a pre-set threshold (e.g., 0.6), the system determines that the current result may contain errors, illusions, or logical inconsistencies, and automatically triggers a secondary inference request. This secondary inference is not a simple repetition of the same process, but rather routes the request to a "backup service instance"—this instance is usually configured with stronger capabilities, such as a larger number of parameters (upgrading from 7 billion parameters to 13 billion or even 70 billion parameters), higher quantization precision (upgrading from INT8 to FP16 or BF16), or a larger context window. Routing decisions can be made by querying the service registry through the orchestrator, selecting a currently available and higher-performing service instance from the pre-built "upgrade paths," and re-executing the inference with the same input prompt. After obtaining the secondary inference result, the system compares the original result with the secondary result. Comparison methods can include: semantic similarity calculation (e.g., using SBERT or a cross-encoder to evaluate the vector cosine distance), key information extraction and comparison (e.g., named entity consistency, logical relationship matching), or rule-based difference quantification for open-ended generation tasks (e.g., edit distance weighting, answer contradiction detection). The comparison results—such as difference scores, whether key errors have been corrected, or the secondary result having significantly higher confidence—are fed back to the complexity analysis module to dynamically update the complexity probability distribution of the prompt request (or similar requests). Specifically, if the secondary result is significantly better than the original result and the original confidence is low, the system increases the probability weight of high complexity, making it easier for future requests with similar semantics to be directly scheduled to high-performance instances; conversely, if the two results are not significantly different and the original confidence is not low, the complexity level is maintained or reduced to avoid excessive resource waste. This closed-loop feedback mechanism enables the orchestration method to continuously learn the true difficulty of different requests and gradually optimize the accuracy of the initial complexity assessment, thereby achieving a more refined dynamic balance between inference quality and cost.

[0023] This embodiment first performs complexity analysis on the user prompt request, generating a level or probability distribution; then, based on the complexity results, it calculates a multi-objective orchestration score for candidate instances in a pre-set service matrix, selecting the optimal instance as the target service; finally, it dynamically schedules computing resources to perform inference based on the target instance's running status and returns the result. Its effect is to achieve precise matching between requests and service capabilities, improve resource utilization and inference efficiency through multi-objective optimization, reduce latency, and ensure service quality.

[0024] For example, in the financial health business, user-inputted large language model suggestion requests may involve scenarios such as credit assessment, risk prediction, investment advice, or anti-fraud detection. The system first performs a complexity analysis on the suggestion request, distinguishing between simple account queries and complex multi-dimensional portfolio optimization problems, and generating corresponding complexity levels. Then, based on the complexity level, it calculates a multi-objective orchestration score for candidate service instances in a pre-built service matrix, comprehensively considering inference accuracy, response latency, and compliance requirements. The instance with the highest score is selected as the target service instance. The system dynamically schedules GPU or TPU computing resources based on its real-time load to ensure that risk assessment results or personalized financial plans can be returned quickly even during high-concurrency trading periods, thereby improving the service efficiency and customer trust of financial institutions.

[0025] In the healthcare sector, user requests may encompass tasks such as symptom analysis, medical image interpretation, drug interaction queries, or genomic data processing. The system uses complexity analysis to identify the clinical urgency and computational intensity of requests, distinguishing between routine medication consultations and complex case diagnoses requiring multimodal fusion, and outputting a complexity probability distribution. Based on this distribution, the system calculates multi-objective orchestration scores for candidate service instances, prioritizing a balance between diagnostic accuracy, data privacy protection, and response timeliness. Once the optimal instance is determined, the system dynamically schedules edge computing nodes or high-performance cloud cluster resources to ensure millisecond-level return of auxiliary diagnostic results in emergency scenarios or efficient completion of large-scale genome alignments in research scenarios, ultimately empowering precision medicine and clinical decision support.

[0026] In one embodiment, S100 includes: S101. Receive user-inputted text large language model prompt requests or multimodal large language model prompt requests through the API gateway module; S102. Preprocess the text-based large language model prompt request or the multimodal large language model prompt request; S103. Verify the preprocessed text large language model prompt request or multimodal large language model prompt request, and forward the verified text large language model prompt request or multimodal large language model prompt request to the routing module.

[0027] In this embodiment, user-inputted text-based large language model (MLM) suggestion requests or multimodal large language model (MLM) suggestion requests are first received through the API gateway module. This module serves as the unified entry point for the system, responsible for receiving HTTP or gRPC requests from users. Text-based MLM suggestion requests refer to plain text suggestions input by the user, while multimodal MLM suggestion requests include modal information other than text, such as images, audio, or video, used to trigger the multimodal inference capabilities of the large language model. After receiving the request, the API gateway module performs the basic functions of protocol parsing and request routing, ensuring that different types of requests are correctly identified and incorporated into subsequent processing flows.

[0028] After a request is received, the system performs preprocessing operations on text-based or multimodal language model prompt requests. The preprocessing stage mainly includes two aspects: format standardization and content parsing. Format standardization aims to convert requests sent by different protocols or clients into a unified data structure within the system, facilitating subsequent module processing. Content parsing involves decoding the request body to extract the core prompt text or multimodal content. It may also include necessary encoding conversions, character set normalization, and extraction of additional metadata information, such as request source identifiers, timestamps, or user session information, providing basic data support for subsequent routing decisions and resource scheduling.

[0029] After preprocessing, the system performs a validation process on the preprocessed text-based large language model hint requests or multimodal large language model hint requests. The validation mechanism covers multiple dimensions, including but not limited to request format compliance checks to ensure that the request body structure conforms to the system-defined interface specifications; content integrity verification to confirm that the hint request contains necessary input fields and is not empty; and basic security screening, such as length limit validation or sensitive character filtering of input content to prevent abnormal requests from impacting the system. Only requests that pass all validation rules are considered valid requests; requests that fail validation are rejected and a corresponding error response is returned. For text-based large language model hint requests or multimodal large language model hint requests that pass validation, the API gateway module forwards them to the routing module, which further performs semantic analysis, complexity assessment, and subsequent target model hierarchy determination and multi-target scoring processes, thus entering the core orchestration stage of the system.

[0030] This embodiment receives text or multimodal LLM prompt requests from users through an API gateway module; then preprocesses the requests; finally, it validates the preprocessed requests and forwards those that pass the validation to the routing module. The effect is that by using a unified gateway access, preprocessing and filtering mechanism, it ensures the quality and security of requests entering downstream systems, improves system stability, and reduces the impact of abnormal requests on backend services.

[0031] For example, in the financial health business, the API gateway module serves as a unified entry point to receive user-inputted text-based or multimodal large language model prompts. For instance, a customer might describe their financial needs via voice and upload income verification documents, or a trader might input complex market analysis text. The system preprocesses the requests, including multimodal data parsing, format standardization, sensitive information anonymization, and financial terminology alignment, ensuring seamless integration of voice, image, and text data. The system then proceeds to the verification stage, verifying user identity and permissions, checking for regulatory compliance, identifying potential malicious injection or fraudulent intent, and performing integrity checks on the multimodal files. Requests that pass verification are forwarded to the routing module, which intelligently distributes them to downstream services such as credit assessment, investment advisory, or anti-fraud analysis based on the request type, ensuring financial data security and business continuity.

[0032] In the healthcare business domain, the API gateway module receives text-based or multimodal large language model prompts from doctor workstations, patient terminals, or medical devices. Examples include doctors uploading composite requests containing medical records, medical images, and laboratory test results, or patients verbally describing symptoms and uploading photos of affected areas. The preprocessing stage performs medical data standardization, including DICOM image format conversion, structured clinical text extraction, speech-to-text conversion, and multimodal timestamp alignment, ensuring semantic consistency across heterogeneous medical data. The verification stage strictly adheres to medical data security standards, verifying compliance with HIPAA and other regulations, confirming doctor and patient authorization, detecting ethical risks in the input content, and conducting integrity checks on image quality and key diagnostic fields. Requests that pass verification are forwarded to the routing module, which distributes them to subsystems such as auxiliary diagnosis, medication recommendation, or image analysis based on clinical urgency and specialty, supporting the efficient operation of precision medicine and hierarchical medical systems.

[0033] In one embodiment, S200 includes: S201. Perform intent parsing on the large language model prompt request, and extract the expected output length, required external knowledge domain, inference chain depth, and number of tool calls; S202, The routing module identifies whether the large language model prompt request contains preset indicative keywords; S203. If the large language model suggestion request contains low-complexity keywords, then the large language model suggestion request is determined to be of low complexity level. S204. If the large language model suggestion request contains medium-complexity keywords, then the large language model suggestion request is determined to be of medium complexity level. S205. If the large language model suggestion request contains high-complexity keywords, then the large language model suggestion request is determined to be of a high complexity level. S206. If the large language model prompt request does not contain preset indicative keywords, then the large language model prompt request is encoded by a semantic classifier, and the complexity probability distribution is output. S207. The expected output length, required external knowledge domain, inference chain depth, and number of tool calls are used as auxiliary features and input into the complexity analysis or semantic classifier to correct the complexity level or complexity probability distribution.

[0034] In this embodiment, after receiving a prompt request from the large language model, before the complexity analysis step, an intent parsing module is first introduced to extract structured auxiliary features from the original request, thereby refining the initial complexity assessment. Intent parsing can employ a lightweight natural language understanding model (such as an intent classifier fine-tuned based on BERT) or a rule-based parser. The system identifies quantity indicators in the request (such as "summarize in 50 words or less" or "write a 1000-word article") or, based on the question type (such as "list three reasons" typically corresponding to short output, "detailed background" corresponding to long output), and combines this with a trained length regression model to predict the most likely output length range of the request (e.g., three discrete levels of short, medium, and long, or a specific word count range). The system analyzes the entities and terms in the request, matches them with knowledge base indexes (such as medical, legal, financial, and technological), and may output a probability distribution for a single or multiple domains. For example, "explain the pathogenesis of myocardial infarction" would trigger a medical domain tag. For problems requiring multi-step logical reasoning (such as math word problems, causal analysis, code debugging, etc.), the system uses dependency parsing and a reasoning pattern library (such as structures like "if...then...therefore...") to estimate the number of logical hops required to complete a correct answer. A regression model can be trained to output the number of hops (e.g., 1-5 hops) and mapped to three levels of reasoning: shallow, medium, and deep. When a request contains keywords indicating the need to access external tools (such as "calculate," "search," "query database," "execute code," "call API"), or when the request semantically clearly requires online retrieval, mathematical calculation, or file operations, the system statistically analyzes the types of different tools required and the estimated number of calls. For example, "find today's exchange rate and then calculate the amount after conversion to Euros" involves search tools and calculation tools, totaling 2.

[0035] Upon receiving a validated large language model prompt request, the routing module first activates the rule engine to perform keyword detection on the request, determining whether it contains preset indicative keywords. This rule engine performs pattern matching based on a pre-configured keyword library. The keyword library maintains corresponding vocabulary sets for different complexity levels. Low-complexity keywords typically point to information retrieval, simple enumeration, or fact-finding tasks, such as expressions containing "list," "query," or "what." Medium-complexity keywords involve certain analysis, comparison, or reasoning requirements, such as expressions containing "analysis," "comparison," or "evaluation." High-complexity keywords correspond to deep reasoning, complex derivation, or multi-step problem-solving tasks, such as expressions containing "derivation," "proof," or "comprehensive judgment." The routing module scans the prompt request text using string matching or regular expressions to identify whether any of the above-mentioned keyword categories are matched.

[0036] If the detection results show that the large language model's prompt request contains low-complexity keywords, the routing module directly classifies the prompt request as low-complexity. Low-complexity tasks typically have clear answer boundaries, require less inference depth from the model, and are suitable for processing by model instances with small parameter counts, fast response times, and low operating costs, such as lightweight models with fewer than 10 bytes of parameters.

[0037] If the detection results show that the large language model's prompt request contains medium-complexity keywords, the routing module classifies the prompt request as medium-complexity. Medium-complexity tasks require the model to have a certain level of logical analysis and information integration capabilities, but have not yet reached the level of deep reasoning. They are suitable for processing by medium-sized model instances, such as models with a parameter count between 10B and 70B, achieving a balance between accuracy and response efficiency.

[0038] If the detection results show that the large language model's prompt request contains highly complex keywords, the routing module will classify the prompt request as high complexity. High complexity tasks require the model to perform complex logical reasoning, multi-step derivation, or in-depth analysis of a specialized domain. These tasks place the highest demands on the model's understanding and generation capabilities and are best handled by large-scale model instances, such as high-performance models with more than 70 bytes of parameters, to ensure that the output quality meets the task requirements.

[0039] If the routing module confirms that the large language model's prompt request does not contain any preset indicative keywords, meaning the rule engine failed to obtain a valid basis for complexity determination through keyword matching, the system switches to the semantic model-based routing branch. At this point, the routing module calls a pre-trained lightweight semantic classifier to encode the prompt request. This classifier uses a distillation model based on the Transformer architecture, such as DistilBERT (Distilled BERT, a lightweight pre-trained language model that undergoes knowledge distillation), and performs deep semantic feature extraction and contextual understanding of the input text through a self-attention mechanism. After encoding, the semantic classifier outputs the probability distribution of the prompt request belonging to low, medium, and high complexity categories, usually presented as probability vectors, such as probability values ​​corresponding to the low, medium, and high levels. This probability distribution reflects the model's comprehensive judgment of the semantic complexity of the prompt request. It can be used to select the category with the highest probability as the determined complexity level, or it can be directly passed as a complete probability distribution to the subsequent multi-objective scoring function for weighted summation with the model capability matching matrix to calculate the model relevance score of each candidate service instance.

[0040] After extracting the aforementioned auxiliary features, the system fuses and corrects them with the original complexity level or probability distribution. If the complexity analysis uses discrete levels (low / medium / high), the correction logic can employ a decision strategy: when the expected output length is "long," the inference chain depth is "deep," or the number of tool calls is ≥2, the complexity level is directly increased by one level; if the required external knowledge domain is highly time-sensitive or specialized (such as "latest policies" or "nuclear reactor design"), the complexity level is also tended to be increased. If the complexity analysis uses a probability distribution (e.g., low: 0.2, medium: 0.5, high: 0.3), a feature weighting mapping function is designed to encode each auxiliary feature into a bias vector, and the original probability distribution is additively adjusted or updated using Bayesian methods. For example, a lightweight neural network receives the embedding vectors of four features, outputs the corrected weights for three complexity categories, and then multiplies them point-by-point with the original probability distribution and normalizes them. Through this intent parsing and auxiliary feature injection, the system can distinguish between requests that appear simple in wording but actually require a large amount of external knowledge or complex logic (such as "What is the ultimate fate of the universe?"), avoiding misjudging them as low-complexity requests and causing them to be scheduled to undercapacitated service instances. Conversely, for requests with extremely short output requirements and no reasoning needed (such as "Say hello"), even if they contain some technical terms, the complexity level can be reduced, saving computing resources. The entire correction process is completed within milliseconds, ensuring real-time performance.

[0041] In this embodiment, the routing module detects whether the request contains preset indicative keywords. If it contains low, medium, or high complexity keywords, the corresponding level is directly determined. If it does not contain keywords, the request is encoded using a semantic classifier, and a complexity probability distribution is output. The effect is that by combining rule matching and semantic understanding, it can quickly identify typical scenarios and perform refined probability assessments on ambiguous requests, improving the accuracy and coverage of complexity judgment and providing a reliable basis for subsequent service orchestration.

[0042] For example, in the financial health business domain, the routing module first detects whether the request prompted by the large language model contains preset indicative keywords. If the user enters low-complexity keywords such as "check balance," "recent transactions," or "interest rate," the system marks the request as low complexity and directly assigns it to a lightweight service instance for a quick response. If it contains medium-complexity keywords such as "portfolio optimization," "multi-factor risk model," or "cross-market arbitrage strategy," it is marked as medium complexity and routed to a service instance with medium computing power. If it involves high-complexity keywords such as "stress test scenario simulation," "Monte Carlo VaR calculation," or "complex derivative pricing," it is marked as high complexity and scheduled to a high-performance computing cluster for processing. When the user enters natural language such as "help me check this stock," which does not contain preset keywords, the semantic classifier performs deep encoding on the request, analyzes features such as the number of asset classes involved, computational dimensions, and data timeliness, outputs a complexity probability distribution, and dynamically matches the optimal service resources to ensure efficient coverage of all scenarios, from simple account queries to complex quantitative analysis.

[0043] In the healthcare business domain, the routing module performs keyword detection on prompt requests to achieve complexity stratification. If a patient enters low-complexity keywords such as "what medicine to take for fever," "outpatient hours," or "drug side effects," the system determines it to be of low complexity level, and a lightweight service instance immediately returns health education or referral information. If a doctor enters medium-complexity keywords such as "lung cancer staging assessment," "diabetes complication screening," or "drug interaction analysis," it is marked as medium complexity level and routed to a service instance with medical knowledge graph reasoning capabilities. If high-complexity keywords are involved, such as "multi-omics data integration analysis," "rare disease differential diagnosis," or "personalized chemotherapy regimen design," it is marked as high complexity level and scheduled to a high-performance node equipped with a large-scale medical pre-trained model. When inputting vague descriptions without preset keywords, such as "this patient's condition is rather special," the semantic classifier jointly encodes clinical text, imaging reports, and test data, comprehensively assesses the complexity of the condition, the need for multidisciplinary collaboration, and the level of evidence-based medicine, outputs a complexity probability distribution, and intelligently matches differentiated computing resources from primary care consultations to complex cases at tertiary hospitals, ensuring a balance between medical quality and response efficiency.

[0044] In one embodiment, S300 includes: S301. The orchestrator retrieves the service matrix and candidate service instances of the corresponding large language model from the service registry based on the complexity level or complexity probability distribution. S302. Perform a weighted dot product between the complexity level or complexity probability distribution and the capability vector of the service matrix to generate a model relevance score, and normalize the model relevance score. S303. Obtain the real-time inference latency and unit request resource cost of the candidate service instance from the service registry; S304. Normalize the real-time inference latency and unit request resource cost respectively to generate latency score and cost score; S305. Obtain the model parameter count, quantization accuracy, context window length, and hardware accelerator type for each candidate service instance; S306. Based on the complexity level or complexity probability distribution, perform a weighted score on the number of model parameters, quantization accuracy, context window length, and hardware accelerator type to generate an execution adaptability score. S307. Based on the preset preference weights, the normalized model relevance score, execution adaptability score, latency score, and cost score are weighted and summed to generate a multi-objective orchestration score for candidate service instances.

[0045] In this embodiment, during the routing analysis phase, the system first receives the user's prompt request through the API gateway. This gateway serves as the system's unified entry point, responsible for receiving HTTP or gRPC requests. Subsequently, the hybrid classifier in the routing module performs complexity analysis on the prompt. This analysis can employ two parallel methods: one is keyword-based rule matching, which detects whether the prompt request contains preset indicative keywords. If it contains keywords indicating low complexity, it is matched to the low complexity level; if it contains keywords indicating high complexity, it is matched to the high complexity level. The other is semantic model-based routing, which uses a pre-trained lightweight semantic classifier to encode and classify the prompt request. This classifier is the DistilBERT (Distilled BERT) model based on the Transformer architecture, capable of outputting the probability distribution of each complexity category. The routing module ultimately outputs the probability distribution of the prompt request belonging to low, medium, and high complexity levels, denoted as a vector. .

[0046] Next, the service matrix matching phase begins. The orchestrator retrieves the service matrix and candidate service instances for the corresponding large language model from the Service Registry based on complexity level or complexity probability distribution. The service matrix consists of combinations of multiple large language models and multiple inference backends, with each combination corresponding to at least one service instance. The Service Registry stores metadata for all available models and their corresponding backends (such as vLLM, a high-throughput large language model inference engine that uses PagedAttention technology to optimize memory management; or TGI, Text Generation Inference, an inference service framework for deploying large language models), including real-time load, average latency, and runtime cost. The system pre-configured a configurable complexity-model capability matching matrix M, with matrix elements... Representation Model Historical performance evaluation scores when handling tasks of complexity level c. Model relevance score R(p, The probability distribution vector is obtained by performing a weighted dot product with the corresponding model capability vector, i.e.:

[0047] .

[0048] Finally, for all candidate model sets Min-Max normalization is performed to obtain the correlation score of the normalized model. Its value range is [0, 1].

[0049] During the real-time metric acquisition and normalization phase, the service registry periodically collects and reports metrics such as response latency and resource consumption (e.g., GPU memory consumption per second) for inference requests through a sidecar deployed on each inference instance. For each candidate service instance... The system obtains its current average inference latency. Request cost estimates from the unit For the set of all current candidate service instances The delay T and cost C are both Min-Max normalized, and a transformation method is used where lower values ​​result in higher scores. The formula for calculating the delay score is as follows: The formula for calculating the cost fraction is: , where ε is a very small positive number (e.g. This is used to prevent the denominator from being zero when all candidate service instances have the same latency or cost. The normalized score is set to 1 by default when the denominator is zero.

[0050] Finally, the weighted summation stage begins. Based on the operator's preset preference parameters (α, λ, μ), the weights of the three objectives are calculated: Where α represents accuracy preference, λ represents delay preference, μ represents resource efficiency preference, and w satisfies R + w T + w C = 1. Finally, the scores are arranged. Calculated by the following formula: The orchestrator selects the service instance with the highest score as the execution target and dynamically schedules computing resources to execute the prompt request based on the current state of the target instance. Dynamic scheduling includes automatic scaling based on concurrent request volume and average latency, using Little's Law (the relationship L=λW used in queuing theory to estimate concurrent capacity, where L is the average number of requests in the system, λ is the request arrival rate, and W is the average response time) for automatic scaling. The target replica number is equal to the rounded-up (request rate × average latency / concurrent processing capacity threshold). In addition, the system maintains a preheating pool to mitigate cold start latency under the scale-to-zero (reducing the number of running replicas of a model service to 0 when idle) strategy. The standby replicas in the preheating pool keep the container process running and complete the inference engine initialization, preload model weights into memory, and control the activation status through readiness probes (a mechanism in Kubernetes used to detect whether a container is ready to receive traffic).

[0051] Furthermore, in calculating the multi-objective orchestration score, an execution suitability scoring mechanism is introduced to more accurately match the complexity of the request with the actual execution capabilities of the candidate service instances. The orchestrator first extracts key attributes for each candidate service instance from the service registry, including the number of model parameters (e.g., 7 billion, 13 billion, 70 billion), quantization precision (e.g., FP16, BF16, INT8, INT4), context window length (e.g., 4096, 8192, 128k tokens), and hardware accelerator type. These attributes are typically reported and stored by the container platform or resource manager during service instance registration. The orchestrator weights these four attributes based on the previously generated complexity level or complexity probability distribution to obtain the execution suitability score. The weighting principle is that the more complex the request, the higher the number of parameters, the higher the precision, the longer the context window, and the higher the performance of the hardware. For example, for high complexity levels (or when high complexity has a large weight in the complexity probability distribution), the system will assign a weight of 0.4 to the model parameter count, 0.3 to the quantization precision, 0.2 to the context window length, and 0.1 to the hardware accelerator type. For low complexity levels, the weights may be reversed: parameter count weight 0.1, precision weight 0.1, context window weight 0.1, and hardware type weight 0.7 (emphasizing low-cost or low-latency hardware). If the input is a probability distribution (e.g., low: 0.1, medium: 0.3, high: 0.6), this distribution can be used as the mixing coefficient for the weights. The fitness score for each level is calculated first, and then the expectation is taken. The scoring rules for each attribute are illustrated below: Parameter quantity is mapped to a 0-1 range according to a preset threshold (0.2 for 7 billion, 0.5 for 13 billion, and 1.0 for 70 billion); quantization accuracy is scored inversely proportional to accuracy loss (FP16 / BF16=1.0, INT8=0.7, INT4=0.4); context window length is calculated based on the matching degree between the actual length and the target length (the expected window inferred from complexity); hardware accelerator type is assigned a score based on a pre-established performance benchmark table. A weighted sum is then used to obtain the execution suitability score for each candidate instance (usually also normalized to the 0-1 range). This score is then combined with the model relevance score, latency score, and cost score. The model relevance score reflects the degree of matching between the request content and the model's training domain / capabilities (e.g., calculated via vector dot product); the latency score is a normalization of real-time inference latency (lower latency, higher score); and the cost score is a normalization of the cost per unit of request resources (e.g., GPU seconds, API call fees) (lower cost, higher score). Before weighted summation, all scores are normalized using Min-Max or Z-score to ensure consistency of dimensions.Finally, the scores are weighted and summed according to preset preference weights (e.g., system-wide default weights: relevance 0.35, adaptability 0.25, latency 0.25, cost 0.15) to obtain the multi-objective orchestration score for each candidate service instance. A higher score indicates that the instance performs optimally in terms of execution capability, domain matching, response speed, and resource economy for the current request complexity. By introducing an execution adaptability score, the system can avoid mismatching high-complexity requests to instances with fewer parameters, lower precision, or smaller contexts, which could lead to output quality degradation. It can also avoid wasting low-complexity requests on high-performance instances, causing resource redundancy, thus achieving a fine balance between quality and efficiency.

[0052] In this embodiment, the orchestrator retrieves the corresponding LLM service matrix and candidate instances from the service registry based on complexity level or probability distribution. It generates a model relevance score by weighted dot product of complexity and capability vectors and normalizes it. Simultaneously, it obtains the real-time inference latency and unit resource cost of candidate instances, normalizing them into latency and cost scores, respectively. Finally, it weights and sums the three scores according to preset preference weights to generate a multi-objective orchestration score. Its effect lies in comprehensively considering model matching degree, latency, and cost, achieving a refined trade-off in service selection, and improving resource scheduling efficiency and user experience.

[0053] For example, in the financial health business domain, the orchestrator retrieves the service matrix and candidate service instances of the corresponding large language model from the service registry based on complexity level or complexity probability distribution. For simple account queries, the service matrix of a lightweight dialogue model is retrieved; for complex quantitative analysis, the service matrix of a financial-specific large model is retrieved. The system performs a weighted dot product of complexity level and model capability vector. For example, high-complexity requests are more closely matched with models that have time series analysis and risk modeling capabilities, generating a model relevance score and normalizing it. At the same time, the real-time inference latency and unit request resource cost of each candidate instance are obtained from the service registry and normalized into latency score and cost score, respectively. A weighted sum is performed according to preset preference weights: low latency is emphasized during trading hours, low cost is emphasized for nighttime batch processing, and high accuracy is emphasized for regulatory reporting. Finally, a multi-objective orchestration score is generated to select instances that balance inference quality, response speed, and operating cost, ensuring that the differentiated needs of millisecond-level response in high-frequency trading and deep inference in complex risk control models are met.

[0054] In the healthcare business domain, the orchestrator retrieves service matrices from the service registry based on complexity levels or probability distributions, such as lightweight health consultation models, specialist diagnostic models, or large multimodal fusion models. It then performs a weighted dot product of complexity metrics and model capability vectors; for example, rare disease differential diagnosis requests are more likely to match models with cross-departmental knowledge fusion and literature retrieval capabilities, generating a normalized model relevance score. Simultaneously, it acquires and normalizes the real-time inference latency and unit request resource cost of candidate instances. Based on clinical scenarios, it sets preference weights: prioritizing low latency for emergency triage, high accuracy for image-assisted diagnosis, and low cost for chronic disease management, and generates a multi-objective orchestration score through weighted summation. This mechanism ensures rapid response for lightweight consultations during peak outpatient hours and automatically schedules complex cases to high-performance clusters for multimodal in-depth analysis, optimizing computing resource allocation while ensuring medical quality and supporting the efficient operation of the hierarchical medical system.

[0055] In one embodiment, S500 includes: S501. Obtain the current running status of the target service instance; S502. If the current running status is that the instance is activated, then the large language model prompt request is routed to the target service instance to execute the inference task; S503. When the current running state is that the instance is activated but the load exceeds the preset overload threshold, select a standby replica with the same model and quantization accuracy as the target service instance from the preheating pool. S504. Activate the standby replica as a parallel service instance, and use a request-level or task-level load balancing strategy to distribute part of the computational tasks or batch requests of the large language model prompt request to the parallel service instance to execute the inference task. S505. Merge the inference results of the target service instance and the inference sub-results of the parallel service instance to generate the final inference result; S506. If the current running status is that the instance is not activated, check if there are any standby replicas in the preheating pool; S507. If the preheating pool has standby replicas, activate the standby replicas to convert them into target service instances and execute inference tasks. S508. If the preheating pool has no standby replicas, then the container platform is called through the orchestrator to create a container instance, and the container instance is used as the target service instance to execute the inference task.

[0056] In this embodiment, in the self-hosted large language model orchestration system based on multi-objective optimization, after the user's input prompt request has undergone complexity analysis in the routing stage, and the orchestrator selects the optimal target service instance from the service matrix through the normalized multi-objective scoring function, the system enters the specific task execution and resource scheduling stage.

[0057] The orchestrator leads the process of obtaining the current running status of the target service instance. The orchestrator first queries the service registry. This module periodically collects and reports metrics such as response latency and resource consumption for inference requests through sidecar containers deployed on each inference instance, thereby maintaining real-time status information for all available large language models and their inference backend combinations. The service registry records information including the current number of replicas, average inference latency, load status, and whether each service instance is in an active state capable of receiving requests. This information provides the data foundation for subsequent scheduling decisions.

[0058] If the query result indicates that the target service instance is active, it means that the instance currently has active replicas running, and these replicas have passed the readiness probe check and been successfully registered to the service's (a Service, an abstract resource used to expose network access to containerized applications) endpoint list (an object that maintains a set of backend addresses that can receive traffic), and can normally provide inference services. A successful return from the readiness probe (a health check mechanism in container orchestration platforms used to detect whether a container is ready to receive traffic) is a necessary condition for replicas to be included in traffic forwarding. In this case, the system directly routes the large language model hint request to the target service instance through the Application Programming Interface Gateway (APIGateway, the system's unified entry point component). The corresponding inference backend loads the model weights and performs the inference calculation, finally returning the generated result to the user. This is the optimal execution path because it requires no additional startup or activation operations and has the lowest latency.

[0059] When the target service instance is activated but the load exceeds a preset overload threshold, an elastic parallel inference mechanism based on a preheating pool is used to alleviate high load pressure and avoid request queuing timeouts or service quality degradation. The system first continuously collects instantaneous load metrics for each activated service instance through a monitoring agent, such as concurrent requests, GPU utilization, queue depth, or average queuing latency per request. When any metric exceeds the preset overload threshold for two consecutive sampling points (e.g., concurrent requests > 10 and GPU utilization > 85%, or queuing latency > 200ms), a traffic splitting decision is triggered. The orchestrator then performs matching filtering from the preheating pool, which maintains a set of standby replicas of loaded models in a dormant state. These replicas share the same model name, quantization precision (e.g., the INT8 version of LLaMA-2-13B), and necessary environmental dependencies as the target service instance. The system prioritizes standby replicas with hardware affinity consistent with the target instance (e.g., the same type of GPU, the same node, or the same availability zone) to ensure minimal communication overhead. If a perfectly matching replica exists, its state is switched from "standby" to "active" via a container runtime interface (such as the Docker API or Kubernetes), completing instance warm-up (typically only requiring state switching and connection pool initialization, taking milliseconds). Once active, this replica becomes a parallel service instance, forming a temporary inference cluster with the original target instance. Next, the system employs request-level or task-level load balancing strategies based on the request type. If the incoming requests are in batches (e.g., a batch of independent hint requests), request-level load balancing is used: the orchestrator maintains a load balancer that distributes newly arriving requests across parallel service instances using weighted round-robin or least-connections algorithms based on the current queue length or real-time load score of each instance, without splitting individual requests. If the incoming request is a large, internally decomposable request (e.g., a task generating extremely long sequences, distributed matrix operations, or a one-time inference of multiple sub-problems), task-level load balancing is used: the orchestrator divides the original hint request into multiple subtasks based on logical or data dimensions. For example, for a long document summarization task requiring the processing of 64k tokens, the document can be divided into four 16k blocks based on semantic paragraphs. Each block, along with the same instructions, is sent to the target service instance and three parallel instances respectively. For matrix operations in the transformer model, tensor model parallel segmentation can be performed according to the attention head dimension or sequence length dimension. During distribution, the orchestrator ensures that each subtask carries the necessary context identifiers and result merging instructions. Each parallel service instance independently performs inference, generating inference sub-results. Finally, the system needs to merge the results from different instances into the final inference result.The merging logic depends on the traffic splitting strategy: For request-level traffic splitting (multiple independent requests), the result of each instance is a complete response itself, requiring no merging; the result is simply returned to the corresponding caller by request ID. For task-level traffic splitting (a request is split into multiple sub-tasks), a result aggregator needs to be set up. The aggregator collects the sub-results returned by all parallel instances and combines them according to predefined merging rules. For example, for document segmentation, the aggregator concatenates the sub-results in paragraph order and removes duplicate transitional statements; for tensor parallel inference, the aggregator merges hidden states through all-reduce communication or element-wise weighted averaging, and then generates a token through the final output layer. Throughout the process, the merging module also needs to handle timeout and partial failure tolerance logic: if a parallel instance fails to return a result within the specified time, the system can use only the returned sub-results for downgraded merging and record the exception for subsequent instance health checks. After the merge is complete, the system returns the final inference result to the user and releases the temporarily activated parallel service instance. If the overload persists, the instance can remain active and be added to the load balancing pool. If the load returns to normal, the instance will be downgraded to a standby replica after a period of no traffic and returned to the warm-up pool for future use. This mechanism ensures service quality under high load while avoiding the long-term occupation of redundant resources.

[0060] If the target service instance is currently inactive, it indicates that the model service has no active replicas providing services externally. It may be in a scale-to-zero state (a strategy of reducing the number of running replicas to zero to completely release resources), meaning that all running replicas have been reduced to zero to release graphics processor (GPU, a hardware accelerator specifically used for parallel computing and graphics rendering) resources and reduce idle power consumption. At this time, the system will not immediately trigger a completely new container creation process, but will first check whether there are standby replicas in the warmup pool (a set of standby replicas maintained to alleviate cold start latency).

[0061] When standby replicas exist in the preheating pool, these replicas keep the container process running and have completed the initialization of the inference engine (a software component that performs forward computation of large language models and text generation, such as a high-throughput large language model inference engine vLLM that optimizes memory management using paging attention techniques, or a text generation inference framework TGI), while preloading model weights into the GPU memory or system memory. During standby, standby replicas are prevented from being registered in the service's endpoint list by configuring the ready probe to return a failure status, thus not receiving external requests, but the container itself continues to run to maintain the memory residency of the model weights. When the orchestrator decides to activate, it sends an activation signal to the standby replicas, for example, by modifying the state file within the container or calling the internal application programming interface (API, a specification for communication and data exchange between applications), causing the ready probe to return a success status. The replica is then added to the service's endpoint list and begins receiving requests. Since the model weights have been preloaded, the activation latency has been significantly reduced from minutes of cold start (the waiting time required for the system to be initialized from zero to be available for service) to seconds, after which prompt requests can be routed to the instance to perform inference tasks.

[0062] If no standby replicas are available in the preheating pool, it means that the model service has neither active replicas nor preheating standby resources. At this point, the orchestrator creates a brand new container instance (a standby application unit running in a containerized environment) by calling the application programming interface of the container orchestration platform (Kubernetes, an open-source platform for automated container deployment, scaling, and management), and uses it as the target service instance. This process requires completing all steps, including pulling the container image, starting the container, initializing the inference engine, and loading model weights from storage media to the graphics processor's memory, resulting in a relatively long cold start delay. After the container instance is created and passes the health check, the system routes the request to that instance to execute the inference task.

[0063] The orchestrator performs capacity planning based on Little's Law (a mathematical formula describing the average number of requests in a system equals the request arrival rate multiplied by the average response time; used in this system to calculate the target number of replicas) from queuing theory. It periodically calculates the target number of replicas for each model. The target number of replicas is equal to the request rate (the number of requests arriving at the target service per unit time) multiplied by the average latency (the average time taken for a single request to complete inference from arrival), divided by the concurrency threshold (the maximum number of requests a single service instance replica can handle simultaneously), and then rounded up. When the request rate drops, causing instance idle time to exceed a preset threshold, the system triggers a scaling-down operation, reducing the number of replicas to the minimum number configured in the preheating pool, or even zero. When requests arrive, service capacity is quickly restored through the aforementioned activation or creation mechanism. The minimum number of replicas in the preheating pool is configured differently based on the model level. Smaller models with fewer parameters typically have more standby replicas to handle high-frequency, simple requests, while larger models with a large number of parameters have fewer or even zero standby replicas to achieve a balance between responsiveness and resource consumption. This layered design enables the system to achieve dynamic optimization of multiple objectives, including accuracy, latency, and cost, in a self-hosted environment that ensures data privacy.

[0064] This embodiment first obtains the current running status of the target service instance. If the instance is activated, the request is directly routed to execute inference; if it is not activated, the preheating pool is checked first and a standby replica is activated; if there is no standby replica, the container platform is called through the orchestrator to create a new instance. The effect is to build a three-level elastic scheduling system of "active instance - preheated replica - cold start", which minimizes cold start latency, improves system response speed and resource elasticity, and ensures high availability and low latency experience of inference service.

[0065] For example, in the financial health business, the system continuously obtains the current running status of the target service instance to ensure business continuity. If the instance is currently active, such as a credit assessment model that is continuously online during peak trading hours, the user's large language model prompt request is directly routed to that instance to execute inference tasks, achieving millisecond-level risk assessment. If the instance is not active, such as an investment advisor model that is dormant during low-frequency periods at night, the system checks if there are any standby replicas in the preheating pool. If standby replicas exist, they are immediately activated and converted into target service instances, ensuring that customers' morning financial advice does not require a cold start. If there are no standby replicas in the preheating pool, such as a sudden market fluctuation causing a surge in quantitative analysis requests exceeding the plan, the orchestrator calls the container platform to quickly create a container instance as the target service instance, dynamically scaling up to execute inference tasks. This mechanism balances the stringent real-time requirements of financial scenarios with the need for cost optimization, ensuring zero latency during trading hours while avoiding the waste of idle resources.

[0066] In the healthcare business domain, the system monitors the operational status of target service instances in real time to support clinical timeliness. If an instance is activated, such as an outpatient auxiliary diagnostic model running continuously, patient symptom descriptions or image analysis requests are directly routed to execute inference, shortening waiting times. If an instance is not activated, such as an emergency specialist model at night in dormancy, the system checks if there are standby replicas in the preheating pool; if there are standby replicas, they are quickly activated and converted into target service instances, ensuring that emergency auxiliary diagnoses for green channels such as chest pain centers are immediately available. If there are no standby replicas in the preheating pool, such as a surge in epidemiological analysis requests due to a public health emergency, the orchestrator calls the container platform to create container instances as target service instances, elastically scaling up to execute large-scale data inference. This mechanism ensures second-level response in emergency scenarios while balancing the computing power costs of medical institutions with the needs of patient safety through the combination of the preheating pool and elastic scaling.

[0067] In one embodiment, after dynamically scheduling computing resources based on the current running state of the target service instance, executing the large language model prompt request using the computing resources, and returning the inference result, the process includes: Collect and analyze response latency, resource consumption, and load metrics generated during inference tasks. The response latency metrics, resource consumption metrics, and load metrics are transmitted to the service registry. The service registry updates the service matrix based on response latency metrics, resource consumption metrics, and load metrics; A sliding time window was used to analyze the trend of the average orchestration score of each service instance across multiple historical windows. When the orchestration score of a specific service instance is detected to be continuously decreasing and the decrease exceeds the dynamic threshold, the health probe or offline diagnostic process of the service instance is triggered. If a diagnosis confirms performance degradation or anomalies, the service instance will be temporarily removed from the service matrix, and a backup service instance will be started to take over its workload.

[0068] In this embodiment, in the self-hosted large language model orchestration system based on multi-objective optimization, index collection and service matrix update constitute a key data closed loop connecting the inference execution layer and the scheduling decision layer. Its purpose is to provide real-time and accurate decision-making basis for complexity matching in the routing stage and multi-objective scoring in the orchestration stage.

[0069] The process of collecting response latency, resource consumption, and load metrics generated during inference task execution is handled by the sidecar container deployed on each inference instance. The sidecar container is an auxiliary container deployed in the same network namespace as the main application container. It monitors the main container's running status non-intrusively by sharing process namespaces or network interfaces. After the inference backend completes an inference task for a large language model prompt request, the sidecar container immediately collects three core metrics: response latency metrics record the time span from when the request enters the inference backend to when a complete response is returned, typically measured in milliseconds, reflecting the processing speed of the current service instance; resource consumption metrics quantify the computing resources consumed in executing the inference, such as GPU memory usage time, peak memory usage, or CPU cycle consumption—these metrics are directly related to operating costs; and load metrics characterize the concurrent pressure on the current service instance, including the number of requests being processed, the length of the waiting queue, and GPU utilization, used to assess the instance's saturation level.

[0070] The process of transmitting the above three types of metrics to the service registry is implemented through a periodic reporting mechanism. The sidecar container aggregates and preprocesses the collected raw metric data locally, such as calculating the average latency, peak resource consumption, and average load within a sliding time window. Then, it sends these metrics to the service registry via a remote procedure call framework or Hypertext Transfer Protocol. The service registry is the core module in the system that centrally maintains the metadata of all available large language models and their inference backend combinations. It not only stores the static attributes of the models but also dynamically maintains the real-time status and historical performance data of each service instance. The metric transmission process is typically asynchronous to avoid performance interference with the main inference process, while reducing network overhead through techniques such as batch compression and short connection reuse.

[0071] Upon receiving response latency, resource consumption, and load metrics, the service registry updates the service matrix. The service matrix is ​​a set of selectable services comprised of combinations of multiple large language models and multiple inference backends, where each combination corresponds to at least one service instance, and each service instance has an independent performance profile within the matrix. The update process first merges the newly arrived metric data with the historical data of the corresponding service instance, using methods such as exponentially weighted moving averages or quantile statistics to smooth out instantaneous fluctuations and obtain a more representative performance benchmark. Subsequently, the service registry updates the latency and cost scores of the service instance in the service matrix. These normalized values ​​directly participate in the subsequent orchestrator's calculation of the multi-objective scoring function. For example, if the average inference latency of a service instance increases due to increased load, its normalized latency score will decrease accordingly, thus reducing its competitiveness in the scoring ranking; conversely, if an instance's resource consumption metrics show a low cost per request, its normalized cost score will increase, making it more advantageous under a cost-first strategy. In addition, the load metric is used to determine whether an instance is approaching its concurrent processing capacity threshold. When the load remains high, the service registry will send a scaling suggestion to the orchestrator, triggering capacity planning calculations based on Little's rule.

[0072] By continuously tracking response latency metrics, the system can dynamically perceive real-time performance changes of each service instance, ensuring that latency weights are accurately reflected in the multi-objective scoring function. By monitoring resource consumption metrics, the system achieves refined cost accounting for expensive hardware resources such as graphics processors, supporting precise triggering of scaling down to zero strategies. By analyzing load metrics, the system can predict resource bottlenecks and initiate expansion or preheating processes in advance, avoiding service degradation caused by sudden traffic surges. The service registry, as a data aggregation center, provides a unified data foundation for the routing module's hybrid classification strategy and the orchestrator's dynamic scheduling through its updated service matrix. This allows for quantitative comparison and joint optimization of task complexity, model capabilities, and real-time system status within the same framework.

[0073] During the dynamic maintenance of the service registry, to improve the self-healing capability and long-term stability of the orchestration system, the system maintains a sliding time window queue for each candidate service instance. The window size can be dynamically adjusted according to the request frequency of the instance, such as a fixed 5 minutes, 10 consecutive time windows (one every 30 seconds), or based on request batch count. At the end of each window, the orchestrator recalculates the instance's average multi-objective orchestration score within that window (a normalized weighted result considering multiple dimensions such as model relevance, execution adaptability, latency, and cost), and pushes this average value into the sliding window queue, while removing the data from the oldest window. The system then performs trend analysis on the historical average scores in the queue: a common approach is to calculate the slope of a linear regression; if the slope is negative and the absolute value exceeds a preset static threshold (e.g., a decrease of 0.05 per window), it is considered a continuous decline. Another more robust method is to use cumulative monitoring of month-on-month declines, i.e., the orchestration score of k consecutive windows (e.g., 3 consecutive windows) is lower than that of the previous window, and the cumulative decline exceeds a dynamic threshold. The dynamic threshold can be calculated by multiplying the standard deviation of the score change within the most recent stable period (e.g., the previous 20 windows) by a sensitivity coefficient (e.g., 2.5 times the standard deviation), or by using outlier detection with an exponentially weighted moving average, allowing the threshold to adapt to the normal fluctuation range of different service instances. Once a sustained downward trend is detected and its magnitude exceeds the dynamic threshold, the system no longer waits for the instance to completely fail, but immediately triggers a health probe or offline diagnostic process. A health probe is a lightweight, low-intrusion, rapid check, such as sending a pre-defined "heartbeat" prompt (e.g., "ping" or a fixed simple Q&A "return OK") to the service instance, measuring response time, error rate, and the correctness of the returned results. If the health probe fails consecutively or the response time increases sharply, performance degradation is confirmed. For greater accuracy, the system can also initiate an offline diagnostic process—temporarily removing the instance from the traffic path (or marking it as "diagnosing") and performing more comprehensive tests, including running a standardized set of prompts, comparing the output quality with the similarity to a baseline model, and checking for memory leaks, abnormal CPU / GPU usage, corrupted model files, network packet loss, etc. If the diagnostic results confirm an unrecoverable anomaly (such as hardware errors, software deadlocks, corrupted model weights, or persistent high latency due to frequency degradation), the orchestrator will temporarily remove the service instance from the service matrix. Specifically, it will set the instance's health status to "unavailable" or "isolated" in the service registry and notify the routing module to stop scheduling any new requests to that instance. Simultaneously, the orchestrator will select a backup service instance from the preheating pool or idle resource pool that has the same model, the same quantization accuracy, and the same or compatible hardware accelerator type as the failed instance, activate it, and connect it to the service matrix to take over the workload of the original instance.To facilitate a smooth failover, the system employs an elegant migration strategy: long-running requests sent to a failed instance but not yet completed are allowed to continue execution or time out and be retried; new requests are immediately routed to a standby instance. Through this proactive detection and automatic isolation failover mechanism based on orchestration score trends, the system can intervene early when service instances are not completely down but their performance continues to deteriorate, significantly reducing the impact on user experience while avoiding the unnecessary resource consumption and orchestration score distortion caused by continuously sending requests to degraded instances.

[0074] This embodiment collects three types of metrics during the execution of inference tasks: response latency, resource consumption, and load status, and transmits them to the service registry. The service registry then dynamically updates the service matrix based on this real-time operational data. The effect is to form a closed-loop optimization system of "execution-collection-feedback-update," enabling the service matrix to continuously reflect the true performance status of instances. This provides a more accurate basis for subsequent complexity analysis and multi-objective orchestration, achieving adaptive evolution of the system scheduling strategy.

[0075] For example, in the financial health business domain, the system continuously collects response latency, resource consumption, and load metrics generated by inference tasks. After a credit assessment model completes batch enterprise risk control analysis, the system records data such as the average response time per request, peak GPU memory usage, and concurrent request queue depth, and transmits these metrics to the service registry in real time. The service registry dynamically updates the service matrix accordingly. For instance, if an investment advisor model experiences a latency spike during quarterly earnings season, its latency score weight is automatically lowered; if an anti-fraud model experiences abnormal resource consumption during promotional periods, cost score calibration is triggered. Through this continuous feedback loop, the service matrix accurately reflects the real performance of each instance under different financial scenarios such as peak trading periods and market fluctuations, making subsequent orchestration decisions more aligned with actual business needs and ensuring optimal resource matching for the entire spectrum of requests, from simple balance inquiries to complex derivative pricing.

[0076] In the healthcare business domain, the system comprehensively collects multi-dimensional operational metrics for inference tasks. After an auxiliary diagnostic model completes CT scans for lung nodules, the system captures data such as single-case analysis time, memory usage curves, and node CPU utilization, and synchronously transmits this data to the service registry. The service registry iteratively updates the service matrix based on these metrics. For example, if a chronic disease management model experiences consistently high load during peak seasons for elderly health checkups, its load weight is increased, triggering a capacity expansion warning. If a multimodal fusion model experiences a surge in resource consumption during rare disease consultations, its cost score is adjusted to balance research input and clinical output. This feedback mechanism ensures that the service matrix accurately reflects the operational characteristics of each instance in medical scenarios such as peak outpatient hours, emergency night clinics, and research breakthroughs. This guarantees precise scheduling of differentiated computing power needs, from routine health consultations to complex multidisciplinary consultations, supporting continuous optimization of medical quality and operational efficiency.

[0077] In one embodiment, before dynamically scheduling computing resources based on the current running state of the target service instance, executing the large language model prompt request using computing resources, and returning the inference result, the process includes: Periodically collect the average request rate, average inference latency, and number of current service instance replicas of the large language model within a time window; Based on Little's rule, the target number of replicas is calculated according to the average request rate and average inference latency. Based on the number of parameters, the large language model is divided into three levels: small model, medium model, and large model. Configure the minimum number of standby copies in the warm-up pool for large language models at different levels; When the target number of replicas is greater than the current number of service instance replicas and the cooldown period has expired, activate standby replicas from the preheating pool to convert them into service instances for scaling. When the idle time of a service instance of a large language model exceeds the idle threshold, the redundant service instance will be downgraded to a standby replica in the preheating pool, and only the minimum number of standby replicas in the preheating pool will be retained for scaling down. When the idle time of the service instance of the large language model exceeds the idle time limit, the service instance and standby replicas of the large language model will be scaled down to zero. Record the historical request arrival time series of the large language model, and use the time series prediction model to predict the time window of the next active cycle; At a preheating offset time point before the active cycle arrives, the orchestrator actively calls the container platform to pre-create at least one standby replica and places the standby replica into the preheating pool.

[0078] In this embodiment, periodically collecting the average request rate, average inference latency, and current service instance replicas of the large language model within a time window refers to the system continuously collecting operational data at fixed time intervals (e.g., a five-minute telemetry window) through agent components deployed on each inference instance side. Specifically, the request rate represents the number of requests arriving at a specific model service per unit of time, typically measured in requests per second; the average inference latency records the average time taken for a single request to receive a response from entering the system; and the current service instance replicas are obtained in real-time through the container orchestration platform interface, reflecting the number of container instances currently active and capable of providing inference services for the model service. This data is aggregated into the service registry module to support subsequent capacity planning decisions.

[0079] Calculating the target replica count based on Little's Law applies a classic relation from queuing theory to capacity estimation in large language model inference scenarios. Little's Law states that the average number of requests in a system equals the request arrival rate multiplied by the average response time, i.e., L equals λ multiplied by W. In this implementation scenario, the formula for calculating the target replica count is: multiply the average request rate by the average inference latency, divide by the concurrency threshold of a single service instance replica, and finally round up. The concurrency threshold refers to the maximum number of concurrent requests a single replica can handle simultaneously; this parameter is pre-set based on the inference engine performance and hardware resource configuration. Through this calculation, the system can scientifically estimate the minimum number of replicas required to meet the current load demand, avoiding over- or under-provisioning of resources.

[0080] Based on the number of parameters, large language models are divided into three levels: small, medium, and large models, which forms the basis for differentiated configuration of the preheating pool. Specifically, models with fewer than 10 billion parameters are defined as small models, those with between 10 billion and 70 billion parameters are defined as medium models, and those with more than 70 billion parameters are defined as large models. This hierarchical division reflects the significant differences in resource consumption, loading time, and response characteristics between models of different sizes. The number of parameters is a core indicator for measuring the scale of a neural network model, directly determining the size of the model weight file and the GPU memory required for loading.

[0081] Configuring a minimum number of standby replicas for large language models at different levels reflects a trade-off between resource efficiency and response speed. The warm pool maintains a certain number of standby replicas. These replicas keep the container process running and complete the inference engine initialization, while preloading model weights into the GPU's video memory or system memory, but do not provide inference services externally. The specific configuration rules are: a minimum of 2 standby replicas for small models, 1 for medium models, and 0 or 1 for large models. Higher-level models have higher video memory consumption, making the resource cost of maintaining standby replicas high; therefore, a smaller warm pool is configured. Lower-level models are configured with a relatively larger number of standby replicas to handle sudden requests and reduce activation latency.

[0082] When the target number of replicas exceeds the current number of service instance replicas and the cooldown period has expired, the system performs a scaling operation. The cooldown period is the minimum time interval set to prevent the system from frequently triggering scaling due to load fluctuations; the default value is 300 seconds. If the scaling requirement is met, the system first checks if there are available standby replicas in the preheating pool. If so, it sends an activation signal to the standby replica (e.g., by modifying the state file within the container or calling an internal interface), causing its readiness probe to return a success status, thus adding it to the service endpoints and enabling it to begin receiving external requests. Because model weights have already been preloaded, this activation method has significantly lower latency than a cold start process that starts from scratch. If there are insufficient replicas in the preheating pool, a new service instance is created through the container orchestration platform interface.

[0083] When the idle time of a service instance of a large language model exceeds the idle threshold (τ), the system performs scaling down and preheating pool degradation operations. Specifically, the system degrades redundant service instances that exceed the minimum number of standby replicas in the preheating pool to standby replicas in the preheating pool. These replicas return a failure status via a configuration-ready probe, causing them to be removed from the server point and no longer receive external inference requests, but the container process remains running and the model weights are loaded. This mechanism reduces idle resource consumption while maintaining fast response capabilities, because although standby replicas occupy GPU memory, they do not perform inference computations, and their computing resources can be reused by other tasks.

[0084] When a service instance of a large language model remains idle for longer than the idle time limit, the system performs a complete scale-to-zero operation. This means destroying all service instances and standby replicas in the preheating pool, reducing the replica count to zero. This strategy is suitable for scenarios with long periods of no requests, maximizing the release of GPU resources and eliminating all idle resource occupation. However, because model weights are not preloaded, subsequent requests will trigger a complete cold start process, including container creation, inference engine initialization, and model weight loading, potentially increasing latency from seconds to minutes. Therefore, this strategy is typically used in conjunction with a preheating pool mechanism, and is enabled by operations personnel based on business characteristics in resource-sensitive scenarios.

[0085] The system first continuously records the historical request arrival time series for each deployed large language model (grouped and merged by model name, quantization precision, and hardware type). This series can be timestamped down to the second or millisecond level and stored in a time-series database or circular buffer. To capture periodic business load patterns (e.g., daily peaks, weekly activities, nighttime troughs), the system maintains at least 7 to 30 days of historical data. An orchestrator embeds a time-series forecasting model, which can be a classic statistical model such as SARIMA (Seasonally Regressive Integral Moving Average) to handle request arrival rates with clear daily and weekly cycles; or a lightweight machine learning model such as Prophet (an open-source time-series forecasting tool from Facebook) that can automatically identify trends, seasonality, and holiday effects; or, where computational resources allow, deep models such as LSTM or TCN (Temporal Convolutional Network) can be used to predict the probability of multi-step arrival intervals. The forecasting model is executed periodically in a sliding window manner (e.g., retraining or updating model parameters every 5 minutes), outputting the time window for the next active period. An active window is defined as the time period during which the request arrival rate exceeds a preset wake-up threshold (e.g., 2 requests per second) or the interval between consecutive requests is less than a threshold (e.g., 10 seconds). The prediction result typically includes the window's start time, end time, and confidence level. Then, the system calculates the warm-up trigger time by subtracting an empirically configured warm-up offset time point (e.g., the maximum of 2 minutes or 5 model loading cycles) from the predicted active window start time. The warm-up offset needs to comprehensively consider container image pull time, model weight loading time from object storage to GPU memory, inference engine initialization time, and health check time. The orchestrator maintains a timed trigger (e.g., based on Cron or an internal clock loop). When the system time reaches the warm-up trigger time, the orchestrator actively calls the underlying container platform's API (e.g., Kubernetes' Deployment scaling interface or a custom resource CRD) to create at least one standby replica based on the estimated load intensity of the prediction window. The created copy uses the exact same container image, environment variables, model version, and quantization precision as the target large language model, but its initial state is marked as "standby," meaning that after loading the model, it does not receive external traffic, but only keeps the process alive and waits for activation.

[0086] After creation, the orchestrator registers the replica's identification information (such as Pod IP, port, resource ID) and metadata such as its model and preheating timestamp into the preheating pool, and adds it to the preheating instance list in the service registry. The preheating pool periodically performs heartbeat checks to ensure the health of standby replicas; if a standby replica survives in the preheating pool for more than the maximum idle time without being activated, it will be reclaimed to release resources. Through this proactive predictive preheating mechanism, the system can ensure that model instances are already in a "hot" state before the actual request peak arrives, thereby completely eliminating the cold start latency during request routing (which can typically reach tens of seconds or even minutes), retaining only a few milliseconds of overhead for instance activation and traffic access. This is particularly suitable for business scenarios with significant periodic fluctuations (such as high load during weekdays and low load at night), avoiding the resource waste caused by maintaining a large number of idle instances for a long time, while ensuring immediate response capabilities during peak periods.

[0087] This embodiment periodically collects average request rate, inference latency, and replica count to calculate the target replica count. Based on parameter levels, the LLM is divided into three tiers: small, medium, and large, and a differentiated preheating pool with a minimum number of standby replicas is configured. When the target replica count exceeds the current replica count and the cooldown period expires, the system expands; if idle time exceeds a threshold, it is downgraded to a preheating pool with a minimum number of replicas; if idle time exceeds the time limit, the system shrinks to zero. The effect is to achieve three levels of elastic scheduling: accurate prediction, tiered preheating, and gradient cooling, balancing response speed and resource cost, and avoiding frequent cold starts and resource waste.

[0088] For example, in the financial health business, the system periodically collects the average request rate, average inference latency, and current number of service instance replicas for large language models within a time window. Based on Little's rule, the target number of replicas is calculated by multiplying the average request rate by the average inference latency. For instance, during the quarterly financial report release period, a surge in investment analysis requests causes the target number of replicas to expand from 2 to 8. The system divides the models into three levels based on the number of parameters: small models for real-time market data Q&A, medium models for portfolio risk assessment, and large models for stress testing and complex derivative pricing. A minimum number of standby replicas is configured in a preheating pool for each level, such as 2 for small models and 1 for large models. When the target number of replicas exceeds the current number and the cooldown period has passed, standby replicas are activated from the preheating pool to achieve second-level expansion, ensuring zero latency during trading hours. When the idle time of service instances exceeds a threshold after the market closes at night, redundant instances are downgraded to standby replicas, retaining only the minimum number of replicas to reduce costs. When the idle time exceeds the time limit during long holidays, all instances and replicas are scaled down to zero to avoid resource idleness and achieve an optimal balance between financial business elasticity and cost.

[0089] In the healthcare business domain, the system continuously monitors the operational metrics of large language models and dynamically calculates the target replica count based on Little's rule. For example, during flu season, a surge in symptom self-check requests increases the target replica count from 3 to 10. The system is tiered by parameter quantity: small models are used for medication reminders and health education, medium models for interpreting test reports, and large models for multimodal image diagnosis and genomic analysis. Each model is configured with a minimum number of standby replicas in a preheating pool to handle sudden emergency demands. When the target replica count exceeds the current replica count and the cooldown period ends, standby replicas are immediately expanded to ensure diagnostic timeliness in emergency scenarios such as chest pain centers. When instances are idle for longer than the time limit after the early morning outpatient hours, redundant instances are downgraded to standby replicas, retaining only the minimum number of replicas to maintain basic standby capacity. When idle time exceeds the time limit during holidays or system maintenance, the system is completely scaled down to zero to release resources. This tiered elastic mechanism ensures rapid clinical response during peak emergency periods while avoiding the ineffective use of medical computing resources, supporting the refined operation and management of medical institutions.

[0090] In one embodiment, a multi-objective large language model orchestration apparatus is provided, which corresponds one-to-one with the multi-objective large language model orchestration method described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the multi-objective large language model orchestration device of the present invention. The modules include an API gateway module 10, a routing module 20, a multi-objective scoring module 30, a dynamic orchestration module 40, and an inference backend module 50. Detailed descriptions of each functional module are as follows: API Gateway Module 10 is used to receive large language model prompt requests input by the user. The routing module 20 is used to perform complexity analysis on the large language model prompt request and generate a complexity level or complexity probability distribution. The multi-objective scoring module 30 is used to calculate the multi-objective orchestration score of candidate service instances in the preset service matrix based on the complexity level or complexity probability distribution. The dynamic orchestration module 40 is used to select the candidate service instance with the highest multi-objective orchestration score as the target service instance of the large language model. The inference backend module 50 is used to dynamically schedule computing resources according to the current running status of the target service instance, execute the large language model prompt request through computing resources, and return the inference result; Confidence module 60 is used to perform confidence self-evaluation on the inference result and generate a confidence score; The secondary inference module 70 is used to automatically trigger a secondary inference request when the confidence score is lower than a preset confidence threshold, and to route the secondary inference request to a backup service instance. The complexity update module 80 is used to compare the secondary reasoning result with the original reasoning result and update the complexity probability distribution of the large language model prompt request based on the comparison result.

[0091] In one embodiment, the API gateway module 10 includes: The API gateway module receives text-based large language model suggestion requests or multimodal large language model suggestion requests from users. Preprocess the text-based large language model suggestion request or the multimodal large language model suggestion request; The preprocessed text-based large language model suggestion request or multimodal large language model suggestion request is validated, and the validated text-based large language model suggestion request or multimodal large language model suggestion request is forwarded to the routing module.

[0092] In one embodiment, the routing module 20 includes: The intent of the large language model prompt request is parsed to extract the expected output length, required external knowledge domain, inference chain depth, and number of tool calls. The routing module identifies whether the large language model prompt request contains preset indicative keywords; If the large language model suggestion request contains low-complexity keywords, then the large language model suggestion request is determined to be of low complexity level; If the large language model suggestion request contains medium-complexity keywords, then the large language model suggestion request is determined to be of medium complexity level; If the large language model suggestion request contains high-complexity keywords, then the large language model suggestion request is determined to be of a high complexity level; If the large language model prompt request does not contain preset indicative keywords, then the large language model prompt request is encoded by a semantic classifier, and the complexity probability distribution is output. The expected output length, required external knowledge domain, inference chain depth, and number of tool calls are used as auxiliary features and input into the complexity analysis or semantic classifier to correct the complexity level or complexity probability distribution.

[0093] In one embodiment, the multi-objective scoring module 30 includes: The orchestrator retrieves the service matrix and candidate service instances corresponding to the large language model from the service registry based on the complexity level or complexity probability distribution. The complexity level or complexity probability distribution is weighted and multiplied by the capability vector of the service matrix to generate a model relevance score, and the model relevance score is then normalized. Obtain the real-time inference latency and unit request resource cost of candidate service instances from the service registry; The real-time inference latency and unit request resource cost are normalized respectively to generate latency score and cost score; Obtain the model parameter count, quantization accuracy, context window length, and hardware accelerator type for each candidate service instance; Based on the complexity level or complexity probability distribution, the model parameter quantity, quantization accuracy, context window length, and hardware accelerator type are weighted and scored to generate an execution suitability score. Based on preset preference weights, the normalized model relevance score, execution adaptability score, latency score, and cost score are weighted and summed to generate a multi-objective orchestration score for candidate service instances.

[0094] In one embodiment, the inference backend module 50 includes: Obtain the current running status of the target service instance; If the current running status is that the instance is activated, the large language model hint request will be routed to the target service instance to perform the inference task; When the current running state is that the instance is activated but the load exceeds the preset overload threshold, a standby replica with the same model and quantization accuracy as the target service instance is selected from the preheating pool. The standby replica is activated as a parallel service instance. Using a request-level or task-level load balancing strategy, part of the computational tasks or batch requests of the large language model prompt request are distributed to the parallel service instance to execute the inference task. The inference results of the target service instance and the inference sub-results of the parallel service instances are merged to generate the final inference result; If the current running status is that the instance is not activated, check if there are any standby replicas in the preheating pool; If the preheating pool has standby replicas, then activate the standby replicas to convert them into target service instances and execute inference tasks; If the preheating pool has no standby replicas, the container platform is invoked through the orchestrator to create a container instance, which is then used as the target service instance to execute the inference task.

[0095] In one embodiment, the service matrix update module includes: Collect and analyze response latency, resource consumption, and load metrics generated during inference tasks. The response latency metrics, resource consumption metrics, and load metrics are transmitted to the service registry. The service registry updates the service matrix based on response latency metrics, resource consumption metrics, and load metrics; A sliding time window was used to analyze the trend of the average orchestration score of each service instance across multiple historical windows. When the orchestration score of a specific service instance is detected to be continuously decreasing and the decrease exceeds the dynamic threshold, the health probe or offline diagnostic process of the service instance is triggered. If a diagnosis confirms performance degradation or anomalies, the service instance will be temporarily removed from the service matrix, and a backup service instance will be started to take over its workload.

[0096] In one embodiment, the orchestrator scaling module includes: Periodically collect the average request rate, average inference latency, and number of current service instance replicas of the large language model within a time window; Based on Little's rule, the target number of replicas is calculated according to the average request rate and average inference latency. Based on the number of parameters, the large language model is divided into three levels: small model, medium model, and large model. Configure the minimum number of standby copies in the warm-up pool for large language models at different levels; When the target number of replicas is greater than the current number of service instance replicas and the cooldown period has expired, activate standby replicas from the preheating pool to convert them into service instances for scaling. When the idle time of a service instance of a large language model exceeds the idle threshold, the redundant service instance will be downgraded to a standby replica in the preheating pool, and only the minimum number of standby replicas in the preheating pool will be retained for scaling down. When the idle time of the service instance of the large language model exceeds the idle time limit, the service instance and standby replicas of the large language model will be scaled down to zero. Record the historical request arrival time series of the large language model, and use the time series prediction model to predict the time window of the next active cycle; At a preheating offset time point before the active cycle arrives, the orchestrator actively calls the container platform to pre-create at least one standby replica and places the standby replica into the preheating pool.

[0097] Specific limitations regarding the multi-objective large language model orchestration device can be found in the aforementioned limitations on the multi-objective large language model orchestration method, and will not be repeated here. Each module in the aforementioned multi-objective large language model orchestration device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0098] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements server-side functions or steps of a multi-objective large language model orchestration method.

[0099] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a multi-objective large language model orchestration method.

[0100] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Receive large language model prompts from user input; The complexity of the large language model prompt request is analyzed to generate a complexity level or complexity probability distribution; Calculate the multi-objective orchestration score of candidate service instances in the preset service matrix based on the complexity level or complexity probability distribution; The candidate service instance with the highest multi-objective orchestration score is used as the target service instance of the large language model. Based on the current running status of the target service instance, computing resources are dynamically scheduled to execute the large language model prompt request and return the inference result. The reasoning results are self-assessed to generate a confidence score. When the confidence score is lower than the preset confidence threshold, a secondary inference request is automatically triggered and the secondary inference request is routed to the backup service instance. The results of the secondary inference are compared with the original inference results, and the complexity probability distribution of the large language model prompt request is updated based on the comparison results.

[0101] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, and a computer program is stored thereon, which, when executed by a processor, performs the following steps: Receive large language model prompts from user input; The complexity of the large language model prompt request is analyzed to generate a complexity level or complexity probability distribution; Calculate the multi-objective orchestration score of candidate service instances in the preset service matrix based on the complexity level or complexity probability distribution; The candidate service instance with the highest multi-objective orchestration score is used as the target service instance of the large language model. Based on the current running status of the target service instance, computing resources are dynamically scheduled to execute the large language model prompt request and return the inference result. The reasoning results are self-assessed to generate a confidence score. When the confidence score is lower than the preset confidence threshold, a secondary inference request is automatically triggered and the secondary inference request is routed to the backup service instance. The results of the secondary inference are compared with the original inference results, and the complexity probability distribution of the large language model prompt request is updated based on the comparison results.

[0102] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0103] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0104] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0105] It should be noted that any software tools or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0106] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for arranging large language models based on multiple objectives, characterized in that, Includes the following steps: Receive large language model prompts from user input; The complexity of the large language model prompt request is analyzed to generate a complexity level or complexity probability distribution; Calculate the multi-objective orchestration score of candidate service instances in the preset service matrix based on the complexity level or complexity probability distribution; The candidate service instance with the highest multi-objective orchestration score is used as the target service instance of the large language model. Based on the current running status of the target service instance, computing resources are dynamically scheduled to execute the large language model prompt request and return the inference result. The reasoning results are self-assessed to generate a confidence score. When the confidence score is lower than the preset confidence threshold, a secondary inference request is automatically triggered and the secondary inference request is routed to the backup service instance. The results of the secondary inference are compared with the original inference results, and the complexity probability distribution of the large language model prompt request is updated based on the comparison results.

2. The large language model arrangement method based on multi-objectives as described in claim 1, characterized in that, Receive large language model suggestion requests from user input, including: The API gateway module receives text-based large language model suggestion requests or multimodal large language model suggestion requests from users. Preprocess the text-based large language model prompt request or the multimodal large language model prompt request; The preprocessed text-based large language model suggestion request or multimodal large language model suggestion request is validated, and the validated text-based large language model suggestion request or multimodal large language model suggestion request is forwarded to the routing module.

3. The large language model arrangement method based on multi-objectives as described in claim 1, characterized in that, Perform complexity analysis on the large language model prompt request to generate a complexity level or complexity probability distribution, including: The intent of the large language model prompt request is parsed to extract the expected output length, required external knowledge domain, inference chain depth, and number of tool calls. The routing module identifies whether the large language model prompt request contains preset indicative keywords; If the large language model suggestion request contains low-complexity keywords, then the large language model suggestion request is determined to be of low complexity level; If the large language model suggestion request contains medium-complexity keywords, then the large language model suggestion request is determined to be of medium complexity level; If the large language model suggestion request contains high-complexity keywords, then the large language model suggestion request is determined to be of a high complexity level; If the large language model prompt request does not contain preset indicative keywords, then the large language model prompt request is encoded by a semantic classifier, and the complexity probability distribution is output. The expected output length, required external knowledge domain, inference chain depth, and number of tool calls are used as auxiliary features and input into the complexity analysis or semantic classifier to correct the complexity level or complexity probability distribution.

4. The large language model arrangement method based on multi-objectives as described in claim 1, characterized in that, Based on the complexity level or complexity probability distribution, calculate the multi-objective orchestration score of candidate service instances in the preset service matrix, including: The orchestrator retrieves the service matrix and candidate service instances corresponding to the large language model from the service registry based on the complexity level or complexity probability distribution. The complexity level or complexity probability distribution is weighted and multiplied by the capability vector of the service matrix to generate a model relevance score, and the model relevance score is then normalized. Obtain the real-time inference latency and unit request resource cost of candidate service instances from the service registry; The real-time inference latency and unit request resource cost are normalized respectively to generate latency score and cost score; Obtain the model parameter count, quantization accuracy, context window length, and hardware accelerator type for each candidate service instance; Based on the complexity level or complexity probability distribution, the model parameter quantity, quantization accuracy, context window length, and hardware accelerator type are weighted and scored to generate an execution suitability score. Based on preset preference weights, the normalized model relevance score, execution adaptability score, latency score, and cost score are weighted and summed to generate a multi-objective orchestration score for candidate service instances.

5. The large language model arrangement method based on multi-objectives as described in claim 1, characterized in that, Based on the current running status of the target service instance, computing resources are dynamically scheduled to execute the large language model prompt request and return the inference result, including: Obtain the current running status of the target service instance; If the current running status is that the instance is activated, the large language model hint request will be routed to the target service instance to perform the inference task; When the current running state is that the instance is activated but the load exceeds the preset overload threshold, a standby replica with the same model and quantization accuracy as the target service instance is selected from the preheating pool. The standby replica is activated as a parallel service instance. Using a request-level or task-level load balancing strategy, part of the computational tasks or batch requests of the large language model prompt request are distributed to the parallel service instance to execute the inference task. The inference results of the target service instance and the inference sub-results of the parallel service instances are merged to generate the final inference result; If the current running status is that the instance is not activated, check if there are any standby replicas in the preheating pool; If the preheating pool has standby replicas, then activate the standby replicas to convert them into target service instances and execute inference tasks; If the preheating pool has no standby replicas, the container platform is invoked through the orchestrator to create a container instance, which is then used as the target service instance to execute the inference task.

6. The large language model arrangement method based on multi-objectives as described in claim 1, characterized in that, Based on the current running status of the target service instance, computing resources are dynamically scheduled. After executing the large language model prompt request using the computing resources and returning the inference result, the process includes: Collect and analyze response latency, resource consumption, and load metrics generated during inference tasks. The response latency metrics, resource consumption metrics, and load metrics are transmitted to the service registry. The service registry updates the service matrix based on response latency metrics, resource consumption metrics, and load metrics; A sliding time window was used to analyze the trend of the average orchestration score of each service instance across multiple historical windows. When the orchestration score of a specific service instance is detected to be continuously decreasing and the decrease exceeds the dynamic threshold, the health probe or offline diagnostic process of the service instance is triggered. If a diagnosis confirms performance degradation or anomalies, the service instance will be temporarily removed from the service matrix, and a backup service instance will be started to take over its workload.

7. The large language model arrangement method based on multi-objectives as described in claim 5, characterized in that, Based on the current running status of the target service instance, before dynamically scheduling computing resources, executing the large language model prompt request using computing resources, and returning the inference result, the process includes: Periodically collect the average request rate, average inference latency, and number of current service instance replicas of the large language model within a time window; Based on Little's rule, the target number of replicas is calculated according to the average request rate and average inference latency. Based on the number of parameters, the large language model is divided into three levels: small model, medium model, and large model. Configure the minimum number of standby copies in the warm-up pool for large language models at different levels; When the target number of replicas is greater than the current number of service instance replicas and the cooldown period has expired, activate standby replicas from the preheating pool to convert them into service instances for scaling. When the idle time of a service instance of a large language model exceeds the idle threshold, the redundant service instance will be downgraded to a standby replica in the preheating pool, and only the minimum number of standby replicas in the preheating pool will be retained for scaling down. When the idle time of the service instance of the large language model exceeds the idle time limit, the service instance and standby replicas of the large language model will be scaled down to zero. Record the historical request arrival time series of the large language model, and use the time series prediction model to predict the time window of the next active cycle; At a preheating offset time point before the active cycle arrives, the orchestrator actively calls the container platform to pre-create at least one standby replica and places the standby replica into the preheating pool.

8. A large language model arrangement device based on multiple objectives, characterized in that, The multi-objective-based large language model orchestration device includes: The API gateway module is used to receive large language model suggestion requests from users. The routing module is used to perform complexity analysis on the large language model prompt request and generate a complexity level or complexity probability distribution. The multi-objective scoring module is used to calculate the multi-objective orchestration score of candidate service instances in the preset service matrix based on the complexity level or complexity probability distribution. The dynamic orchestration module is used to select the candidate service instance with the highest multi-objective orchestration score as the target service instance of the large language model. The inference backend module is used to dynamically schedule computing resources based on the current running status of the target service instance, execute the large language model prompt request through computing resources, and return the inference result; The confidence module is used to perform a confidence self-evaluation on the inference results and generate a confidence score; The secondary inference module is used to automatically trigger a secondary inference request when the confidence score is lower than a preset confidence threshold, and to route the secondary inference request to a backup service instance. The complexity update module is used to compare the secondary inference result with the original inference result and update the complexity probability distribution of the large language model prompt request based on the comparison result.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a multi-objective large language model orchestration program stored in the memory and executable on the processor. When executed by the processor, the multi-objective large language model orchestration program implements the steps of the multi-objective large language model orchestration method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a large language model orchestration program based on multiple objectives, which, when executed by a processor, implements the steps of the large language model orchestration method based on multiple objectives as described in any one of claims 1-7.

Images