A Smart Routing and Unified Adaptation Method for Large Language Models

Abstract

This invention discloses a method for intelligent routing and unified adaptation of large language models. It defines all access details of the model through a declarative configuration file, allowing new models to be added without writing any code, achieving zero-code access for large language models. It provides a completely consistent calling interface for upstream applications, shielding the heterogeneity of all downstream large language models and constructing a unified request and response abstraction layer. Through a strategy engine and JSON path technology, it accurately handles complex streaming responses, including thought content, with intelligent parsing and content extraction. It supports dynamic configuration and intelligent strategy hot updates. It achieves intelligent model routing, dynamically selecting the optimal large language model instance. Simultaneously, it provides enterprise-level governance capabilities, integrating circuit breaking, degradation, rate limiting, and monitoring functions to ensure stability for model calls. This comprehensively improves system maintainability, scalability, and user experience consistency.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and artificial intelligence middleware technology, specifically to a method for achieving zero-code access, dynamic adaptation, intelligent scheduling, and unified governance of heterogeneous large language models through a combination of declarative configuration, policy engine, and intelligent routing. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) have made groundbreaking progress in the field of natural language processing. The LLM ecosystem is flourishing, with models such as GPT, Claude, Wenxin Yiyan, Tongyi Qianwen, GLM, and DeepSeek emerging. While these models offer powerful capabilities, they also introduce significant integration complexity.

[0003] Interface heterogeneity: The API endpoints, HTTP methods, authentication methods (such as API Key location), request parameters (such as model, messages, stream), and response structures of each model are very different.

[0004] Capability and Pattern Differences: Some models support streaming output of "thinking chains" or "reasoning processes," the content of which is intertwined with regular dialogue content in the data stream, resulting in inconsistent formats. The triggering and differentiation mechanisms for dialogue modes and thinking modes differ (some are controlled by parameters, while others are distinguished by content markers).

[0005] In practical applications, enterprises often need to connect to multiple models simultaneously to meet different business needs or to evaluate model selection. While some solutions have attempted to address the complexity of multi-model integration, the following issues remain:

[0006] Existing frameworks lack flexibility: Some open-source libraries (such as LangChain) provide basic abstractions, but developers still need to programmatically assemble and configure components, failing to achieve true "configuration-as-access".

[0007] Cloud service provider lock-in: Some cloud service platforms provide a unified API portal, but they are usually locked to a specific model or vendor, lacking the flexibility to cross cloud and private deployment models.

[0008] High integration costs: Some systems integrate different large language models, which usually requires writing adaptation code for each model separately. This results in high development costs, difficult maintenance, poor scalability and portability, and the inability to achieve hot updates of configurations.

[0009] Lack of governance capabilities: Existing adaptation solutions lack governance capabilities such as monitoring, rate limiting, degradation, and circuit breaking of model usage, making it difficult to guarantee stability and reliability in production environments.

[0010] Poor scalability: Existing technologies lack solutions for multi-model adaptation through pure configuration, especially systems that can dynamically adapt to new models without modifying the code.

[0011] Therefore, there is an urgent need for a technical solution that can dynamically and losslessly access and manage multiple large language models through pure configuration, fundamentally reducing integration complexity and improving the scalability and maintainability of the system. Summary of the Invention

[0012] The purpose of this invention is to provide a method for intelligent routing and unified adaptation of large language models based on a configuration and strategy engine. It defines all access details of the model through declarative configuration files, eliminating the need to write any code when adding a new model, thus achieving zero-code access for large language models. It provides a completely consistent calling interface for upstream applications, shielding the heterogeneity of all downstream large language models and constructing a unified request and response abstraction layer. Through a strategy engine and JSON path technology, it accurately handles complex streaming responses, including thought content, with intelligent parsing and content extraction. It supports dynamic configuration and intelligent strategy hot updates. It achieves intelligent model routing, dynamically selecting the optimal large language model instance. Simultaneously, it provides enterprise-level governance capabilities, integrating circuit breaking, degradation, rate limiting, and monitoring functions to ensure stability for model calls. This comprehensively improves system maintainability, scalability, and user experience consistency.

[0013] To achieve the above objectives, the technical solution adopted by this invention is as follows: a large language model intelligent routing and unified adaptation method, based on a large language model intelligent routing and unified adaptation system, the system including a unified interface and risk identification module, a dynamic configuration module, a strategy and dynamic selection module, and a system processing and control module, the method including the following steps:

[0014] Step S1, Unified Interface and Risk Identification Module: First, a unified API interface is provided to the upstream access end to shield the differences between all model input and output parameters. Then, the counting module in the smallest unit of text processing, Token, counts and records the data to support enterprise-level cost governance. Finally, a sliding window is used to dynamically identify risks in the model's input and output.

[0015] Step S2, Dynamic Configuration Module: Defines a configuration file, which consists of three main parts: request template mapping, strategy configuration list, and default strategy configuration. When the system loads a JSON file, it reads the model configuration information and converts it into a configuration strategy cache; it dynamically senses changes in the JSON file and updates the configuration cache in real time at the model granularity.

[0016] Step S3, Strategy and Dynamic Selection Module: Based on the input information from the upstream system, select the configuration information of the corresponding model and select the correct strategy type to process the request and return of the downstream large model; if the input information does not define a specific model, the system dynamically selects a matching model for processing.

[0017] Step S4, System Processing and Control Module: The system builds unified observability and resource governance capabilities, and implements full lifecycle performance monitoring, stability assurance and business operation support for the large model call chain.

[0018] Compared with the prior art, the technical solution proposed in this invention has the following technical effects:

[0019] 1. This invention defines all the access details of the model through a declarative configuration file, so that no code needs to be written when adding a new model, thus achieving zero-code access of large language models.

[0020] 2. This invention provides a completely consistent calling interface for upstream applications, shields the heterogeneity of all large language models downstream, and builds a unified request and response abstraction layer.

[0021] 3. This invention uses a strategy engine and JSON path technology to accurately process complex streaming responses, including thought content, and intelligently parses and extracts content.

[0022] 4. This invention supports dynamic configuration and intelligent strategy hot updates; it enables intelligent model routing and dynamically selects the optimal large language model instance.

[0023] 5. This invention also provides enterprise-level governance capabilities, integrating governance functions such as circuit breaking, degradation, rate limiting, and monitoring to ensure stability for model calls. This comprehensively improves system maintainability, scalability, and user experience consistency. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the overall process of the method of the present invention.

[0025] Figure 2 This is a schematic diagram of the overall architecture of the system of the present invention.

[0026] Figure 3 This is a schematic diagram of the large model request process in the method of the present invention.

[0027] Figure 4 This is a schematic diagram of the processing engine execution flow in the method of the present invention. Detailed Implementation

[0028] The specific embodiments of the present invention will be described in detail below.

[0029] Combination Figure 1-4The present invention relates to a large language model intelligent routing and unified adaptation system and method based on configuration and policy engine, comprising the following steps:

[0030] Step S1, Unified Interface and Risk Identification Module:

[0031] In this step, a unified API (Application Programming Interface) is first provided to the upstream access end to shield the differences in all model input and output parameters. Then, the token (the smallest unit of text processing) counting module counts and records the data to support enterprise-level cost governance. Finally, a sliding window is used to dynamically identify risks in the model's input and output.

[0032] Step S101, provide a unified interface:

[0033] This module provides a standard, unified API interface, shielding the differences in input and output parameters between different models.

[0034] First, standard API endpoints, request methods, and authentication information are provided. Second, input parameters are defined, including the requested model name (if empty, the optimal model is intelligently determined by the dynamic selection engine), whether deep thinking is required (default: no), whether online search is required (default: no), session identifier (generated each time a conversation is created), and user-input prompts. Finally, output parameters are defined, including session identifier, type, time, model name, confidence level, whether the conversation has ended, reason for ending, and the formal output content. The formal content consists of an index, role, main text, deep thinking content, and whether deep thinking was included. Output is streamed via Server-Sent Events (a technology based on the HTTP protocol that allows servers to push updates to clients in real time), achieving a smooth output experience.

[0035] Step S102, record the number of tokens:

[0036] This module counts the number of texts input and output, facilitating enterprise-level cost management.

[0037] During input, the system extracts prompts from the user's input; during output, it extracts the text from the deep thinking section and the main body, obtains the correct number of text units through a token counting service, and saves it to the current session and the database. Throughout this process, it determines whether the user's remaining credit limit can support the current session to ensure accurate cost control.

[0038] Step S103, Model Risk Assessment Module:

[0039] This module performs risk assessment on the prompts entered by the user and the content returned by the large model.

[0040] User-input prompts are directly processed by the model's risk assessment module to identify risky content (illegal, violent, discriminatory, privacy-related, misleading, etc.). For the streaming content returned by the model, a sliding window mechanism is used, requiring simultaneous generation, detection, and output. The model's risk assessment module achieves streaming content detection through incremental sliding windows and a buffer mechanism.

[0041] Let the token sequence output by the model be... (t = 1,2,…,T), where Indicates the index of the current time step. This indicates the final generated length (unknown in streaming scenarios, and gradually increases). Indicates the first The module maintains a token generated at each time step. Recently Dynamic buffer for each token (When t < L, the entire history is taken), where This indicates the maximum length of the buffer, used to limit memory and focus on recent contexts. The buffer is dynamically updated as new tokens arrive.

[0042] And define the sliding window size as ( The overlap retention length is... Risk assessment is performed when triggering conditions are met (such as encountering a sentence termination character, or the buffer length reaching a certain threshold). Execute when the time interval exceeds a threshold, at which point the current discrimination window is extracted (the closest one at the end of the buffer is taken). (Number of tokens):

[0043]

[0044] in This indicates recently generated content and its immediate context. Indicates taking the last one Each element.

[0045] Finally, the acquired content is input into the risk assessment module. If a high risk is detected, generation is immediately terminated and a blocking mechanism is triggered, and a risk log is recorded. If the risk is below the risk threshold, the log is recorded and generation continues. After completing one assessment, a portion of historical tokens is reserved as context overlap for the next round, and the buffer is subsequently updated according to the overlap strategy.

[0046]

[0047] reserve This avoids semantic fragmentation; when a new token arrives, it will be appended in subsequent steps. end.

[0048] Step S2, Dynamic Configuration Module:

[0049] In this step, the system loads the JSON file, reads the model configuration information, and dynamically detects changes in the JSON file, updating the configuration cache in real time.

[0050] Step S201, Load model configuration information:

[0051] This step focuses on setting, loading, and parsing the core configuration file to ensure correct input and parsing of the model.

[0052] First, define the JSON file framework, which consists of three main parts: request template mapping, strategy configuration list, and default strategy configuration.

[0053]

[0054] in For this structured JSON object, To request a template mapping, Configure a list of strategies. This is the default strategy configuration. The request template mapping defines the required request parameters and formats for each model, while the strategy configuration list defines the request and output formats, stream processing methods, etc., for each model. The default strategy configuration is used for models not defined in the strategy configuration list (unknown models or unified models). Both the request template mapping and the strategy configuration list support inheritance. New request template mappings can inherit from existing ones, and new strategy configurations can inherit from existing strategy configurations in the strategy configuration list or the default strategy configuration, only needing to override the differences, reducing configuration redundancy.

[0055] Request template mapping Multiple sets of different request templates are defined. Each request template includes a request template name, a parent request template name, model input parameters, additional parameters, and embedding format:

[0056]

[0057] in To request the template mapping collection index, For indexing, taken from the index set , Indicates the first Each request template mapping object. Each request template mapping:

[0058]

[0059] in Indicates the first The template name of the request template mapping object; Request the template mapping name for the parent class:

[0060]

[0061] The parent request template mapping name can be (Indicates no parent class), or is , indicating a relationship of inheritance. This indicates the model's input parameters, additional parameters, and embedding format. For example, both the JJ and QQ models require input parameters such as model, prompt word, and whether to engage in deep thinking. However, the "whether to engage in deep thinking" field in the JJ model has other nested parameters (e.g., additional field 1. additional field 2. whether to engage in deep thinking). By defining the request format of this model in the template mapping, the corresponding large model can be correctly requested.

[0062] The strategy configuration list defines several different sets of model strategies:

[0063]

[0064] in Configure collection indexes for the strategy. For indexing, taken from the index set , Indicates the first Each policy configuration object. Each set of policy configurations... Includes the policy configuration name, parent policy configuration name, description, list of applicable major models, request template (the request template mapping described above), and response handling method:

[0065]

[0066] in Indicates the first The name of the policy configuration. Configure the name of the parent strategy for the strategy. Description of the policy configuration, A large list of models configured for the strategy ( Namespaces for all models Request templates configured for the strategy (for request template mapping) In ,Right now ), This refers to the response handling method.

[0067] Response handling method It ensures the correct parsing of the model's returned data and its correct output to the upstream system. This includes the number of data items to skip from the model (e.g., setting it to 2 will filter the first two data items), the model's output stop flag (correctly identifying the downstream model's stop output flag indicates that the model has finished outputting, which can tell the upstream system that the request has been completed and disconnect the streaming connection), preprocessing (including whether to ignore blank lines and truncate text), content parsing, output formatting, and processing strategies. as follows:

[0068]

[0069] in This indicates the number of items to skip from the model's output. This indicates the stop indicator of the model output. Indicates preprocessing. This indicates content parsing. Indicates output formatting. Indicates the processing strategy. Content parsing. It defines which field in the model's returned data retrieves the in-depth thinking content, which field retrieves the main text content, and which field is the in-depth thinking end marker. as follows:

[0070]

[0071] in This indicates a field representing deep thinking. Represents the text field. This indicates the end of deep thinking. (Processing strategy) This includes whether there is in-depth thinking and the type of strategy:

[0072]

[0073] in This indicates whether one has engaged in deep thinking. Indicates the strategy type. Strategy type There are three types of deep thinking: enabled deep thinking, identifier-based deep thinking (in step S101, the input parameter is requested: whether deep thinking is required), and content-based deep thinking (whether deep thinking is required is determined by the returned content; for example, some models have two fields: body text and deep thinking; some models have only one field, which has start and end identifiers). as follows:

[0074]

[0075] in This indicates the activation of deep thinking. This indicates deep thinking based on identifiers. It indicates in-depth thinking based on content.

[0076] The default policy configuration defines a generic model policy configuration. The format is similar to the policy configuration in the policy configuration list. However, there is no parent strategy configuration name. This is used for other models not defined in the strategy configuration list, which typically use the same input and output methods.

[0077] Secondly, the system starts by reading a JSON file and loading all configuration information into memory. Simultaneously, it safely parses the JSON into a set of key (model name) value (model configuration) pairs and a default model configuration, then executes the configuration loading function.

[0078]

[0079] in This indicates that the configuration parsing function (including validity checks, which check that all referenced request template mappings must exist) must exist. Field types meet expectations; required fields cannot be empty; all referenced parent strategy configuration names must exist. ), will change the original configuration Convert to runtime cache Deterministic functions, This indicates that the model strategy runtime mapping cache is used.

[0080]

[0081] in This represents the global model namespace, the set of names for all potentially invoked large language models. This indicates a specific model name. Representation Model The corresponding parsed policy object, It is a reserved keyword used to explicitly store the default strategy in the cache. This facilitates unified management and hot updates. If the model... If not declared in any strategy, then Not explicitly stored in Instead, it returns dynamically at runtime. The policy object has been resolved. as follows:

[0082]

[0083] in This indicates the existence of a certain strategy configuration object. Belongs to the strategy configuration list , Indicates model name Included in strategy In the list of applicable models, This represents a policy recursion resolution function (the input is a policy object that may contain references to parent policies). The output is a complete, parent-reference-free strategy object, whose field values ​​are merged according to the "child overrides parent" principle. Configurations are merged along the parent strategy chain, with child configurations overriding parent configurations (differential override semantics), supporting multi-level inheritance. The system uses a parsing function... Original configuration Convert to runtime cache First, verify the validity of the configuration, then configure each policy. Execution of inheritance resolution And then according to its The list maps the parsed results to the corresponding model names. Form key-value pairs At the same time, the default policy will be implemented. Save special key The final generated support Time complexity model strategy query.

[0084] Finally, this configuration information Cache the data so that when the model is requested again, the correct model configuration can be obtained and the correct strategy can be used to process the model input and output.

[0085] Step S202, Dynamic JSON File Detection:

[0086] Through a file system monitoring mechanism, the system dynamically detects changes in JSON files and reads updated configuration information. The original configuration information in step S201 By comparing and filtering out differences, configurations can be hot-loaded at the model level without restarting the service or interrupting the existing request flow.

[0087]

[0088] in for , Cache of runtime policy mappings generated based on the new configuration Structure and Consistent, This is a cache difference calculation function (performing a deep, structured equivalence assessment of the policy object). This refers to models (or default policies) that only contain semantically changed policy content and their new policy values. If... If a value exists, then only the changing model key... Perform atomic substitution:

[0089]

[0090] in This indicates a thread-safe cache update operation.

[0091] Step S3, Strategy and Dynamic Selection Module:

[0092] In this step, based on the input information from the upstream system, the system selects the appropriate model configuration information and chooses the correct strategy type to process the requests and responses from the downstream large model. If the input information does not define a specific model, the system will dynamically select the optimal model for processing.

[0093] Step S301, Optimal Model Selection:

[0094] This step is performed when a session is first created or a model is changed, and is used to select the downstream large model for this session.

[0095] First, the downstream large model name is obtained based on the request input parameters defined in step S101. If this parameter is not defined or the cached model configuration in step S2 does not exist, the dynamic selection engine obtains the optimal model name. The dynamic selection engine selects the optimal large language model through a routing decision chain based on monitoring metrics (capabilities, cost, performance, load). The routing decision chain consists of capability routing (routing to a list of models with the required capabilities, such as inference and code, as indicated in the request), cost routing (configuring unit call cost for each model), performance routing (based on real-time collected model response time, such as P95 / P99 latency and success rate), and load routing (weighted random or least-connection routing based on the real-time load of backend model instances). For example, candidate models are first filtered based on capabilities, then the candidate pool is sorted by cost, and finally the optimal model is selected.

[0096] Step S302, Select processing strategy:

[0097] This step loads the model strategy object information, encapsulates the input parameters, and requests the large model, unifying the various output methods of the large model under different strategy modes into a standardized output mode.

[0098] First, based on the large model name output in step S301, the strategy object information of the model is retrieved from the model strategy cache in step S201. Then, the upstream input is converted into the request parameters of the large model according to the request template mapping object in the strategy object information, and the downstream large model service is called.

[0099] Next, the downstream responses are processed according to the response handling method in the strategy object information. If the number of skipped responses returned by the model is not 0, the corresponding number of stream responses are ignored. Then, each response is preprocessed (whether to ignore blank lines, text that needs to be truncated). Then, the deep thinking content, the main text content, and the deep thinking stop marker are obtained through content parsing. This requires using a JSON parsing algorithm to accurately parse the actual values ​​in the content path configuration. Then, according to the processing strategy in the strategy template, state transition (from deep thinking mode to main text mode) and content transformation are performed.

[0100] To unify the handling of deep thinking and text modes across different models, a lightweight yet powerful strategy engine is abstracted. Its core consists of three precisely defined processing strategies, forming a finite state machine for handling streaming responses. If the strategy is "Enable Deep Thinking," it indicates that the large language model always carries a deep thinking mode. The upstream request's "deep thinking" parameter will be ignored; the deep thinking part must be processed first, followed by the text part. Content parsing will sequentially output the deep thinking, deep thinking stop flag, and text content. If the strategy is "Content-Based Deep Thinking" (which may or may not involve deep thinking), it requires processing via a content parsing algorithm. The presence of a thinking mode in this request is determined by the content or parameter values. If the strategy is "Identifier-Based Deep Thinking," processing will be based on the upstream request's "deep thinking" parameter.

[0101] All three strategies require proper handling of Deep Thinking mode and Main Text mode. There are two scenarios: one where Deep Thinking and Main Text use the same fields, and the other where they use different fields. For fields with the same data, the output should be distinguished as Main Text or Deep Thinking content based on the Deep Thinking identifier. For fields with different data, the output should be distinguished based on whether the Deep Thinking field has a value (there may be Deep Thinking identifiers without an initial value).

[0102] The state machine function performs state-driven content parsing and transformation on each streaming response. The specific steps are as follows:

[0103] The initial state of the state machine is set to deep thinking mode (which can only be changed to content mode). The deep thinking and content buffers are also initialized. Based on the field configuration in the strategy object, the field name of deep thinking (thinking_field) and the field name of content (content_field) are determined. If they are the same field, then thinking_field == content_field.

[0104] The actual value is extracted using the path parameters configured in the strategy. If parsing fails, a log is logged and the segment is skipped. Then, depending on the strategy type, it's determined whether to enable the deep thinking mode. If different fields are used, if the deep thinking field (thinking_field) receives a value from the response, the state machine state remains unchanged and is encapsulated as deep thinking content; if no value is received from the response, the state machine state changes to the main text mode and is encapsulated as main text content. If the same field is used, the state switch depends on the deep thinking start and stop flags in the content. If no start flag is received, it's considered main text by default; upon receiving the start flag, it enters the thinking state; upon receiving the stop flag, the state machine state changes to the main text mode. Successfully parsed content needs to be appended to the buffer.

[0105] Exception and boundary handling are as follows: If deep thinking has not ended (no stop indicator), but the flow has ended, a decision will be made based on the strategy to either force truncation or treat the remaining content as the main text. If the indicators in the same field are disordered (e.g., multiple start points without end points), the most recent valid indicator will be used.

[0106] Finally, the content is anonymized, the output format is standardized, and the content is output to the upstream client. At the same time, the end signal is listened for, the streaming connection is disconnected, and the session content is completed.

[0107] Step S4, System Processing and Control Module:

[0108] In this step, the system builds unified observability and resource governance capabilities to implement full lifecycle performance monitoring, stability assurance, and business operation support for large model call chains. Specifically, this includes: real-time collection of key performance indicators for each model service, dynamic switching to ensure system resilience, and enterprise-level multi-dimensional billing based on fine-grained call data (such as by token quantity, call count, model type, tenant / project, etc.).

[0109] Step S401, Indicator Collection and Recording:

[0110] This step uses embedded probes to continuously and non-intrusively collect runtime performance metrics and resource load status of various model services. The system automatically aggregates and persists the following core data: latency metrics (P95, P99 latency), success rate and error rate (statistical distribution of HTTP status codes), throughput, concurrent connections, token count, caller identifiers (such as tenant ID, application ID), request context (such as whether Deep Thinking is enabled, whether streaming response is enabled), etc. All metrics are reported to a time-series database in a standardized format, supporting long-term storage, backtracking analysis, and visualization dashboard integration.

[0111] Step S402, Monitoring, Alarms and Billing:

[0112] This step, based on the data collected in step S401, constructs a three-tiered capability system: intelligent monitoring and visualization, dynamic alarms and self-healing linkage, and enterprise-level refined billing. Intelligent monitoring and visualization provides a global model health dashboard, displaying real-time availability and latency trends for each model service. Dynamic alarms and self-healing linkage configure multi-level alarm rules (e.g., P99 latency > 2s for 5 minutes, success rate < 95%, sudden increase in error rate, etc.), notifying responsible personnel via email and other channels; simultaneously, it automatically triggers preset elastic strategies, such as temporarily switching to a backup model. Enterprise-level refined billing generates detailed billing by tenant, project, user, or APIKey dimension based on actual call volume (e.g., number of input and output tokens) and model pricing strategy (different models have different unit prices); it supports quota management and budget alerts, automatically notifying or restricting calls when consumption approaches a threshold; it outputs a standard billing format (CSV), compatible with financial system integration, and supports daily / weekly / monthly settlement cycles.

[0113] It should be understood that although this specification describes the system according to embodiments, not every embodiment contains only one independent technical solution. This narrative style is only for clarity, and those skilled in the art should consider this specification as a whole. The technical features in each embodiment can also be reasonably combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A large language model intelligent routing and unified adaptation method, characterized in that, The method is based on a large language model intelligent routing and unified adaptation system, the system includes a unified interface and risk identification module, a dynamic configuration module, a policy and dynamic selection module and a system processing and control module, and the method comprises the following steps: Step S1, unified interface and risk identification, first, provide a unified API interface for the upstream access end, shield all the differences of model input and output parameters, then count and record through the text processing minimum unit Token module, support enterprise-level cost governance, finally, through the sliding window, dynamically identify the input and output of the model; Step S2, dynamic configuration, define the configuration file, which consists of three parts: request template mapping, policy configuration list and default policy configuration, when the system loads the JSON file, read the model configuration information and convert it to configuration policy cache; dynamically perceive the changes of JSON file, and real-time hot update the configuration cache with model granularity; Step S3, policy and dynamic selection, according to the input information of the upstream system, select the corresponding model configuration information, and select the correct policy type for the request and return processing of the downstream large model; if the input information does not define a specific model, the system dynamically selects a matching model for processing; Step S4, system processing and control, the system builds a unified observability and resource governance capability, and implements performance monitoring, stability guarantee and commercial operation support for the whole life cycle of the large model calling link.

2. The large language model intelligent routing and unified adaptation method according to claim 1, characterized in that, The step S1 specifically comprises: Step S101, provide a unified interface: first, provide standard API endpoints, request methods and authentication information; second, define input parameters, and finally define output parameters, output through Server-Sent Events for streaming output; Step S102, record Token number: when input, extract the prompt word input by the user; when output, extract the text of deep thinking and body, get the correct text unit number through the Token counting service, and save it to this session and database; In this process, it is judged whether the user's remaining balance can support this session; Step S103, model risk identification: risk identification is performed on the prompt word input by the user and the content returned by the large model; The prompt word input by the user is identified for risk content through model risk identification, and the streaming content returned by the model is detected, generated and output through the sliding window mechanism, and the model risk identification is detected through the incremental sliding window and buffer mechanism.

3. The large language model intelligent routing and unified adaptation method according to claim 2, characterized in that, In the step S101: The input parameters include the model name of the request, whether deep thinking is needed, whether network search is needed, the identification of the session and the prompt word input by the user; the output parameters include the identification of the session, the type, the time, the model name, the confidence, whether it is ended, the end reason and the output formal content, which consists of index, role, body content, deep thinking content and whether there is deep thinking; if the model name is empty, the matching model is determined by the dynamic selection engine; the identification of the session is generated when the dialogue is created each time.

4. The large language model intelligent routing and unified adaptation method according to claim 2, characterized in that, The step S103 specifically comprises: Let the model output a sequence of Tokens (t = 1,2,…,T), where denotes the current time step index, denotes the final generation length, which is unknown in streaming scenarios and grows step by step, denotes the Token generated at the th time step; The model maintains a dynamic buffer of the last L tokens at each time step L tokens at each time step When t < L, take all history, where L denotes the maximum buffer length, used to limit memory and focus on recent context, the buffer is dynamically updated as new tokens arrive; And define the sliding window size as , , the overlap reserved length as O, 0≤O≤W; the risk discrimination is executed when the trigger condition is met, at this time the current discrimination window is extracted, and the last Token in the buffer is taken. wherein indicates the most recently generated content and the direct context, indicates taking the last elements; the triggering condition includes encountering a sentence end, the buffer length reaching or the time interval exceeding a threshold value; Finally, the obtained content is input into model risk discrimination, if there is high risk, immediately terminate generation and trigger blocking, and record risk log; If it is lower than the risk threshold, record the log and continue to generate; After completing a discrimination, reserve part of the historical Token as context overlap for the next round, and update the buffer according to the overlap strategy: reserved The individual Token avoids semantic fragmentation, and when a new Token arrives, it is appended to the end in a subsequent step. The individual Token avoids semantic fragmentation, and when a new Token arrives, it is appended to the end in a subsequent step.

5. The large language model intelligent routing and unified adaptation method according to claim 1, characterized in that, The step S2 specifically comprises: Step S201, load model configuration information: first define the JSON file framework, which consists of three parts: request template mapping, strategy configuration list and default strategy configuration: wherein is the structured JSON object, is the request template map, is the policy configuration list, is the default policy configuration; The request template mapping is used to define the request parameters and formats required by each model, the strategy configuration list is used to define the request format, output format and stream processing method of each model, and the default strategy configuration is used for models not defined in the strategy configuration list, including unknown models or unified models; Request template mapping Define multiple sets of different request templates, each of which contains a request template name, a parent request template name, the input parameters of the model, and additional parameters and embedded formats: wherein is a request template mapping set index, is an index taken from the index set , represents the th request template mapping object; each request template mapping: wherein represents the template name of the request template mapping object; parent class request template mapping name: The parent class request template mapping name is , indicating no parent class, or , indicating an inheritance relationship; indicates the input parameters and additional parameters and embedded formats of the model. By defining the request format of the model in the template mapping, the corresponding large model is correctly requested. The strategy configuration list defines multiple different model strategies: wherein is a set index of policy configurations, is an index taken from the index set , represents the th policy configuration object; each group of policy configurations contains the name of the policy configuration, the name of the parent policy configuration, the description, the list of applicable large models, the request template, and the response processing method: wherein represents the name of the policy configuration, is the parent policy configuration name of the policy configuration, is the description of the policy configuration, is the list of large models of the policy configuration, is the namespace of all models, is the request template of the policy configuration, that is ), is the response processing mode; the response processing mode includes the number of returned items by skipping the model, the stop identifier of the model output, preprocessing, content analysis, output formatting, and processing strategy; represents as follows: wherein represents the number of returned items by skipping the model, represents a stop identifier output by the model, represents pre-processing, represents content analysis, represents output formatting, represents a processing strategy; content analysis defines which field in the model returned data is the deep thinking content, which field is the body content, and which field is the deep thinking end identifier; is represented as follows: wherein represents a deep thinking field, represents a text field, represents a deep thinking end identifier; Processing strategy Includes: whether there was deep thinking and strategy type: wherein indicates whether there is deep thinking, indicates a strategy type; strategy type There are deep thinking, identification-based deep thinking, and content-based deep thinking; is indicated as follows: wherein denotes enabling deep thinking, denotes identity-based deep thinking, denotes content-based deep thinking; The default policy configuration defines a general model policy configuration for models not defined in the policy configuration list. Secondly, the system starts to read the JSON file and loads all the configuration information into the memory, and thread-safely parses it into a set of key-value pair mappings and default model configurations, and executes the configuration loading function: wherein represents a configuration parsing function, containing validity check, check content includes all referenced request template mapping must exist in , field type matches expectation, mandatory field cannot be empty and all referenced parent policy configuration name must exist in , the original configuration is converted into a deterministic function of the runtime cache , represents the model policy runtime mapping cache: wherein represents the global model namespace, a collection of names of all possible invoked large language models, represents a certain concrete model name, represents the model the corresponding resolved policy object, d is a reserved keyword for explicitly storing the default policy in the cache ; if the model is not declared in any policy, then it is not explicitly stored in but is dynamically returned at runtime ; the resolved policy object is as follows: wherein indicates that there is a certain policy configuration object; belongs to the policy configuration list , indicates the model name , included in the applicable model list of the policy , indicates the policy recursive inheritance parsing function, merging the configuration along the parent policy chain, and the child configuration overrides the parent configuration; The system parses the function Converts the original configuration Converts to a runtime cache , first checks the legality of the configuration, and then for each policy configuration Inheritance parsing is performed , and then according to its List maps the parsing result to the corresponding model name , forming a key-value pair ; at the same time, the default policy Is stored in a special key ; the finally generated Support Time complexity model strategy query; Finally, the configuration information is sent to the network device caching is performed; Step S202, dynamic perception of JSON file: Through the file system monitoring mechanism, the system dynamically perceives the change of the JSON file, reads the updated configuration information , and compares the original configuration information in step S201 , screens out the difference change, and performs the configuration hot loading in the model granularity, without restarting the service or interrupting the existing request flow: wherein is , is a runtime policy mapping cache generated based on the new configuration , the structure is consistent with , is a cache difference calculation function is a model containing only the semantic changes of the policy content and its new policy value; if there is a value, only the changed model key atomic replacement is performed: wherein represents a thread-safe cache update operation.

6. The large language model intelligent routing and unified adaptation method according to claim 3, characterized in that, The step S3 specifically comprises: Step S301, model selection: This step is executed when creating a session for the first time or changing the model, and is used to select the downstream large model for this session; First, according to the request parameters defined in step S101, the name of the downstream large model is obtained, if the parameter is not defined or the model configuration cached in step S2 does not exist, the matching model name is obtained by the dynamic selection engine; The dynamic selection engine selects the matching large language model according to the monitoring index information through the routing decision chain; The routing decision chain includes capability routing, cost routing, performance routing and load routing; Step S302, select processing strategy: This step loads model strategy object information, encapsulates the input, and requests the large model, and unifies the various output modes of the large model under different strategy modes into a standardized output mode; First, according to the large model name output in step S301, the strategy object information of the model is obtained from the model strategy cache in step S201; Then, the input of the upstream is converted into the request parameters of the large model according to the request template mapping object in the strategy object information, and the downstream large model service is called; Then, according to the response processing mode in the strategy object information, the response of the downstream is processed; If the number of returned jumps is not 0, the corresponding number of stream responses is ignored; Then, preprocess each response; Then, through content analysis, the deep thinking content, text content and deep thinking stop identifier are obtained, and through the JSON analysis algorithm, the actual value in the content path configuration is parsed; Then, according to the processing strategy of the strategy template, state conversion and content conversion are performed; in order to uniformly process the deep thinking and text mode of different models, a lightweight strategy engine is abstracted, if the strategy is to enable deep thinking, the large language model is marked with a deep thinking mode at all times, the parameter of whether the upstream request is deep thinking is ignored, the deep thinking part must be processed first, then the text part is processed, and the deep thinking, deep thinking stop identifier and text content are sequentially output through content analysis; If the strategy is content-based deep thinking, it needs to be processed through a content analysis algorithm, and whether this request is with a thinking mode is determined by the content or parameter value; If the strategy is based on the identification of deep thinking, it will be processed according to the parameter of whether the upstream request is deep thinking; the three strategies need to process the deep thinking mode and the text mode, which will have two cases: one is that the deep thinking and the text use the same field, and the other is that the deep thinking and the text use different fields; the same field needs to be based on the identification of deep thinking to distinguish whether the output is text or deep thinking content, and the different fields need to be based on whether the field of deep thinking has a value to distinguish; Then, through the state machine function, the state-driven content analysis and conversion of each stream response are performed, and the specific steps are as follows: The initial state of the state machine is set to deep thinking mode, and the deep thinking and text buffer areas are initialized, the deep thinking field name thinking_field and the text field name content_field are determined according to the field configuration in the strategy object, if they are the same field, thinking_field == content_field; The actual value is extracted using the path parameter configured in the strategy, if the parsing fails, the log is recorded and the field is skipped; then, it is decided whether to enable the deep thinking mode according to the strategy type; If different fields are used, the deep thinking field thinking_field gets the value from the response, the state machine state remains unchanged, and is encapsulated as deep thinking content, and if no value is obtained from the response, the state machine state is changed to text mode, and is encapsulated as text content; If the same field is used, the state is switched depending on the deep thinking start identifier and stop identifier in the content, if no start identifier is received, it is considered as text by default, if the start identifier is received, the thinking state is entered, and if the stop identifier is received, the state machine state is changed to text mode; the content analysis is successful and needs to be appended to the buffer area; The abnormality and boundary processing is as follows: if the deep thinking is not ended but the stream is ended, it is decided according to the strategy whether to forcibly truncate or consider the remaining content as text; if the identifier in the same field is disordered, the last valid identifier is used; Finally, the content is desensitized, the output format is unified, and the content is output to the upstream client, the end identifier is listened to, the stream connection is disconnected, and the session content is completed.

7. The large language model intelligent routing and unified adaptation method according to claim 1, characterized in that, The step S4 specifically comprises: Step S401, indicator collection and recording: through the embedded probe, continuously and low-invasively collect the runtime performance indicators and resource load status of each model service; the system automatically aggregates and persists the following core data: delay indicators, success rate and error rate, throughput, concurrent connection number, Token number, caller identification and request context; all indicators are reported to the time series database through a standardized format; Step S402, monitoring, alarm and billing: based on the data collected in step S401, a three-layer capability system is built: intelligent monitoring and visualization, dynamic alarm and self-healing linkage, and enterprise-level refined billing; Intelligent monitoring and visualization provide a global model health dashboard, which real-time displays the availability and delay trend of each model service; Dynamic alarm and self-healing linkage configure multi-level alarm rules, notify the responsible person through various ways; at the same time, automatically trigger the preset elastic strategy, including temporary switching to standby model; Enterprise-level refined billing generates billing details according to the actual call volume and model pricing strategy, based on tenant, project, user or APIKey dimension; supports quota management and budget early warning, automatically notifies or restricts calls when consumption approaches the threshold; outputs standard bill format CSV, compatible with financial system docking, supports daily / weekly / monthly cycle settlement.

Images